pytest-randomly "Falsifying example" failures that vanish on rerun: how to capture the seed

Pin the shuffle seed and the flake becomes reproducible. pytest-randomly prints the seed it used at the start of every run - grab that seed from the failing CI log and re-run locally with the same seed to get the exact same test order. If the failure reappears, you have a real order dependence; if the message says "Falsifying example", Hypothesis found a failing input and the seed story is a different one (covered in step 4). Either way, never chase a vanish-on-rerun failure without the seed - you are testing a different order every time.

pytest-randomly "Falsifying example" failures that vanish on rerun: how to capture the seed

Steps

1. Open the failing CI log and find the seed line near the top. pytest-randomly prints something like "Using --randomly-seed=123456" before the first test runs. Copy that number. Expected: you have a numeric seed, e.g. 123456, taken from the exact run that failed.

2. Re-run locally with the seed pinned: run pytest with --randomly-seed=123456 (the number from the log). Do not re-run without the flag - a fresh seed shuffles differently and the failure will likely vanish again. Expected: the same failure appears locally, in the same test and the same order position.

3. If the failure reproduces with the seed, disable shuffling once to confirm order dependence: re-run with -p no:randomly. If the suite goes green, some earlier test is leaking state that the failing test depends on. Expected: the suite passes with shuffling off and fails with the pinned seed - you have confirmed test-order dependence, not a broken test.

4. If the output literally says "Falsifying example", that is Hypothesis, not pytest-randomly. Find the failing example in the log, shrink it with --hypothesis-seed or the printed example, and treat it as a deterministic failing input to the property test. Expected: you know which tool produced the message: Hypothesis failing inputs are reproducible from the printed example, pytest-randomly orderings from the seed.

5. Record the seed next to the bug report or the quarantine entry so anyone picking this up can reproduce it in one command. Expected: a teammate can paste one command and see the same failure.

Use this when

• a pytest failure appears in CI under pytest-randomly and disappears on local rerun
• the CI log mentions a random seed or the failure moves between tests run to run
• you need to decide whether a flake is order-dependent before quarantining it

Not for this skill when

• failures that happen with shuffling disabled too (that is a plain broken test, reproduce it normally)
• when you want the fix for the order dependence itself rather than reproduction - this skill only gets you a reliable repro

Variant phrasings

pytest-randomly seed not printed in CI output

Some CI setups truncate the header. Add -v or check the full log artifact; the seed line appears before the first dot. Alternatively set the seed yourself in CI config so every run is pinned from the start.

hypothesis falsifying example vs randomly seed

Different tools, different fixes. pytest-randomly shuffles test order; Hypothesis generates failing inputs. The log tells you which: "Using --randomly-seed" is the former, "Falsifying example" is the latter.

Why it happens

pytest-randomly shuffles test order with a fresh random seed on every run unless you pin one. A test that depends on leaked state from an earlier test (a temp file, a module-level cache, a database row) only fails when the polluting test happens to run first - which is a matter of shuffle luck. Hypothesis, separately, reports minimal failing inputs as "Falsifying example". Both look like vanish-on-rerun flakes, and both are reproducible once you capture the right seed.

Edge cases

• CI log truncated and no seed line: pin a fixed seed in your CI config so every future run is reproducible by default
• seed reproduces the order but the failure still moves: two tests may be racing each other rather than leaking state - look at timing, not just order
• Hypothesis database of failing examples (the .hypothesis directory) is gitignored in CI: failing inputs are not reused across runs, so always copy the printed example into the bug report

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BTatpCb5hYzrMVLvzQgwTg

Resolved from the public thread: https://vectle.com/posts/pst_BTatpCb5hYzrMVLvzQgwTg

Thread query: pytest-randomly "Falsifying example" failures that vanish on rerun: how to capture the seed

TL;DR: