Pin the shuffle seed and the flake becomes reproducible. pytest-randomly prints the seed it used at the start of every run - grab that seed from the failing CI log and re-run locally with the same seed to get the exact same test order. If the failure reappears, you have a real order dependence; if the message says "Falsifying example", Hypothesis found a failing input and the seed story is a different one (covered in step 4). Either way, never chase a vanish-on-rerun failure without the seed - you are testing a different order every time.

```text
pytest-randomly "Falsifying example" failures that vanish on rerun: how to capture the seed
```

## Steps
1. Open the failing CI log and find the seed line near the top. pytest-randomly prints something like "Using --randomly-seed=123456" before the first test runs. Copy that number.
   Expected: you have a numeric seed, e.g. 123456, taken from the exact run that failed.

2. Re-run locally with the seed pinned: run pytest with --randomly-seed=123456 (the number from the log). Do not re-run without the flag - a fresh seed shuffles differently and the failure will likely vanish again.
   Expected: the same failure appears locally, in the same test and the same order position.

3. If the failure reproduces with the seed, disable shuffling once to confirm order dependence: re-run with -p no:randomly. If the suite goes green, some earlier test is leaking state that the failing test depends on.
   Expected: the suite passes with shuffling off and fails with the pinned seed - you have confirmed test-order dependence, not a broken test.

4. If the output literally says "Falsifying example", that is Hypothesis, not pytest-randomly. Find the failing example in the log, shrink it with --hypothesis-seed or the printed example, and treat it as a deterministic failing input to the property test.
   Expected: you know which tool produced the message: Hypothesis failing inputs are reproducible from the printed example, pytest-randomly orderings from the seed.

5. Record the seed next to the bug report or the quarantine entry so anyone picking this up can reproduce it in one command.
   Expected: a teammate can paste one command and see the same failure.

## Use this when

- a pytest failure appears in CI under pytest-randomly and disappears on local rerun
- the CI log mentions a random seed or the failure moves between tests run to run
- you need to decide whether a flake is order-dependent before quarantining it

## Not for this skill when

- failures that happen with shuffling disabled too (that is a plain broken test, reproduce it normally)
- when you want the fix for the order dependence itself rather than reproduction - this skill only gets you a reliable repro

## Variant phrasings

### pytest-randomly seed not printed in CI output

Some CI setups truncate the header. Add -v or check the full log artifact; the seed line appears before the first dot. Alternatively set the seed yourself in CI config so every run is pinned from the start.

### hypothesis falsifying example vs randomly seed

Different tools, different fixes. pytest-randomly shuffles test order; Hypothesis generates failing inputs. The log tells you which: "Using --randomly-seed" is the former, "Falsifying example" is the latter.

## Why it happens

pytest-randomly shuffles test order with a fresh random seed on every run unless you pin one. A test that depends on leaked state from an earlier test (a temp file, a module-level cache, a database row) only fails when the polluting test happens to run first - which is a matter of shuffle luck. Hypothesis, separately, reports minimal failing inputs as "Falsifying example". Both look like vanish-on-rerun flakes, and both are reproducible once you capture the right seed.

## Edge cases

- CI log truncated and no seed line: pin a fixed seed in your CI config so every future run is reproducible by default
- seed reproduces the order but the failure still moves: two tests may be racing each other rather than leaking state - look at timing, not just order
- Hypothesis database of failing examples (the .hypothesis directory) is gitignored in CI: failing inputs are not reused across runs, so always copy the printed example into the bug report

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BTatpCb5hYzrMVLvzQgwTg