my agent ran the failing test alone and said "not reproducible" - pytest-randomly had shuffled the order in CI

TL;DR "Not reproducible" is the wrong answer when CI shuffles test order with pytest-randomly. Capture the failing seed from CI and replay it locally - the bug is in the order, not the test.

my agent ran the failing test alone and said "not reproducible"  -  pytest-randomly had shuffled the order in CI

Steps

1. Get the seed from the failing CI run. pytest-randomly prints its seed at the start of the run (look for "Using --randomly-seed=" in the CI log).

Expected: you have the exact seed number from the red run.

1. Replay that seed locally. Run pytest with -p randomly --randomly-seed=[the seed] on the same test selection CI used.

Expected: the failure reproduces locally with the identical order.

1. Bisect the order. With the seed fixed, remove tests from the selection until the failure disappears; the last removed test (or group) is the polluter.

Expected: you identify which earlier test's state breaks the victim.

1. Fix the shared state: add the missing teardown to the polluter or isolate the victim's fixtures so order stops mattering.

Expected: the suite passes under the failing seed and under several other random seeds.

1. Teach the agent the seed workflow. Its runbook should say: for any CI-only failure under pytest-randomly, first step is always "get the seed", never "run the test solo".

Expected: the agent stops declaring CI-only failures unreproducible.

Use this when

• CI uses pytest-randomly (or any order shuffling) and local runs do not
• the agent ran the test alone and gave up
• the failure appears and disappears across CI runs with no code change
• you have (or can get) the seed from the failing run

Not for this skill when

• the suite does not shuffle order anywhere
• the test fails standalone too (order is irrelevant)
• the seed was not recorded and the failure no longer reproduces (note it and move on)
• the flake is timing-based rather than order-based

Variant phrasings

• "pytest-randomly failure cannot reproduce locally"
• "how to capture the randomly seed from a failing CI run"
• "test passes alone but fails under shuffled order"

Why it happens

pytest-randomly reorders tests per run using a seed. A victim test that depends on (or is broken by) another test's leftover state fails only under seeds that place the polluter first. Running the test solo uses no seed at all, so the agent tests a scenario that never happens in CI and concludes the failure is imaginary.

Edge cases

• The seed only reproduces with the same test selection and the same plugin versions; pin both.
• xdist plus randomly shuffles per worker, which complicates replay; reproduce with the same worker count first.
• If CI does not log the seed, add -v or a conftest hook that prints it; without the seed you are guessing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nH6FY0THdZfqdOnEkOLqzw