pytest "DID NOT RAISE" flaky: expected exception raised only sometimes under parallel xdist

An exception that fires only sometimes is not a flaky assertion, it is a precondition that only sometimes exists. Under xdist the test runs in a different order or on a worker missing the setup another test used to provide. Make the test build its own precondition and the raise becomes deterministic.

pytest "DID NOT RAISE" flaky: expected exception raised only sometimes under parallel xdist

Steps

1. Run the test alone, repeatedly, with -n0. If it always raises alone, the exception depends on something the parallel run changes: ordering, worker isolation, or shared state.

   Expected: You know whether the test is self-sufficient or secretly depends on its neighbors.

2. Read what the exception needs in order to fire: a duplicate key, a missing row, a locked file, a validation trigger. Then check whether the test itself creates that precondition or just assumes it.

   Expected: You can name the exact precondition the raise depends on.

3. Look at the tests that run near it: one of them probably creates the precondition as a side effect (inserting the duplicate row, creating the file). Under xdist that test may run on another worker or in another order.

   Expected: You find the neighbor test whose side effect the raise was freeloading on.

4. Make the test self-sufficient: build the precondition inside the test or in its own fixture, so the raise fires no matter the order or worker.

   Expected: The test raises deterministically under -n0, -n4, and in any order.

5. Rerun the suite a few times under xdist and confirm the DID NOT RAISE flake is gone, not just quieter.

   Expected: Multiple parallel runs pass with zero occurrences of the failure.

Use this when

• pytest.raises intermittently reports DID NOT RAISE under xdist
• an exception test passes alone but flakes in parallel
• the raise depends on database rows, files, or state another test creates

Not for this skill when

• the exception never raises even in isolation (the code path is wrong)
• the flake happens in serial runs too (look for timing, not ordering)
• the test expects no exception at all

Variant phrasings

• pytest DID NOT RAISE flaky under xdist
• expected exception raised only sometimes parallel
• pytest.raises passes alone fails intermittently with xdist

Why it happens

pytest.raises asserts that a block of code throws. Whether it throws depends on the world the code sees when it runs. In serial runs the world is built up by every earlier test, so a test can pass for years while secretly depending on a neighbor's leftovers. xdist shuffles the order and splits tests across workers, and suddenly the neighbor's leftovers are not there. The test did not become flaky; the hidden dependency became visible.

Edge cases

• Workers also get fresh interpreters, so module-level caches that masked the dependency in serial runs do not exist under xdist.
• If the precondition is expensive to build, scope it to the test's module or class rather than rebuilding per test - but keep it inside the test file.
• Random ordering plugins make this worse and also make it reproducible: capture the failing seed and rerun with it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AZxd4SlBuqcRo52VB3B2Dw