pytest-xdist "INTERNALERROR" crashing CI runs: how to debug the worker failure

INTERNALERROR means a worker process died, not that a test failed - so stop reading test output and read the crash report. Rerun without xdist to isolate it, find which test the worker died on, and look for unpicklable returns, segfaults from native code, or a conftest that kills the interpreter.

pytest-xdist "INTERNALERROR" crashing CI runs: how to debug the worker failure

Steps

1. Read the crash report at the top of the output, not the tail. INTERNALERROR prints the worker-side traceback first; that traceback names the real exception the worker died with.

   Expected: You have the worker's traceback and the exception type that killed it.

2. Rerun the same suite with -n0. If it passes serially, the crash is xdist-specific: something about the test cannot survive the worker boundary, usually pickling.

   Expected: You know whether the crash needs xdist to happen or is a general crash.

3. If it crashes serially too, bisect to the killing test: run files one at a time, then use --lf or deselection to isolate. The last test before the crash is your suspect.

   Expected: You have the single test or fixture that kills the process.

4. Check the suspect for worker-killers: returning unpicklable objects from tests, C extensions that segfault, os._exit or sys.exit calls, and fixtures that exhaust memory or file descriptors.

   Expected: You find the line or object that cannot cross the worker boundary.

5. Fix the killer: make returned objects picklable, guard the native call, or mark the test to run outside xdist. Rerun the full parallel suite to confirm the INTERNALERROR is gone.

   Expected: The parallel run completes with normal test results and no crash report.

Use this when

• a parallel run aborts with INTERNALERROR instead of results
• CI shows a crash report from an xdist worker
• workers die partway through the suite with no test failure recorded

Not for this skill when

• tests fail normally with FAILED lines (that is not a worker crash)
• the error happens during collection before workers spawn
• the run hangs instead of crashing (that is a deadlock, different skill)

Variant phrasings

• pytest-xdist INTERNALERROR how to debug
• xdist worker crashed CI INTERNALERROR
• pytest INTERNALERROR worker failure which test caused it

Why it happens

xdist runs your tests in separate worker processes and pickles results back to the controller. INTERNALERROR is the controller telling you a worker died unexpectedly - the test never got to report a result because its process is gone. The causes cluster: something the test returned could not be pickled, native code segfaulted, or something called exit inside the worker. Reading it as a test failure sends you hunting in the wrong place; it is a process crash wearing a test runner's clothes.

Edge cases

• Add -v so the output shows which test each worker was running when it died; without it you are guessing.
• Memory exhaustion looks like a silent worker death with no traceback; watch the CI runner's memory graph if the crash report is empty.
• If the crash is flaky, suspect resource exhaustion or a race in test ordering rather than a single bad test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PiMQ7PmFiOFExyh4bTYbbg