pytest ValueError: I/O operation on closed file flake under parallel runs - how to debug

TL;DR

Two tests share one file object, usually a module-level open() or a module-scoped fixture, and one test's teardown closes it while another test still holds it. Serial runs pass because the order is always safe; parallel scheduling with xdist reshuffles the order and exposes the race. Find the shared handle, give each test its own file through a function-scoped fixture or tmp_path, and the flake disappears.

The query

pytest ValueError: I/O operation on closed file flake under parallel runs  -  how to debug

Use this when

• ValueError: I/O operation on closed file flakes under pytest -n
• the error moves between test files from run to run
• a module-level open() or a wide-scoped file fixture exists

Not for

• the error happens in a single process without xdist (the close/write ordering bug is deterministic; fix it directly)
• the closed file belongs to application code rather than test fixtures (fix the app's lifecycle)
• the error is about sockets or pipes rather than files (same idea, different handle type)

Steps

1. Find the shared file object

Search for files opened outside of with blocks or at module scope. Those are the handles that outlive a single test.

grep -rn "^f = open\|^file = open\|open(" tests/conftest.py tests/test_billing.py | head -20

Expected output: the shared handle is found: a module-level open() or a wide-scoped fixture holding a file.

2. Check the fixture scopes around it

A session- or module-scoped fixture that opens a file and closes it in teardown is shared across tests; whichever test triggers teardown first wins the race.

# conftest.py: the classic shape of the bug
@pytest.fixture(scope="module")
def report_file():
    f = open("report.csv", "w")
    yield f
    f.close()  # one test's teardown closes it for everyone

Expected output: the scope that shares the handle across tests is identified.

3. Isolate the file per test

Narrow the fixture to function scope and write to a unique path per test with tmp_path, so no test can close another test's file.

@pytest.fixture
def report_file(tmp_path):
    p = tmp_path / "report.csv"
    with open(p, "w") as f:
        yield f

Expected output: each test gets its own handle on its own path; sharing is impossible.

4. Rerun with parallelism and repeat

Run the previously flaky selection with xdist several times. Repetition is the acceptance check for a race fix.

pytest -n 8 tests/test_billing.py --count=3 -q 2>/dev/null || pytest -n 8 tests/test_billing.py -q

Expected output: the suite is green across repeated parallel runs with no closed-file errors.

Tool compatibility

• pytest 7 - 8
• pytest-xdist 3.x

Variant phrasings

• I/O operation on closed file flake under parallel runs
• ValueError closed file pytest-xdist
• pytest parallel closed file handle flake

Why it happens

xdist workers each run many tests in one process, and fixtures wider than function scope are shared across those tests. Teardown ordering between tests is not guaranteed, so a teardown that closes a shared file can run while another test still writes to it. Serial runs mask this because the order happens to be safe every time; parallel scheduling makes the unsafe interleaving likely.

Edge cases / pitfalls

• logging handlers that hold open files across tests are a sneaky variant; check handlers added in conftest
• pytest's own capture can interact with closed file descriptors; the traceback usually still names your file
• caplog and capsys fixtures are function-scoped by design; custom file fixtures should follow the same pattern
• if the file must be shared for performance, guard it with a lock and never close it in per-test teardown

Provenance

Resolved from the public thread: https://vectle.com/posts/pst1LytsevhM7KCk-ch9cSzA