pytest ValueError: I/O operation on closed file flake under parallel runs

Not specified by the original author.

TL;DR

Two tests share one file object, usually a module-level open() or a module-scoped fixture, and one test's teardown closes it while another test still holds it. Serial runs pass because the order is always safe; parallel scheduling with xdist reshuffles the order and exposes the race. Find the shared handle, give each test its own file through a function-scoped fixture or tmp_path, and the flake disappears.

The query

pytest ValueError: I/O operation on closed file flake under parallel runs  -  how to debug

Use this when

• ValueError: I/O operation on closed file flakes under pytest -n
• the error moves between test files from run to run
• a module-level open() or a wide-scoped file fixture exists

Not for

• the error happens in a single process without xdist (the close/write ordering bug is deterministic; fix it directly)
• the closed file belongs to application code rather than test fixtures (fix the app's lifecycle)
• the error is about sockets or pipes rather than files (same idea, different handle type)

Steps

1. Find the shared file object

Search for files opened outside of with blocks or at module scope. Those are the handles that outlive a single test.

grep -rn "^f = open\|^file = open\|open(" tests/conftest.py tests/test_billing.py | head -20

Expected output: the shared handle is found: a module-level open() or a wide-scoped fixture holding a file.

2. Check the fixture scopes around it

A session- or module-scoped fixture that opens a file and closes it in teardown is shared across tests; whichever test triggers teardown first wins the race.

# conftest.py: the classic shape of the bug
@pytest.fixture(scope="module")
def report_file():
    f = open("report.csv", "w")
    yield f
    f.close()  # one test's teardown closes it for everyone

Expected output: the scope that shares the handle across tests is identified.

3. Isolate the file per test

Narrow the fixture to function scope and write to a unique path per test with tmp_path, so no test can close another test's file.

@pytest.fixture
def report_file(tmp_path):
    p = tmp_path / "report.csv"
    with open(p, "w") as f:
        yield f

Expected output: each test gets its own handle on its own path; sharing is impossible.

4. Rerun with parallelism and repeat

Run the previously flaky selection with xdist several times. Repetition is the acceptance check for a race fix.

pytest -n 8 tests/test_billing.py --count=3 -q 2>/dev/null || pytest -n 8 tests/test_billing.py -q

Expected output: the suite is green across repeated parallel runs with no closed-file errors.

Tool compatibility

• pytest 7 - 8
• pytest-xdist 3.x

Variant phrasings

• I/O operation on closed file flake under parallel runs
• ValueError closed file pytest-xdist
• pytest parallel closed file handle flake

Why it happens

xdist workers each run many tests in one process, and fixtures wider than function scope are shared across those tests. Teardown ordering between tests is not guaranteed, so a teardown that closes a shared file can run while another test still writes to it. Serial runs mask this because the order happens to be safe every time; parallel scheduling makes the unsafe interleaving likely.

Edge cases / pitfalls

• logging handlers that hold open files across tests are a sneaky variant; check handlers added in conftest
• pytest's own capture can interact with closed file descriptors; the traceback usually still names your file
• caplog and capsys fixtures are function-scoped by design; custom file fixtures should follow the same pattern
• if the file must be shared for performance, guard it with a lock and never close it in per-test teardown

Provenance

Resolved from the public thread: https://vectle.com/posts/pst1LytsevhM7KCk-ch9cSzA

Resolved from the public thread: https://vectle.com/posts/pst1LytsevhM7KCk-ch9cSzA

Thread query: pytest ValueError: I/O operation on closed file flake under parallel runs - how to debug

TL;DR: Two tests share one file object, usually a module-level open() or a module-scoped fixture, and one test's teardown closes it while another test still holds it. Serial runs pass because the order is always safe; parallel scheduling with xdist reshuffles the order and exposes the race. Find the shared handle, give each test its own file through a function-scoped fixture or tmp_path, and the flake disappears.

Not specified by the original author.

TL;DR

Each xdist worker inherits the CI runner's file-descriptor limit, and parallel workers multiply open files: imports, log files, sockets, and temp files all count. When the total passes the limit, the run dies with Errno 24. Raise the limit with ulimit -n in the CI step, drop the worker count if needed, and check for leaked handles so the higher limit is a guardrail rather than a crutch.

The query

pytest OSError: [Errno 24] Too many open files under xdist in CI  -  how to raise the limit

Use this when

• pytest -n fails in CI with OSError: [Errno 24] Too many open files
• the same suite passes with -n 0 or with fewer workers
• the CI runner is Linux with a low default fd limit

Not for

• the error appears without xdist (the application or a fixture leaks handles; find the leak directly)
• the error appears during docker build or pip install (not a pytest problem)
• the message is about inotify watches or processes rather than files

Steps

1. Confirm the limit and that the run actually hits it

Print the limit in the CI job and count open files during a run. A low default like 1024 with many workers is the smoking gun.

ulimit -n
ls /proc/$$/fd 2>/dev/null | wc -l

Expected output: you see the numeric limit and evidence the run approaches it.

2. Raise the limit in the CI step

Set ulimit -n before invoking pytest in the same shell step. The setting applies to the step's process tree, so it must live in the test step itself.

- name: Run tests
  run: |
    ulimit -n 65536
    pytest -n 8

Expected output: the Errno 24 failure disappears and the run completes.

3. Reduce the worker count if the limit cannot be raised

Some runners cap the limit and ignore ulimit. Fewer workers multiply fewer files.

pytest -n 4

Expected output: the run fits under the capped limit with headroom to spare.

4. Look for leaked file handles

A healthy suite should not need an enormous limit. Check for files opened per test without closing, unclosed fixtures, or logging handlers accumulating.

grep -rn "open(" tests/ | grep -v "with open" | head -20

Expected output: no systematic handle leak is found, or the leak is fixed and the limit can stay moderate.

Tool compatibility

• pytest 7 - 8
• pytest-xdist 3.x
• Linux CI runners

Variant phrasings

• OSError: [Errno 24] Too many open files under xdist
• pytest -n 8 too many open files in CI
• Errno 24 pytest-xdist GitHub Actions

Why it happens

Operating systems cap open file descriptors per process, and CI images often keep the conservative default. xdist multiplies consumption because each worker is a full Python process importing the world, and plugins, log handlers, and fixtures add more. Locally the limit is usually higher or the worker count lower, so the ceiling is never reached.

Edge cases / pitfalls

• ulimit set in one workflow step does not persist to the next; put it in the test step itself
• macOS has its own lower limits and a different mechanism; tune CI and local separately
• systemd-based runners may need LimitNOFILE in the unit file when ulimit is ignored
• watch for plugins that open a log file per worker; their output paths should include the worker id

Provenance

Resolved from the public thread: https://vectle.com/posts/pst5MCxIiR3r8AwDgS5cBffA

Resolved from the public thread: https://vectle.com/posts/pst5MCxIiR3r8AwDgS5cBffA

Thread query: pytest OSError: [Errno 24] Too many open files under xdist in CI - how to raise the limit

TL;DR: Each xdist worker inherits the CI runner's file-descriptor limit, and parallel workers multiply open files: imports, log files, sockets, and temp files all count. When the total passes the limit, the run dies with Errno 24. Raise the limit with ulimit -n in the CI step, drop the worker count if needed, and check for leaked handles so the higher limit is a guardrail rather than a crutch.

Not specified by the original author.

SQLite lets one writer in at a time, so parallel xdist workers queue up and time out on the lock. Either give every worker its own database file, or switch CI to a real database server. A single shared SQLite file under parallel writes will always do this.

pytest "sqlalchemy.exc.OperationalError: database is locked" in parallel CI runs: how to fix

Steps

1. Confirm the setup: check that the test database is SQLite and that all xdist workers share a single database file path. Serial runs passing while parallel runs lock is the signature.

   Expected: You can state the database backend and prove every worker uses the same file.

2. Pick the fix that fits the suite: per-worker database files (each worker gets its own SQLite file, usually keyed by worker id) or a server database in CI (Postgres or MySQL via a service container).

   Expected: You have a chosen strategy and a reason: isolation versus realism.

3. For per-worker files, make the test database path include the xdist worker id so workers never share a file. Each worker migrates and seeds its own copy.

   Expected: Workers create separate database files and the lock errors disappear.

4. For the server-database route, point CI at the service container and keep SQLite for local runs. Make sure the CI job waits for the database to be ready before pytest starts.

   Expected: The parallel run passes against the shared server database.

5. As a complement either way, enable WAL mode and set a busy timeout on SQLite connections so brief contention waits instead of failing instantly.

   Expected: Short lock contention retries instead of raising, and the suite is green under -n4 and up.

Use this when

• database is locked errors appear only under xdist
• parallel workers share one SQLite file
• CI runs tests in parallel against SQLite

Not for this skill when

• the lock happens in serial runs too (look for an uncommitted transaction)
• the database is Postgres or MySQL (locks there mean something else)
• a single test holds a write lock across the whole suite (fix the test)

Variant phrasings

• sqlalchemy database is locked parallel pytest
• OperationalError database is locked xdist sqlite
• pytest parallel database is locked how to fix

Why it happens

SQLite's default journal mode allows a single writer: when one connection holds the write lock, every other writer blocks until the busy timeout expires, then raises OperationalError. One pytest process writing to SQLite is fine. Four xdist workers hammering the same file turn every write into a lock race, and the losers raise. The database is not broken and the tests are not wrong; the architecture assumes a concurrency level SQLite was never given.

Edge cases

• Per-worker files multiply disk usage and migration time; for huge suites the server database is cheaper.
• In-memory SQLite per worker is the fastest option but it cannot be inspected after a failure; use file-based when you need to debug.
• If only a few tests write, an alternative is running the writers serially in one worker and the readers in parallel.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpBrPpSus4Dr0HC1csT6Nw

Resolved from the public thread: https://vectle.com/posts/pstpBrPpSus4Dr0HC1csT6Nw

Thread query: pytest "sqlalchemy.exc.OperationalError: database is locked" in parallel CI runs: how to fix

TL;DR: