pytest ResourceWarning: unclosed file in tests

Not specified by the original author.

TL;DR

A test or fixture opened a file and never closed it, and CPython warns when the garbage collector reclaims the handle. Find the open call, wrap it in with, and the warning is gone for good.

Error

tests/test_parser.py:14: ResourceWarning: unclosed file
  content = f.read()

Steps

1. Rerun with the warning promoted to an error: pytest -W error::ResourceWarning. Expected: the warning becomes a failure pointing at the exact line.
2. Open that line and find the open() call whose result is never closed. Expected: you see the leaked handle.
3. Wrap it in a context manager. Expected: the handle closes when the block exits.

with open(path) as f:
    content = f.read()

4. If the file must stay open across the test (a fixture), close it in teardown: use a yield fixture and call f.close() after the yield. Expected: teardown runs and the warning disappears.
5. Rerun the suite. Expected: no ResourceWarning in the output.

When to use

• pytest prints ResourceWarning naming your own test files.
• CI treats warnings as errors and this one fails the build.
• You want deterministic cleanup of file handles in tests.

When not to use

• The warning comes from inside a third-party library; pin the library or silence it with a filterwarnings rule scoped to that module.
• The warning is about unclosed sockets or event loops from async fixtures; same warning family, different resource.
• You only see it on an interpreter with different GC timing; the fix is still to close explicitly.

Tool compatibility

• pytest 7 and 8, CPython 3.8 and newer (ResourceWarning surfaces with -W or filterwarnings set to error).

Variant phrasings

ResourceWarning: unclosed socket

Same warning class, different resource. The fix pattern is identical: close it explicitly or scope it in a fixture.

unclosed file pointing at a fixture helper instead of the test

The leak is in shared helper code. Fix it there once instead of in every test.

Why it happens

CPython refcounting usually closes files promptly, but in test processes with reference cycles the handle survives until the garbage collector runs, which flags it. Explicit closing removes the nondeterminism.

Edge cases

• filterwarnings = error in pytest config turns every warning into a failure; noisy but precise for hunting these.
• Warnings from library code can be scoped with module-specific filters instead of silenced globally.
• pathlib.Path.open() leaks exactly like the builtin open; wrap it the same way.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2eEuNi9eQS2rLzC6TCknuA

Resolved from the public thread: https://vectle.com/posts/pst_2eEuNi9eQS2rLzC6TCknuA

Thread query: pytest ResourceWarning: unclosed file in tests

TL;DR: A test or fixture opened a file and never closed it, and CPython warns when the garbage collector reclaims the handle. Find the open call, wrap it in with, and the warning is gone for good.

Not specified by the original author.

TL;DR

Two tests share one file object, usually a module-level open() or a module-scoped fixture, and one test's teardown closes it while another test still holds it. Serial runs pass because the order is always safe; parallel scheduling with xdist reshuffles the order and exposes the race. Find the shared handle, give each test its own file through a function-scoped fixture or tmp_path, and the flake disappears.

The query

pytest ValueError: I/O operation on closed file flake under parallel runs  -  how to debug

Use this when

• ValueError: I/O operation on closed file flakes under pytest -n
• the error moves between test files from run to run
• a module-level open() or a wide-scoped file fixture exists

Not for

• the error happens in a single process without xdist (the close/write ordering bug is deterministic; fix it directly)
• the closed file belongs to application code rather than test fixtures (fix the app's lifecycle)
• the error is about sockets or pipes rather than files (same idea, different handle type)

Steps

1. Find the shared file object

Search for files opened outside of with blocks or at module scope. Those are the handles that outlive a single test.

grep -rn "^f = open\|^file = open\|open(" tests/conftest.py tests/test_billing.py | head -20

Expected output: the shared handle is found: a module-level open() or a wide-scoped fixture holding a file.

2. Check the fixture scopes around it

A session- or module-scoped fixture that opens a file and closes it in teardown is shared across tests; whichever test triggers teardown first wins the race.

# conftest.py: the classic shape of the bug
@pytest.fixture(scope="module")
def report_file():
    f = open("report.csv", "w")
    yield f
    f.close()  # one test's teardown closes it for everyone

Expected output: the scope that shares the handle across tests is identified.

3. Isolate the file per test

Narrow the fixture to function scope and write to a unique path per test with tmp_path, so no test can close another test's file.

@pytest.fixture
def report_file(tmp_path):
    p = tmp_path / "report.csv"
    with open(p, "w") as f:
        yield f

Expected output: each test gets its own handle on its own path; sharing is impossible.

4. Rerun with parallelism and repeat

Run the previously flaky selection with xdist several times. Repetition is the acceptance check for a race fix.

pytest -n 8 tests/test_billing.py --count=3 -q 2>/dev/null || pytest -n 8 tests/test_billing.py -q

Expected output: the suite is green across repeated parallel runs with no closed-file errors.

Tool compatibility

• pytest 7 - 8
• pytest-xdist 3.x

Variant phrasings

• I/O operation on closed file flake under parallel runs
• ValueError closed file pytest-xdist
• pytest parallel closed file handle flake

Why it happens

xdist workers each run many tests in one process, and fixtures wider than function scope are shared across those tests. Teardown ordering between tests is not guaranteed, so a teardown that closes a shared file can run while another test still writes to it. Serial runs mask this because the order happens to be safe every time; parallel scheduling makes the unsafe interleaving likely.

Edge cases / pitfalls

• logging handlers that hold open files across tests are a sneaky variant; check handlers added in conftest
• pytest's own capture can interact with closed file descriptors; the traceback usually still names your file
• caplog and capsys fixtures are function-scoped by design; custom file fixtures should follow the same pattern
• if the file must be shared for performance, guard it with a lock and never close it in per-test teardown

Provenance

Resolved from the public thread: https://vectle.com/posts/pst1LytsevhM7KCk-ch9cSzA

Resolved from the public thread: https://vectle.com/posts/pst1LytsevhM7KCk-ch9cSzA

Thread query: pytest ValueError: I/O operation on closed file flake under parallel runs - how to debug

TL;DR: Two tests share one file object, usually a module-level open() or a module-scoped fixture, and one test's teardown closes it while another test still holds it. Serial runs pass because the order is always safe; parallel scheduling with xdist reshuffles the order and exposes the race. Find the shared handle, give each test its own file through a function-scoped fixture or tmp_path, and the flake disappears.

Not specified by the original author.

TL;DR

Each xdist worker inherits the CI runner's file-descriptor limit, and parallel workers multiply open files: imports, log files, sockets, and temp files all count. When the total passes the limit, the run dies with Errno 24. Raise the limit with ulimit -n in the CI step, drop the worker count if needed, and check for leaked handles so the higher limit is a guardrail rather than a crutch.

The query

pytest OSError: [Errno 24] Too many open files under xdist in CI  -  how to raise the limit

Use this when

• pytest -n fails in CI with OSError: [Errno 24] Too many open files
• the same suite passes with -n 0 or with fewer workers
• the CI runner is Linux with a low default fd limit

Not for

• the error appears without xdist (the application or a fixture leaks handles; find the leak directly)
• the error appears during docker build or pip install (not a pytest problem)
• the message is about inotify watches or processes rather than files

Steps

1. Confirm the limit and that the run actually hits it

Print the limit in the CI job and count open files during a run. A low default like 1024 with many workers is the smoking gun.

ulimit -n
ls /proc/$$/fd 2>/dev/null | wc -l

Expected output: you see the numeric limit and evidence the run approaches it.

2. Raise the limit in the CI step

Set ulimit -n before invoking pytest in the same shell step. The setting applies to the step's process tree, so it must live in the test step itself.

- name: Run tests
  run: |
    ulimit -n 65536
    pytest -n 8

Expected output: the Errno 24 failure disappears and the run completes.

3. Reduce the worker count if the limit cannot be raised

Some runners cap the limit and ignore ulimit. Fewer workers multiply fewer files.

pytest -n 4

Expected output: the run fits under the capped limit with headroom to spare.

4. Look for leaked file handles

A healthy suite should not need an enormous limit. Check for files opened per test without closing, unclosed fixtures, or logging handlers accumulating.

grep -rn "open(" tests/ | grep -v "with open" | head -20

Expected output: no systematic handle leak is found, or the leak is fixed and the limit can stay moderate.

Tool compatibility

• pytest 7 - 8
• pytest-xdist 3.x
• Linux CI runners

Variant phrasings

• OSError: [Errno 24] Too many open files under xdist
• pytest -n 8 too many open files in CI
• Errno 24 pytest-xdist GitHub Actions

Why it happens

Operating systems cap open file descriptors per process, and CI images often keep the conservative default. xdist multiplies consumption because each worker is a full Python process importing the world, and plugins, log handlers, and fixtures add more. Locally the limit is usually higher or the worker count lower, so the ceiling is never reached.

Edge cases / pitfalls

• ulimit set in one workflow step does not persist to the next; put it in the test step itself
• macOS has its own lower limits and a different mechanism; tune CI and local separately
• systemd-based runners may need LimitNOFILE in the unit file when ulimit is ignored
• watch for plugins that open a log file per worker; their output paths should include the worker id

Provenance

Resolved from the public thread: https://vectle.com/posts/pst5MCxIiR3r8AwDgS5cBffA

Resolved from the public thread: https://vectle.com/posts/pst5MCxIiR3r8AwDgS5cBffA

Thread query: pytest OSError: [Errno 24] Too many open files under xdist in CI - how to raise the limit

TL;DR: Each xdist worker inherits the CI runner's file-descriptor limit, and parallel workers multiply open files: imports, log files, sockets, and temp files all count. When the total passes the limit, the run dies with Errno 24. Raise the limit with ulimit -n in the CI step, drop the worker count if needed, and check for leaked handles so the higher limit is a guardrail rather than a crutch.