pytest OSError: [Errno 24] Too many open files under xdist in CI - how to raise the limit

TL;DR

Each xdist worker inherits the CI runner's file-descriptor limit, and parallel workers multiply open files: imports, log files, sockets, and temp files all count. When the total passes the limit, the run dies with Errno 24. Raise the limit with ulimit -n in the CI step, drop the worker count if needed, and check for leaked handles so the higher limit is a guardrail rather than a crutch.

The query

pytest OSError: [Errno 24] Too many open files under xdist in CI  -  how to raise the limit

Use this when

• pytest -n fails in CI with OSError: [Errno 24] Too many open files
• the same suite passes with -n 0 or with fewer workers
• the CI runner is Linux with a low default fd limit

Not for

• the error appears without xdist (the application or a fixture leaks handles; find the leak directly)
• the error appears during docker build or pip install (not a pytest problem)
• the message is about inotify watches or processes rather than files

Steps

1. Confirm the limit and that the run actually hits it

Print the limit in the CI job and count open files during a run. A low default like 1024 with many workers is the smoking gun.

ulimit -n
ls /proc/$$/fd 2>/dev/null | wc -l

Expected output: you see the numeric limit and evidence the run approaches it.

2. Raise the limit in the CI step

Set ulimit -n before invoking pytest in the same shell step. The setting applies to the step's process tree, so it must live in the test step itself.

- name: Run tests
  run: |
    ulimit -n 65536
    pytest -n 8

Expected output: the Errno 24 failure disappears and the run completes.

3. Reduce the worker count if the limit cannot be raised

Some runners cap the limit and ignore ulimit. Fewer workers multiply fewer files.

pytest -n 4

Expected output: the run fits under the capped limit with headroom to spare.

4. Look for leaked file handles

A healthy suite should not need an enormous limit. Check for files opened per test without closing, unclosed fixtures, or logging handlers accumulating.

grep -rn "open(" tests/ | grep -v "with open" | head -20

Expected output: no systematic handle leak is found, or the leak is fixed and the limit can stay moderate.

Tool compatibility

• pytest 7 - 8
• pytest-xdist 3.x
• Linux CI runners

Variant phrasings

• OSError: [Errno 24] Too many open files under xdist
• pytest -n 8 too many open files in CI
• Errno 24 pytest-xdist GitHub Actions

Why it happens

Operating systems cap open file descriptors per process, and CI images often keep the conservative default. xdist multiplies consumption because each worker is a full Python process importing the world, and plugins, log handlers, and fixtures add more. Locally the limit is usually higher or the worker count lower, so the ceiling is never reached.

Edge cases / pitfalls

• ulimit set in one workflow step does not persist to the next; put it in the test step itself
• macOS has its own lower limits and a different mechanism; tune CI and local separately
• systemd-based runners may need LimitNOFILE in the unit file when ulimit is ignored
• watch for plugins that open a log file per worker; their output paths should include the worker id

Provenance

Resolved from the public thread: https://vectle.com/posts/pst5MCxIiR3r8AwDgS5cBffA