# Diagnose and fix PyTorch DataLoader worker crashes and stalls
Use this when a PyTorch training loop dies with a DataLoader worker error, or when
training silently stalls while workers seem idle. DataLoader failures are almost
always one of five distinct problems with different fixes, and the error message
or failure signature tells you which one before you change any code.
## 1. Classify the failure from observable evidence
Match the failure to one of these before changing anything:
**A. Bus error / shared memory.** The error says:
```
RuntimeError: DataLoader worker (pid 363) is killed by signal: Bus error. It is
possible that dataloader's workers are out of shared memory. Please try to raise
your shared memory limit.
```
or:
```
ERROR: Unexpected bus error encountered in worker. This might be caused by
insufficient shared memory (shm).
```
Batches travel from workers to the main process through shared-memory tensors
backed by `/dev/shm`. When `/dev/shm` is too small (Docker containers default to
**64 MB**), a large batch kills the worker with SIGBUS. Fix with section 2.
**B. Worker "exited unexpectedly" with no clear cause.** The error says:
```
RuntimeError: DataLoader worker (pid(s) 1234) exited unexpectedly
```
This message means a worker died; the root cause is elsewhere. Check in this
order:
1. **OOM killer.** The worker ran out of host RAM (each worker holds
`prefetch_factor` batches plus the dataset worker's own memory). Check the
kernel log for the killer: `dmesg | grep -i -E 'oom|killed process'`.
2. **Exception in `__getitem__`.** Anything raised inside your dataset's
`__getitem__` (a bad file, a library that isn't fork-safe) kills the worker
and surfaces as this message. Reproduce it single-process first with
`num_workers=0`; the real traceback appears only there.
**C. "received 0 items of ancdata" or "Too many open files".** The errors look
like:
```
RuntimeError: received 0 items of ancdata
```
```
OSError: [Errno 24] Too many open files
```
With the default `file_descriptor` sharing strategy, every tensor sent between
processes consumes a file descriptor, and workers also inherit open files from
the forked parent. Fix with section 4.
**D. "Cannot re-initialize CUDA in forked subprocess".** A worker calls CUDA
(e.g. `.to('cuda')` inside `__getitem__`, or a CUDA tensor created before the
fork). Forked processes cannot re-initialize CUDA. Move all CUDA work out of
workers, or pass `multiprocessing_context='spawn'` to the DataLoader. Note that
CUDA tensors should be created in the main process; workers moving data with
`pin_memory=True` is the supported pattern.
**E. Hang with no error.** The loop stops advancing and GPUs go idle. Usual
causes:
- A worker blocks on a lock: the classic case is an HDF5/lmdb file handle opened
in the parent and shared into workers. HDF5 file handles are not fork-safe —
open the file inside `__getitem__` (or in `worker_init_fn`) per worker, never
share one across workers.
- `persistent_workers=False` (the default) respawns workers every epoch; combined
with a slow dataset `__init__`, epoch starts look like a hang. Set
`persistent_workers=True` (requires `num_workers > 0`, available since
PyTorch 1.7) to keep workers alive across epochs.
## 2. Fix the shared-memory (bus error) case
Check the current limit first:
```bash
df -h /dev/shm
```
Estimate demand: roughly `num_workers * prefetch_factor * batch_bytes` must fit.
`prefetch_factor` defaults to 2, so 8 workers prefetching 2 batches of 1 GB
images need on the order of 16 GB.
- **Docker:** the 64 MB default is the problem. Relaunch with a larger tmpfs,
e.g. `--shm-size=8g` (or `shm_size: '8gb'` in docker-compose).
- **Kubernetes:** the default 64 MB `/dev/shm` comes from containerd. Request an
`emptyDir` with `medium: Memory` mounted at `/dev/shm`, sized to the estimate.
- **No control over the environment:** reduce demand instead — lower
`num_workers`, lower `prefetch_factor` (e.g. `prefetch_factor=1`), or move the
heaviest transforms so batches are smaller when they cross the process
boundary.
`num_workers=0` also "fixes" it by removing shared memory entirely, but it is a
diagnostic step, not a fix: it serializes data loading into the training process.
## 3. Isolate whether workers are the problem at all
Run the same loop with `num_workers=0`:
- **Passes:** the bug is in multiprocessing (shm, fds, fork-safety, worker OOM).
Work through sections 2, 4, 5.
- **Fails:** the bug is in your dataset or transform code. Fix it there; worker
settings were never the issue.
Also try `num_workers=1` before `num_workers=N`: it keeps multiprocessing on
while making per-worker accounting (memory, file handles) trivial to read.
## 4. Fix file-descriptor exhaustion
On Linux the default sharing strategy is `file_descriptor`; on macOS and
Windows it is `file_system`. When descriptors run out, switch strategies at the
top of the program (before any DataLoader or tensor sharing):
```python
import torch.multiprocessing as mp
mp.set_sharing_strategy('file_system')
```
This routes shared tensors through files in `/dev/shm` instead of descriptors.
Also raise the process limit (`ulimit -n 65536` in the shell, or the Docker
`--ulimit nofile=65536:65536` flag) and close files you open in the parent
before the DataLoader forks. `lsof -p [pid] | wc -l` on a worker shows the live
descriptor count when you need ground truth.
## 5. Checklist for the failing command
1. Reproduce and classify (A–E) from the error text or hang signature.
2. If bus error: `df -h /dev/shm`, compare against
`num_workers * prefetch_factor * batch_bytes`, raise `/dev/shm` or reduce
demand.
3. If "exited unexpectedly": run `num_workers=0` to get the real traceback;
check `dmesg` for the OOM killer.
4. If ancdata / "Too many open files": `set_sharing_strategy('file_system')`,
raise `ulimit -n`, reduce inherited open files.
5. If CUDA in a worker: move CUDA out of `__getitem__`, or use
`multiprocessing_context='spawn'`.
6. If hanging: check for a shared file handle/lock across workers (open data
files per worker in `worker_init_fn`); consider `persistent_workers=True`.
7. `num_workers=0` is a diagnostic, not a fix — confirm the root cause before
settling on worker settings.