# Diagnose and fix PyTorch DataLoader worker crashes and stalls

Classify a PyTorch DataLoader failure from its error signature (bus error, worker exited unexpectedly, ancdata/Too many open files, CUDA re-init, hang) and apply the matching fix: raise /dev/shm, run num_workers=0 to surface the real traceback, switch the multiprocessing sharing strategy, move CUDA out of workers, or fix fork-unsafe file handles.

Exact reference: {"kind":"skill_version","skill_id":"skl_1ZIeCoRgNrTWndj3XX2Jvw","version_id":"skv_dgQ9TRmQns-y4TEC5x0c1A"}

Applicability: [{"constraint":">=1.7 for persistent_workers; earlier versions otherwise","technology":"PyTorch","version_scheme":"semver"}]

# Diagnose and fix PyTorch DataLoader worker crashes and stalls

Use this when a PyTorch training loop dies with a DataLoader worker error, or when
training silently stalls while workers seem idle. DataLoader failures are almost
always one of five distinct problems with different fixes, and the error message
or failure signature tells you which one before you change any code.

## 1. Classify the failure from observable evidence

Match the failure to one of these before changing anything:

**A. Bus error / shared memory.** The error says:

```
RuntimeError: DataLoader worker (pid 363) is killed by signal: Bus error. It is
possible that dataloader's workers are out of shared memory. Please try to raise
your shared memory limit.
```

or:

```
ERROR: Unexpected bus error encountered in worker. This might be caused by
insufficient shared memory (shm).
```

Batches travel from workers to the main process through shared-memory tensors
backed by `/dev/shm`. When `/dev/shm` is too small (Docker containers default to
**64 MB**), a large batch kills the worker with SIGBUS. Fix with section 2.

**B. Worker "exited unexpectedly" with no clear cause.** The error says:

```
RuntimeError: DataLoader worker (pid(s) 1234) exited unexpectedly
```

This message means a worker died; the root cause is elsewhere. Check in this
order:
1. **OOM killer.** The worker ran out of host RAM (each worker holds
   `prefetch_factor` batches plus the dataset worker's own memory). Check the
   kernel log for the killer: `dmesg | grep -i -E 'oom|killed process'`.
2. **Exception in `__getitem__`.** Anything raised inside your dataset's
   `__getitem__` (a bad file, a library that isn't fork-safe) kills the worker
   and surfaces as this message. Reproduce it single-process first with
   `num_workers=0`; the real traceback appears only there.

**C. "received 0 items of ancdata" or "Too many open files".** The errors look
like:

```
RuntimeError: received 0 items of ancdata
```

```
OSError: [Errno 24] Too many open files
```

With the default `file_descriptor` sharing strategy, every tensor sent between
processes consumes a file descriptor, and workers also inherit open files from
the forked parent. Fix with section 4.

**D. "Cannot re-initialize CUDA in forked subprocess".** A worker calls CUDA
(e.g. `.to('cuda')` inside `__getitem__`, or a CUDA tensor created before the
fork). Forked processes cannot re-initialize CUDA. Move all CUDA work out of
workers, or pass `multiprocessing_context='spawn'` to the DataLoader. Note that
CUDA tensors should be created in the main process; workers moving data with
`pin_memory=True` is the supported pattern.

**E. Hang with no error.** The loop stops advancing and GPUs go idle. Usual
causes:
- A worker blocks on a lock: the classic case is an HDF5/lmdb file handle opened
  in the parent and shared into workers. HDF5 file handles are not fork-safe —
  open the file inside `__getitem__` (or in `worker_init_fn`) per worker, never
  share one across workers.
- `persistent_workers=False` (the default) respawns workers every epoch; combined
  with a slow dataset `__init__`, epoch starts look like a hang. Set
  `persistent_workers=True` (requires `num_workers > 0`, available since
  PyTorch 1.7) to keep workers alive across epochs.

## 2. Fix the shared-memory (bus error) case

Check the current limit first:

```bash
df -h /dev/shm
```

Estimate demand: roughly `num_workers * prefetch_factor * batch_bytes` must fit.
`prefetch_factor` defaults to 2, so 8 workers prefetching 2 batches of 1 GB
images need on the order of 16 GB.

- **Docker:** the 64 MB default is the problem. Relaunch with a larger tmpfs,
  e.g. `--shm-size=8g` (or `shm_size: '8gb'` in docker-compose).
- **Kubernetes:** the default 64 MB `/dev/shm` comes from containerd. Request an
  `emptyDir` with `medium: Memory` mounted at `/dev/shm`, sized to the estimate.
- **No control over the environment:** reduce demand instead — lower
  `num_workers`, lower `prefetch_factor` (e.g. `prefetch_factor=1`), or move the
  heaviest transforms so batches are smaller when they cross the process
  boundary.

`num_workers=0` also "fixes" it by removing shared memory entirely, but it is a
diagnostic step, not a fix: it serializes data loading into the training process.

## 3. Isolate whether workers are the problem at all

Run the same loop with `num_workers=0`:
- **Passes:** the bug is in multiprocessing (shm, fds, fork-safety, worker OOM).
  Work through sections 2, 4, 5.
- **Fails:** the bug is in your dataset or transform code. Fix it there; worker
  settings were never the issue.

Also try `num_workers=1` before `num_workers=N`: it keeps multiprocessing on
while making per-worker accounting (memory, file handles) trivial to read.

## 4. Fix file-descriptor exhaustion

On Linux the default sharing strategy is `file_descriptor`; on macOS and
Windows it is `file_system`. When descriptors run out, switch strategies at the
top of the program (before any DataLoader or tensor sharing):

```python
import torch.multiprocessing as mp
mp.set_sharing_strategy('file_system')
```

This routes shared tensors through files in `/dev/shm` instead of descriptors.
Also raise the process limit (`ulimit -n 65536` in the shell, or the Docker
`--ulimit nofile=65536:65536` flag) and close files you open in the parent
before the DataLoader forks. `lsof -p [pid] | wc -l` on a worker shows the live
descriptor count when you need ground truth.

## 5. Checklist for the failing command

1. Reproduce and classify (A–E) from the error text or hang signature.
2. If bus error: `df -h /dev/shm`, compare against
   `num_workers * prefetch_factor * batch_bytes`, raise `/dev/shm` or reduce
   demand.
3. If "exited unexpectedly": run `num_workers=0` to get the real traceback;
   check `dmesg` for the OOM killer.
4. If ancdata / "Too many open files": `set_sharing_strategy('file_system')`,
   raise `ulimit -n`, reduce inherited open files.
5. If CUDA in a worker: move CUDA out of `__getitem__`, or use
   `multiprocessing_context='spawn'`.
6. If hanging: check for a shared file handle/lock across workers (open data
   files per worker in `worker_init_fn`); consider `persistent_workers=True`.
7. `num_workers=0` is a diagnostic, not a fix — confirm the root cause before
   settling on worker settings.


## Supporting basis and limitations

Built from PyTorch DataLoader and torch.multiprocessing documentation, plus recurring failure signatures from discuss.pytorch.org and the PyTorch issue tracker: the /dev/shm bus-error message, the 64 MB Docker shm default, file-descriptor sharing strategies, and the fork/CUDA restriction.

## Change and rationale

New skill: diagnose and fix PyTorch DataLoader worker crashes and stalls.

DataLoader worker failures (bus errors, 'exited unexpectedly', ancdata errors) are among the most-asked PyTorch forum questions, and the standard advice of 'set num_workers=0' masks the root cause. This skill adds a decision procedure that classifies the failure from the error text or hang signature before prescribing the fix.