Diagnose and fix PyTorch DataLoader worker crashes and stalls

[{"constraint": ">=1.7 for persistentworkers; earlier versions otherwise", "technology": "PyTorch", "versionscheme": "semver"}]

Diagnose and fix PyTorch DataLoader worker crashes and stalls

Use this when a PyTorch training loop dies with a DataLoader worker error, or when training silently stalls while workers seem idle. DataLoader failures are almost always one of five distinct problems with different fixes, and the error message or failure signature tells you which one before you change any code.

1. Classify the failure from observable evidence

Match the failure to one of these before changing anything:

A. Bus error / shared memory. The error says:

RuntimeError: DataLoader worker (pid 363) is killed by signal: Bus error. It is
possible that dataloader's workers are out of shared memory. Please try to raise
your shared memory limit.

or:

ERROR: Unexpected bus error encountered in worker. This might be caused by
insufficient shared memory (shm).

Batches travel from workers to the main process through shared-memory tensors backed by /dev/shm. When /dev/shm is too small (Docker containers default to 64 MB), a large batch kills the worker with SIGBUS. Fix with section 2.

B. Worker "exited unexpectedly" with no clear cause. The error says:

RuntimeError: DataLoader worker (pid(s) 1234) exited unexpectedly

This message means a worker died; the root cause is elsewhere. Check in this order:

1. OOM killer. The worker ran out of host RAM (each worker holds prefetch_factor batches plus the dataset worker's own memory). Check the kernel log for the killer: dmesg | grep -i -E 'oom|killed process'.

2. Exception in __getitem__. Anything raised inside your dataset's __getitem__ (a bad file, a library that isn't fork-safe) kills the worker and surfaces as this message. Reproduce it single-process first with num_workers=0; the real traceback appears only there.

C. "received 0 items of ancdata" or "Too many open files". The errors look like:

RuntimeError: received 0 items of ancdata

OSError: [Errno 24] Too many open files

With the default file_descriptor sharing strategy, every tensor sent between processes consumes a file descriptor, and workers also inherit open files from the forked parent. Fix with section 4.

D. "Cannot re-initialize CUDA in forked subprocess". A worker calls CUDA (e.g. .to('cuda') inside __getitem__, or a CUDA tensor created before the fork). Forked processes cannot re-initialize CUDA. Move all CUDA work out of workers, or pass multiprocessing_context='spawn' to the DataLoader. Note that CUDA tensors should be created in the main process; workers moving data with pin_memory=True is the supported pattern.

E. Hang with no error. The loop stops advancing and GPUs go idle. Usual causes:

• A worker blocks on a lock: the classic case is an HDF5/lmdb file handle opened in the parent and shared into workers. HDF5 file handles are not fork-safe — open the file inside __getitem__ (or in worker_init_fn) per worker, never share one across workers.

• persistent_workers=False (the default) respawns workers every epoch; combined with a slow dataset __init__, epoch starts look like a hang. Set persistent_workers=True (requires num_workers > 0, available since PyTorch 1.7) to keep workers alive across epochs.

2. Fix the shared-memory (bus error) case

Check the current limit first:

df -h /dev/shm

Estimate demand: roughly num_workers * prefetch_factor * batch_bytes must fit. prefetch_factor defaults to 2, so 8 workers prefetching 2 batches of 1 GB images need on the order of 16 GB.

• Docker: the 64 MB default is the problem. Relaunch with a larger tmpfs, e.g. --shm-size=8g (or shm_size: '8gb' in docker-compose).

• Kubernetes: the default 64 MB /dev/shm comes from containerd. Request an emptyDir with medium: Memory mounted at /dev/shm, sized to the estimate.

• No control over the environment: reduce demand instead — lower num_workers, lower prefetch_factor (e.g. prefetch_factor=1), or move the heaviest transforms so batches are smaller when they cross the process boundary.

num_workers=0 also "fixes" it by removing shared memory entirely, but it is a diagnostic step, not a fix: it serializes data loading into the training process.

3. Isolate whether workers are the problem at all

Run the same loop with num_workers=0:

• Passes: the bug is in multiprocessing (shm, fds, fork-safety, worker OOM). Work through sections 2, 4, 5.

• Fails: the bug is in your dataset or transform code. Fix it there; worker settings were never the issue.

Also try num_workers=1 before num_workers=N: it keeps multiprocessing on while making per-worker accounting (memory, file handles) trivial to read.

4. Fix file-descriptor exhaustion

On Linux the default sharing strategy is file_descriptor; on macOS and Windows it is file_system. When descriptors run out, switch strategies at the top of the program (before any DataLoader or tensor sharing):

import torch.multiprocessing as mp
mp.set_sharing_strategy('file_system')

This routes shared tensors through files in /dev/shm instead of descriptors. Also raise the process limit (ulimit -n 65536 in the shell, or the Docker --ulimit nofile=65536:65536 flag) and close files you open in the parent before the DataLoader forks. lsof -p [pid] | wc -l on a worker shows the live descriptor count when you need ground truth.

5. Checklist for the failing command

1. Reproduce and classify (A–E) from the error text or hang signature.

2. If bus error: df -h /dev/shm, compare against num_workers * prefetch_factor * batch_bytes, raise /dev/shm or reduce demand.

3. If "exited unexpectedly": run num_workers=0 to get the real traceback; check dmesg for the OOM killer.

4. If ancdata / "Too many open files": set_sharing_strategy('file_system'), raise ulimit -n, reduce inherited open files.

5. If CUDA in a worker: move CUDA out of __getitem__, or use multiprocessing_context='spawn'.

6. If hanging: check for a shared file handle/lock across workers (open data files per worker in worker_init_fn); consider persistent_workers=True.

7. num_workers=0 is a diagnostic, not a fix — confirm the root cause before settling on worker settings.

Built from PyTorch DataLoader and torch.multiprocessing documentation, plus recurring failure signatures from discuss.pytorch.org and the PyTorch issue tracker: the /dev/shm bus-error message, the 64 MB Docker shm default, file-descriptor sharing strategies, and the fork/CUDA restriction.