# Diagnose and fix DDP/NCCL distributed training hangs
Use this when multi-GPU (or multi-node) PyTorch training with
`DistributedDataParallel` hangs — the job stops making progress, GPUs go idle,
or after a long wait the NCCL watchdog aborts with a collective-operation
timeout. DDP hangs are almost always collective mismatches: some rank took a
different code path and never entered (or never left) a collective that other
ranks are waiting on. Classify WHERE the hang occurs before changing anything.
## 1. Classify the failure from where the hang occurs
**A. Hang at startup, before the first batch.** Ranks never rendezvous. Typical
sign:
```
TimeoutError: The client socket has timed out after 3000s while trying to
connect to (HOST, 12355).
```
This is the TCP store / rendezvous failing: `MASTER_ADDR`/`MASTER_PORT`
mismatch between ranks, a firewall blocking the port, or ranks launched with
different `world_size`. Fix the launch (section 4) — this is not a model bug.
**B. Hang at a specific training step, usually in backward or a logging
all-reduce.** The job runs fine for a while, then stops. After the watchdog
timeout (default 30 minutes) you get:
```
[E ProcessGroupNCCL.cpp] [Rank 1] Watchdog caught collective operation timeout:
WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802784 milliseconds
before timing out.
[E ProcessGroupNCCL.cpp] Some NCCL operations have failed or timed out. ...
we are taking the entire process down.
```
This is the signature of a collective mismatch: at least one rank never entered
that collective (or entered a different one). Work through section 3.
**C. Hang only on evaluation / checkpoint / logging steps.** Collectives that
run conditionally — `dist.all_reduce` of a loss dict, a `dist.barrier()` before
saving, broadcast of metrics — must be executed by ALL ranks or by NONE. A
`if rank == 0: save_checkpoint()` is fine; a `if rank == 0: dist.barrier()`
hangs every other rank forever. Same for code that only runs when a condition
holds on some ranks (e.g. NaN detected on rank 3 only → early skip).
## 2. Instrument before debugging: get the watchdog to point at the culprit
Do not debug a silent hang at default settings. Shrink the timeout and turn on
the debug machinery so the failure names the stuck collective and the lagging
rank:
```python
from datetime import timedelta
import torch.distributed as dist
dist.init_process_group(
backend="nccl",
init_method="env://",
timeout=timedelta(seconds=300), # fail fast while debugging; default is 30 min
)
```
```bash
TORCH_DISTRIBUTED_DEBUG=DETAIL \
TORCH_NCCL_BLOCKING_WAIT=1 \
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
torchrun --nproc_per_node=8 train.py
```
What each flag buys you:
- `timeout=timedelta(...)` — the watchdog aborts a stuck collective after this
long instead of the default 30 minutes. Use a short value while debugging,
restore a generous one for production (legitimately slow steps must not trip
it).
- `TORCH_DISTRIBUTED_DEBUG=DETAIL` — logs every collective launch with
sequence numbers, so you can see which ranks entered which collective and
which rank fell behind.
- `TORCH_NCCL_BLOCKING_WAIT=1` — makes collectives block on the calling thread,
so a stuck rank's Python stack trace shows the actual collective call instead
of an unrelated line. (Do not combine with async error handling in
production; use it for diagnosis.)
- `TORCH_NCCL_ASYNC_ERROR_HANDLING=1` — lets the watchdog abort stuck
collectives and, under torchelastic, restart from the last checkpoint instead
of hanging forever.
When the watchdog fires, read the `OpType` and `SeqNum` (or the per-rank debug
log): it names the exact collective the ranks disagreed on. That is where you
look in your code.
## 3. Fix the collective mismatch
Every rank must execute the same sequence of collectives in the same order.
Check these in order:
1. **Conditional backward / conditional collectives.** Any `if` that is true on
some ranks and false on others, wrapping `loss.backward()`, `optimizer.step()`
with DDP gradient sync, `dist.all_reduce`, `dist.barrier`, or broadcast, is a
hang. Restructure so the collective runs unconditionally, or so ALL ranks
skip it together.
2. **Uneven iteration counts.** With `DistributedSampler` and
`drop_last=False`, ranks can see different numbers of batches when the
dataset size is not divisible by `world_size`. A rank that finishes its loop
early exits while others still all-reduce. Set `drop_last=True`, or make
sure every rank runs the same number of steps.
3. **Parameters skipped in forward.** If a parameter is unused in some forward
passes (conditional branches, auxiliary heads), DDP's gradient reduction
disagrees across iterations. The telltale error is:
```
Expected to mark a variable ready only once. This error is caused by one of
the following reasons: 1) ...
```
The fix is `DistributedDataParallel(model, find_unused_parameters=True)`
(costs extra overhead — only use it when branches are genuinely dynamic), or
restructure the model so every parameter participates every iteration.
4. **Multiple forwards per backward.** Calling forward twice before one
backward (e.g. accumulating two losses) desynchronizes DDP's bucket
reduction unless you wrap the extra forwards in `model.no_sync()`.
5. **Eval/checkpoint collectives.** Reduce metrics on all ranks, then let only
rank 0 write files. Never gate a collective itself on `rank == 0`.
A quick bisection: insert `dist.barrier()` plus a per-rank print before the
suspect region. The last barrier that all ranks pass marks the end of the
agreed-upon collective sequence; the hang is in the region right after it.
## 4. Fix rendezvous and network issues (class A)
- All ranks must agree on `MASTER_ADDR`, `MASTER_PORT`, and `world_size`. With
`torchrun`, these are set for you — mixing `torchrun` on some nodes with
manual `python -m torch.distributed.launch` on others is a classic mismatch.
- If NCCL picks the wrong network interface (common on multi-NIC machines),
pin it: `NCCL_SOCKET_IFNAME=eth0` (or `ib0` for InfiniBand). Confirm with
`NCCL_DEBUG=INFO`, which logs the interface NCCL chose at startup.
- A port already in use on the master produces connection timeouts that look
like hangs — pick a free `MASTER_PORT`.
## 5. Checklist for the hanging job
1. Classify: startup rendezvous (A), mid-training collective (B), or
eval/logging collective (C).
2. Instrument: short `timeout=timedelta(seconds=300)`,
`TORCH_DISTRIBUTED_DEBUG=DETAIL`, `TORCH_NCCL_BLOCKING_WAIT=1` — rerun and
read which collective and which rank the watchdog names.
3. Audit for collective mismatch: conditional backward/collectives, uneven
iteration counts (`drop_last`), unused parameters (`find_unused_parameters`),
multiple forwards without `no_sync()`.
4. If startup: verify `MASTER_ADDR`/`MASTER_PORT`/`world_size` agree everywhere;
pin the NIC with `NCCL_SOCKET_IFNAME` if NCCL chose badly.
5. Restore a generous timeout for production; short timeouts are a debugging
tool only.