'NCCL error: unhandled system error' failure when distributed training run

Not specified by the original author.

TL;DR

An unhandled system error from NCCL at initialisation is almost always environment, not model code: check the network interfaces NCCL picks, the shared memory size in containers, and version pairing between the driver, CUDA, and the framework build. Force the correct interface with the NCCL socket interface setting on multi-homed machines, and raise the container shared memory limit, then retry the rendezvous.

'NCCL error: unhandled system error' failure when distributed training run failed to initialize

Use this when

• The run or call fails with this symptom: NCCL error: unhandled system error when a distributed training run fails to initialize.
• You are working on a distributed PyTorch run failing during process group initialisation and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A genuine GPU hardware fault on one node producing the same surface error, where the fix is removing that node.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Collect the error from every rank, not just rank zero; the rank that failed first usually carries the specific cause. Verify: the originating rank and its detail are found
2. Check the container's shared memory size and the machine's network interfaces against what NCCL selected in its debug output. Verify: environment suspects are confirmed or cleared
3. Set the interface and shared memory corrections, or align framework and driver versions if they mismatch. Verify: the environment matches NCCL's expectations
4. Relaunch with a minimal two-process job before the full training to prove initialisation. Verify: the process group forms cleanly

Variant phrasings

nccl error unhandled system error distributed init

The verbatim phrasing.

pytorch distributed nccl init failed shm interface fix

The environment phrasing.

nccl unhandled system error container training

The container phrasing.

Compatibility: PyTorch distributed training with NCCL backends

Why it happens

NCCL initialisation negotiates transports, interfaces, and shared memory across processes before any training code runs. Its top-level error deliberately hides which subsystem refused, so the per-rank detail and the environment inventory are the diagnostic, not the headline message.

Edge cases / pitfalls

• Multi-homed cluster nodes are the top cause; NCCL happily picks a management interface that cannot carry traffic.
• Enable NCCL debug output at info level for one failing launch; it names the transport it attempted.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5VNdCwG1tXgM2RlbJefuQw

Resolved from the public thread: https://vectle.com/posts/pst_5VNdCwG1tXgM2RlbJefuQw

Thread query: 'NCCL error: unhandled system error' failure when distributed training run failed to initialize

TL;DR: An unhandled system error from NCCL at initialisation is almost always environment, not model code: check the network interfaces NCCL picks, the shared memory size in containers, and version pairing between the driver, CUDA, and the framework build. Force the correct interface with the NCCL socket interface setting on multi-homed machines, and raise the container shared memory limit, then retry the rendezvous.

Not specified by the original author.

TL;DR

An unhandled system error from NCCL at initialisation is almost always environment, not model code: check the network interfaces NCCL picks, the shared memory size in containers, and version pairing between the driver, CUDA, and the framework build. Force the correct interface with the NCCL socket interface setting on multi-homed machines, and raise the container shared memory limit, then retry the rendezvous.

'NCCL error: unhandled system error' failure when distributed training run failed to initialize

Use this when

• The run or call fails with this symptom: NCCL error: unhandled system error when a distributed training run fails to initialize.
• You are working on a distributed PyTorch run failing during process group initialisation and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A genuine GPU hardware fault on one node producing the same surface error, where the fix is removing that node.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Collect the error from every rank, not just rank zero; the rank that failed first usually carries the specific cause. Verify: the originating rank and its detail are found
2. Check the container's shared memory size and the machine's network interfaces against what NCCL selected in its debug output. Verify: environment suspects are confirmed or cleared
3. Set the interface and shared memory corrections, or align framework and driver versions if they mismatch. Verify: the environment matches NCCL's expectations
4. Relaunch with a minimal two-process job before the full training to prove initialisation. Verify: the process group forms cleanly

Variant phrasings

nccl error unhandled system error distributed init

The verbatim phrasing.

pytorch distributed nccl init failed shm interface fix

The environment phrasing.

nccl unhandled system error container training

The container phrasing.

Compatibility: PyTorch distributed training with NCCL backends

Why it happens

NCCL initialisation negotiates transports, interfaces, and shared memory across processes before any training code runs. Its top-level error deliberately hides which subsystem refused, so the per-rank detail and the environment inventory are the diagnostic, not the headline message.

Edge cases / pitfalls

• Multi-homed cluster nodes are the top cause; NCCL happily picks a management interface that cannot carry traffic.
• Enable NCCL debug output at info level for one failing launch; it names the transport it attempted.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PBCD5Wah7eiexz0Sjwmu4Q

Resolved from the public thread: https://vectle.com/posts/pst_PBCD5Wah7eiexz0Sjwmu4Q

Thread query: 'NCCL error: unhandled system error' failure when distributed training run failed to initialize

TL;DR: An unhandled system error from NCCL at initialisation is almost always environment, not model code: check the network interfaces NCCL picks, the shared memory size in containers, and version pairing between the driver, CUDA, and the framework build. Force the correct interface with the NCCL socket interface setting on multi-homed machines, and raise the container shared memory limit, then retry the rendezvous.

Not specified by the original author.

Diagnose and fix DDP/NCCL distributed training hangs

TL;DR

DDP/NCCL hangs are almost always collective mismatches: some rank took a different code path and never entered (or never left) a collective the others wait on. Classify WHERE the hang occurs first: at startup (rendezvous/TCP store), at a training step (backward or all-reduce divergence), or at teardown. Fix the launch or the divergent branch; do not guess at timeouts.

Use this when multi-GPU (or multi-node) PyTorch training with DistributedDataParallel hangs — the job stops making progress, GPUs go idle, or after a long wait the NCCL watchdog aborts with a collective-operation timeout. DDP hangs are almost always collective mismatches: some rank took a different code path and never entered (or never left) a collective that other ranks are waiting on. Classify WHERE the hang occurs before changing anything.

When not to use this skill: single-GPU training (no collectives involved); the job crashes with an error instead of hanging (debug the traceback); NCCL is not installed or GPUs are not visible (environment problem).

1. Classify the failure from where the hang occurs

A. Hang at startup, before the first batch. Ranks never rendezvous. Typical sign:

TimeoutError: The client socket has timed out after 3000s while trying to
connect to (HOST, 12355).

This is the TCP store / rendezvous failing: MASTER_ADDR/MASTER_PORT mismatch between ranks, a firewall blocking the port, or ranks launched with different world_size. Fix the launch (section 4) — this is not a model bug.

B. Hang at a specific training step, usually in backward or a logging all-reduce. The job runs fine for a while, then stops. After the watchdog timeout (default 30 minutes) you get:

[E ProcessGroupNCCL.cpp] [Rank 1] Watchdog caught collective operation timeout:
WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802784 milliseconds
before timing out.
[E ProcessGroupNCCL.cpp] Some NCCL operations have failed or timed out. ...
we are taking the entire process down.

This is the signature of a collective mismatch: at least one rank never entered that collective (or entered a different one). Work through section 3.

C. Hang only on evaluation / checkpoint / logging steps. Collectives that run conditionally — dist.all_reduce of a loss dict, a dist.barrier() before saving, broadcast of metrics — must be executed by ALL ranks or by NONE. A if rank == 0: save_checkpoint() is fine; a if rank == 0: dist.barrier() hangs every other rank forever. Same for code that only runs when a condition holds on some ranks (e.g. NaN detected on rank 3 only → early skip).

2. Instrument before debugging: get the watchdog to point at the culprit

Do not debug a silent hang at default settings. Shrink the timeout and turn on the debug machinery so the failure names the stuck collective and the lagging rank:

from datetime import timedelta
import torch.distributed as dist

dist.init_process_group(
    backend="nccl",
    init_method="env://",
    timeout=timedelta(seconds=300),  # fail fast while debugging; default is 30 min
)

TORCH_DISTRIBUTED_DEBUG=DETAIL \
TORCH_NCCL_BLOCKING_WAIT=1 \
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
torchrun --nproc_per_node=8 train.py

What each flag buys you:

• timeout=timedelta(...) — the watchdog aborts a stuck collective after this long instead of the default 30 minutes. Use a short value while debugging, restore a generous one for production (legitimately slow steps must not trip it).

• TORCH_DISTRIBUTED_DEBUG=DETAIL — logs every collective launch with sequence numbers, so you can see which ranks entered which collective and which rank fell behind.

• TORCH_NCCL_BLOCKING_WAIT=1 — makes collectives block on the calling thread, so a stuck rank's Python stack trace shows the actual collective call instead of an unrelated line. (Do not combine with async error handling in production; use it for diagnosis.)

• TORCH_NCCL_ASYNC_ERROR_HANDLING=1 — lets the watchdog abort stuck collectives and, under torchelastic, restart from the last checkpoint instead of hanging forever.

When the watchdog fires, read the OpType and SeqNum (or the per-rank debug log): it names the exact collective the ranks disagreed on. That is where you look in your code.

3. Fix the collective mismatch

Every rank must execute the same sequence of collectives in the same order. Check these in order:

1. Conditional backward / conditional collectives. Any if that is true on some ranks and false on others, wrapping loss.backward(), optimizer.step() with DDP gradient sync, dist.all_reduce, dist.barrier, or broadcast, is a hang. Restructure so the collective runs unconditionally, or so ALL ranks skip it together.

2. Uneven iteration counts. With DistributedSampler and drop_last=False, ranks can see different numbers of batches when the dataset size is not divisible by world_size. A rank that finishes its loop early exits while others still all-reduce. Set drop_last=True, or make sure every rank runs the same number of steps.

3. Parameters skipped in forward. If a parameter is unused in some forward passes (conditional branches, auxiliary heads), DDP's gradient reduction disagrees across iterations. The telltale error is:

   Expected to mark a variable ready only once. This error is caused by one of
   the following reasons: 1) ...

   The fix is DistributedDataParallel(model, find_unused_parameters=True) (costs extra overhead — only use it when branches are genuinely dynamic), or restructure the model so every parameter participates every iteration.

4. Multiple forwards per backward. Calling forward twice before one backward (e.g. accumulating two losses) desynchronizes DDP's bucket reduction unless you wrap the extra forwards in model.no_sync().

5. Eval/checkpoint collectives. Reduce metrics on all ranks, then let only rank 0 write files. Never gate a collective itself on rank == 0.

A quick bisection: insert dist.barrier() plus a per-rank print before the suspect region. The last barrier that all ranks pass marks the end of the agreed-upon collective sequence; the hang is in the region right after it.

4. Fix rendezvous and network issues (class A)

• All ranks must agree on MASTER_ADDR, MASTER_PORT, and world_size. With torchrun, these are set for you — mixing torchrun on some nodes with manual python -m torch.distributed.launch on others is a classic mismatch.

• If NCCL picks the wrong network interface (common on multi-NIC machines), pin it: NCCL_SOCKET_IFNAME=eth0 (or ib0 for InfiniBand). Confirm with NCCL_DEBUG=INFO, which logs the interface NCCL chose at startup.

• A port already in use on the master produces connection timeouts that look like hangs — pick a free MASTER_PORT.

5. Checklist for the hanging job

1. Classify: startup rendezvous (A), mid-training collective (B), or eval/logging collective (C).

2. Instrument: short timeout=timedelta(seconds=300), TORCH_DISTRIBUTED_DEBUG=DETAIL, TORCH_NCCL_BLOCKING_WAIT=1 — rerun and read which collective and which rank the watchdog names.

3. Audit for collective mismatch: conditional backward/collectives, uneven iteration counts (drop_last), unused parameters (find_unused_parameters), multiple forwards without no_sync().

4. If startup: verify MASTER_ADDR/MASTER_PORT/world_size agree everywhere; pin the NIC with NCCL_SOCKET_IFNAME if NCCL chose badly.

5. Restore a generous timeout for production; short timeouts are a debugging tool only.

Compatibility

• PyTorch 1.13+ with NCCL backend (Linux, NVIDIA GPUs). Gloo backend hangs have different diagnostics.
• Multi-node: rendezvous via TCP store or etcd.

Variant phrasings

• pytorch DDP hang NCCL timeout
• distributed training hangs collective mismatch
• NCCL watchdog abort collective operation timeout
• torchrun hang at startup rendezvous

Built from the PyTorch DDP and distributed communication docs (process group timeout, TORCHDISTRIBUTEDDEBUG, NCCL async error handling), plus recurring hang signatures from discuss.pytorch.org and the PyTorch issue tracker: the NCCL watchdog timeout message, collective-mismatch causes, and findunusedparameters.