NCCL error: unhandled system error' failure when distributed training run failed to initialize

TL;DR

An unhandled system error from NCCL at initialisation is almost always environment, not model code: check the network interfaces NCCL picks, the shared memory size in containers, and version pairing between the driver, CUDA, and the framework build. Force the correct interface with the NCCL socket interface setting on multi-homed machines, and raise the container shared memory limit, then retry the rendezvous.

'NCCL error: unhandled system error' failure when distributed training run failed to initialize

Use this when

• The run or call fails with this symptom: NCCL error: unhandled system error when a distributed training run fails to initialize.
• You are working on a distributed PyTorch run failing during process group initialisation and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A genuine GPU hardware fault on one node producing the same surface error, where the fix is removing that node.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Collect the error from every rank, not just rank zero; the rank that failed first usually carries the specific cause. Verify: the originating rank and its detail are found
2. Check the container's shared memory size and the machine's network interfaces against what NCCL selected in its debug output. Verify: environment suspects are confirmed or cleared
3. Set the interface and shared memory corrections, or align framework and driver versions if they mismatch. Verify: the environment matches NCCL's expectations
4. Relaunch with a minimal two-process job before the full training to prove initialisation. Verify: the process group forms cleanly

Variant phrasings

nccl error unhandled system error distributed init

The verbatim phrasing.

pytorch distributed nccl init failed shm interface fix

The environment phrasing.

nccl unhandled system error container training

The container phrasing.

Compatibility: PyTorch distributed training with NCCL backends

Why it happens

NCCL initialisation negotiates transports, interfaces, and shared memory across processes before any training code runs. Its top-level error deliberately hides which subsystem refused, so the per-rank detail and the environment inventory are the diagnostic, not the headline message.

Edge cases / pitfalls

• Multi-homed cluster nodes are the top cause; NCCL happily picks a management interface that cannot carry traffic.
• Enable NCCL debug output at info level for one failing launch; it names the transport it attempted.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PBCD5Wah7eiexz0Sjwmu4Q