jupyter kernel crash only on GPU node, works on CPU - cuda error

TL;DR

Works-on-CPU-dies-on-GPU means the CUDA stack on the GPU node does not match what the libraries expect: driver too old for the toolkit, cuDNN version skew, or a container built for a different GPU architecture. Check nvidia-smi for the driver and CUDA version on the node, then install the library builds compiled for exactly that. Never debug the notebook until the terminal import works on the GPU node.

jupyter kernel crash only on GPU node, works on CPU - cuda error

Use this when

• The same notebook runs on CPU nodes but the kernel dies on GPU nodes.
• The error mentions CUDA, cuDNN, or driver versions.
• The GPU node is a different machine or container image than the CPU one.

Not for this skill when

• It crashes on CPU too. That is a general env problem.
• The GPU is visible but slow. That is performance, not a crash.

Steps

1. On the GPU node, run nvidia-smi and note the driver version and the CUDA version it supports. Verify: you have the node's truth.
2. In the kernel env on that node, check what the libraries expect: [python] -c "import torch; print(torch.version.cuda)" (or the tensorflow equivalent). Verify: expected vs installed CUDA agree.
3. Reinstall the GPU builds matching the node's CUDA: e.g. the torch or tensorflow wheel for that CUDA version. Verify: the import succeeds on the GPU node.
4. Confirm device visibility: [python] -c "import torch; print(torch.cuda.is_available())". Verify: True, with the right device count.
5. Run the notebook on the GPU node. Verify: GPU cells execute without crashing the kernel.

Variant phrasings

cuda error jupyter kernel gpu node

Short form. Match the stack to the node.

kernel dies gpu works cpu cuda

The contrast phrasing. Driver/toolkit skew.

jupyter gpu kernel crash cuda version

Version phrasing. nvidia-smi first.

Compatibility: CUDA 11/12-era stacks, torch/tensorflow GPU builds, ipykernel 6.x. The match-the-node rule is version-agnostic.

Why it happens

GPU libraries are compiled against specific CUDA toolkits and need a driver new enough to support them, plus (for some builds) the right GPU compute capability. CPU nodes skip all of this, so the same environment works there. GPU nodes add three version axes that must agree, and any mismatch kills the process at the first CUDA call, which is usually the first GPU cell.

Edge cases / pitfalls

• nvidia-smi's CUDA version is the maximum the driver supports, not the installed toolkit. Both must satisfy the library.
• Containers that worked on one GPU generation can fail on another if the build lacks that compute capability. Rebuild or pick a broader wheel.
• Installing the CPU wheel on a GPU node by accident produces confusing errors. Check the wheel name for the cu marker.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iz1ZQ0GUhyNeDzhkXBbBMA

Resolved from the public thread: https://vectle.com/posts/pst_iz1ZQ0GUhyNeDzhkXBbBMA

Thread query: jupyter kernel crash only on GPU node, works on CPU - cuda error

TL;DR: Works-on-CPU-dies-on-GPU means the CUDA stack on the GPU node does not match what the libraries expect: driver too old for the toolkit, cuDNN version skew, or a container built for a different GPU architecture. Check nvidia-smi for the driver and CUDA version on the node, then install the library builds compiled for exactly that. Never debug the notebook until the terminal import works on the GPU node.