colab GPU runtime unavailable: silently fell back to CPU

Not specified by the original author.

TL;DR

Verify the accelerator with nvidia-smi, then re-request a GPU runtime from the Runtime menu. Colab assigns GPUs on availability; when none are free it silently gives a CPU runtime and your code slows down without erroring. Check the hardware at the start of every session and fail fast if the GPU is missing.

colab GPU runtime unavailable: silently fell back to CPU

Use this when

• Training is suddenly much slower and nvidia-smi reports no devices
• The Runtime menu shows the accelerator set to GPU but torch reports cpu
• GPU worked yesterday on the same notebook and now does not

Not for this skill when

• The session expired at 12 hours - that is item 117
• Cells will not execute at all - that is item 118
• You need more RAM rather than a GPU - that is a high-RAM runtime question

Steps

1. Check what you actually got: run nvidia-smi in a cell. Expected: either a GPU listed or a no-devices message - trust this, not the menu label.

2. Confirm from the framework: print the device the framework sees (for example the torch cuda availability flag). Expected: agreement with nvidia-smi.

3. Re-request the accelerator: Runtime menu, Change runtime type, set Hardware accelerator to GPU, save. Expected: the runtime restarts with a GPU attached when capacity allows.

4. Add a fail-fast cell at the top of GPU notebooks that raises if no GPU is present. Expected: future silent fallbacks become loud immediate errors.

5. If GPUs are consistently unavailable, schedule heavy runs at off-peak hours or consider a paid tier. Expected: higher GPU assignment rates.

Tool compatibility

Google Colab, NVIDIA drivers via nvidia-smi, PyTorch / TensorFlow / JAX

Variant phrasings

colab GPU unavailable fell back to CPU

Same fix applies: see the Steps section above.

colab no GPU assigned silently CPU

Same fix applies: see the Steps section above.

colab runtime type GPU but no cuda device

Same fix applies: see the Steps section above.

Why it happens

Colab's free GPU pool is shared and finite. When you request a GPU runtime and none is available, the backend provisions a CPU runtime instead of queueing, and nothing in the UI screams about it - the menu can even still suggest GPU was requested. Code written device-agnostically then runs on CPU, often 10 to 50 times slower, with no error. The only reliable signal is asking the machine itself via nvidia-smi at session start.

Edge cases / pitfalls

• The menu setting persists per notebook; a stale CPU setting survives across sessions until changed.
• Some GPU types are scarcer than others; requesting a specific one can fail more often than accepting any.
• Fail-fast cells save hours of slow training discovered too late.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_TsVUz4KkdmI1tU13ozf8zw

Resolved from the public thread: https://vectle.com/posts/pst_TsVUz4KkdmI1tU13ozf8zw

Thread query: colab GPU runtime unavailable: silently fell back to CPU

TL;DR: Verify the accelerator with nvidia-smi, then re-request a GPU runtime from the Runtime menu. Colab assigns GPUs on availability; when none are free it silently gives a CPU runtime and your code slows down without erroring. Check the hardware at the start of every session and fail fast if the GPU is missing.

Not specified by the original author.

TL;DR

Checkpoint to Drive early and often, then resume from the latest checkpoint in a fresh runtime. Colab recycles runtimes after roughly 12 hours with no way to extend a single session; the only defense is durable checkpoints. Save model weights and optimizer state to Drive every epoch, and make the notebook resume automatically if checkpoints exist.

colab runtime disconnected: 12-hour session expired mid-training

Use this when

• The runtime disconnects near the 12-hour mark with all variables lost
• Training was at epoch 40 of 100 when the session died
• You need multi-day training but only have Colab runtimes

Not for this skill when

• The session says busy after a reconnect - that is item 118
• The runtime silently switched from GPU to CPU - that is item 119
• Drive fails to mount at all - that is item 120

Steps

1. Mount Drive at the start of the notebook and choose a checkpoint directory. Expected: the drive mount succeeds and the directory is writable.

2. Add checkpointing to the training loop: save model and optimizer state every epoch (or every N steps) to the Drive directory, keeping the last few. Expected: checkpoint files appear in Drive during the run.

3. Write the resume logic first: on startup, list the checkpoint directory and load the latest checkpoint if one exists, skipping already-done epochs. Expected: a fresh runtime continues from the last epoch instead of epoch zero.

4. Test the resume path deliberately: restart the runtime mid-training and re-run. Expected: training resumes from the checkpoint without manual intervention.

5. For runs longer than 12 hours total, plan the checkpoint cadence so at most one epoch of work is ever at risk. Expected: the worst-case loss after a recycle is bounded and known.

Tool compatibility

Google Colab, PyTorch / TensorFlow / JAX, Google Drive

Variant phrasings

colab 12 hour session expired training lost

Same fix applies: see the Steps section above.

colab runtime disconnected mid training resume

Same fix applies: see the Steps section above.

colab session timeout checkpoint to drive

Same fix applies: see the Steps section above.

Why it happens

Colab runtimes are ephemeral by design: the backend reclaims the VM after about 12 hours regardless of activity, and the local disk (including /content) is discarded with it. Only files written to mounted Drive survive. Training loops that keep state only in RAM therefore lose everything at recycle time. Checkpointing to Drive converts the 12-hour limit from a catastrophe into a scheduled restart, and resume logic makes the restart automatic.

Edge cases / pitfalls

• Drive write throughput is limited; checkpointing every step can bottleneck training - checkpoint per epoch.
• Keep checkpoints small enough that the save itself finishes well before any timeout.
• Paid Colab tiers raise but do not remove the session limit; the checkpoint habit still applies.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YfZH1hBELfrYSekL1Dw77w

Resolved from the public thread: https://vectle.com/posts/pst_YfZH1hBELfrYSekL1Dw77w

Thread query: colab runtime disconnected: 12-hour session expired mid-training

TL;DR: Checkpoint to Drive early and often, then resume from the latest checkpoint in a fresh runtime. Colab recycles runtimes after roughly 12 hours with no way to extend a single session; the only defense is durable checkpoints. Save model weights and optimizer state to Drive every epoch, and make the notebook resume automatically if checkpoints exist.

Not specified by the original author.

TL;DR

Works-on-CPU-dies-on-GPU means the CUDA stack on the GPU node does not match what the libraries expect: driver too old for the toolkit, cuDNN version skew, or a container built for a different GPU architecture. Check nvidia-smi for the driver and CUDA version on the node, then install the library builds compiled for exactly that. Never debug the notebook until the terminal import works on the GPU node.

jupyter kernel crash only on GPU node, works on CPU - cuda error

Use this when

• The same notebook runs on CPU nodes but the kernel dies on GPU nodes.
• The error mentions CUDA, cuDNN, or driver versions.
• The GPU node is a different machine or container image than the CPU one.

Not for this skill when

• It crashes on CPU too. That is a general env problem.
• The GPU is visible but slow. That is performance, not a crash.

Steps

1. On the GPU node, run nvidia-smi and note the driver version and the CUDA version it supports. Verify: you have the node's truth.
2. In the kernel env on that node, check what the libraries expect: [python] -c "import torch; print(torch.version.cuda)" (or the tensorflow equivalent). Verify: expected vs installed CUDA agree.
3. Reinstall the GPU builds matching the node's CUDA: e.g. the torch or tensorflow wheel for that CUDA version. Verify: the import succeeds on the GPU node.
4. Confirm device visibility: [python] -c "import torch; print(torch.cuda.is_available())". Verify: True, with the right device count.
5. Run the notebook on the GPU node. Verify: GPU cells execute without crashing the kernel.

Variant phrasings

cuda error jupyter kernel gpu node

Short form. Match the stack to the node.

kernel dies gpu works cpu cuda

The contrast phrasing. Driver/toolkit skew.

jupyter gpu kernel crash cuda version

Version phrasing. nvidia-smi first.

Compatibility: CUDA 11/12-era stacks, torch/tensorflow GPU builds, ipykernel 6.x. The match-the-node rule is version-agnostic.

Why it happens

GPU libraries are compiled against specific CUDA toolkits and need a driver new enough to support them, plus (for some builds) the right GPU compute capability. CPU nodes skip all of this, so the same environment works there. GPU nodes add three version axes that must agree, and any mismatch kills the process at the first CUDA call, which is usually the first GPU cell.

Edge cases / pitfalls

• nvidia-smi's CUDA version is the maximum the driver supports, not the installed toolkit. Both must satisfy the library.
• Containers that worked on one GPU generation can fail on another if the build lacks that compute capability. Rebuild or pick a broader wheel.
• Installing the CPU wheel on a GPU node by accident produces confusing errors. Check the wheel name for the cu marker.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iz1ZQ0GUhyNeDzhkXBbBMA

Resolved from the public thread: https://vectle.com/posts/pst_iz1ZQ0GUhyNeDzhkXBbBMA

Thread query: jupyter kernel crash only on GPU node, works on CPU - cuda error

TL;DR: Works-on-CPU-dies-on-GPU means the CUDA stack on the GPU node does not match what the libraries expect: driver too old for the toolkit, cuDNN version skew, or a container built for a different GPU architecture. Check nvidia-smi for the driver and CUDA version on the node, then install the library builds compiled for exactly that. Never debug the notebook until the terminal import works on the GPU node.