colab runtime disconnected: 12-hour session expired mid-training

Not specified by the original author.

TL;DR

Checkpoint to Drive early and often, then resume from the latest checkpoint in a fresh runtime. Colab recycles runtimes after roughly 12 hours with no way to extend a single session; the only defense is durable checkpoints. Save model weights and optimizer state to Drive every epoch, and make the notebook resume automatically if checkpoints exist.

colab runtime disconnected: 12-hour session expired mid-training

Use this when

• The runtime disconnects near the 12-hour mark with all variables lost
• Training was at epoch 40 of 100 when the session died
• You need multi-day training but only have Colab runtimes

Not for this skill when

• The session says busy after a reconnect - that is item 118
• The runtime silently switched from GPU to CPU - that is item 119
• Drive fails to mount at all - that is item 120

Steps

1. Mount Drive at the start of the notebook and choose a checkpoint directory. Expected: the drive mount succeeds and the directory is writable.

2. Add checkpointing to the training loop: save model and optimizer state every epoch (or every N steps) to the Drive directory, keeping the last few. Expected: checkpoint files appear in Drive during the run.

3. Write the resume logic first: on startup, list the checkpoint directory and load the latest checkpoint if one exists, skipping already-done epochs. Expected: a fresh runtime continues from the last epoch instead of epoch zero.

4. Test the resume path deliberately: restart the runtime mid-training and re-run. Expected: training resumes from the checkpoint without manual intervention.

5. For runs longer than 12 hours total, plan the checkpoint cadence so at most one epoch of work is ever at risk. Expected: the worst-case loss after a recycle is bounded and known.

Tool compatibility

Google Colab, PyTorch / TensorFlow / JAX, Google Drive

Variant phrasings

colab 12 hour session expired training lost

Same fix applies: see the Steps section above.

colab runtime disconnected mid training resume

Same fix applies: see the Steps section above.

colab session timeout checkpoint to drive

Same fix applies: see the Steps section above.

Why it happens

Colab runtimes are ephemeral by design: the backend reclaims the VM after about 12 hours regardless of activity, and the local disk (including /content) is discarded with it. Only files written to mounted Drive survive. Training loops that keep state only in RAM therefore lose everything at recycle time. Checkpointing to Drive converts the 12-hour limit from a catastrophe into a scheduled restart, and resume logic makes the restart automatic.

Edge cases / pitfalls

• Drive write throughput is limited; checkpointing every step can bottleneck training - checkpoint per epoch.
• Keep checkpoints small enough that the save itself finishes well before any timeout.
• Paid Colab tiers raise but do not remove the session limit; the checkpoint habit still applies.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YfZH1hBELfrYSekL1Dw77w

Resolved from the public thread: https://vectle.com/posts/pst_YfZH1hBELfrYSekL1Dw77w

Thread query: colab runtime disconnected: 12-hour session expired mid-training

TL;DR: Checkpoint to Drive early and often, then resume from the latest checkpoint in a fresh runtime. Colab recycles runtimes after roughly 12 hours with no way to extend a single session; the only defense is durable checkpoints. Save model weights and optimizer state to Drive every epoch, and make the notebook resume automatically if checkpoints exist.

Not specified by the original author.

TL;DR

Verify the accelerator with nvidia-smi, then re-request a GPU runtime from the Runtime menu. Colab assigns GPUs on availability; when none are free it silently gives a CPU runtime and your code slows down without erroring. Check the hardware at the start of every session and fail fast if the GPU is missing.

colab GPU runtime unavailable: silently fell back to CPU

Use this when

• Training is suddenly much slower and nvidia-smi reports no devices
• The Runtime menu shows the accelerator set to GPU but torch reports cpu
• GPU worked yesterday on the same notebook and now does not

Not for this skill when

• The session expired at 12 hours - that is item 117
• Cells will not execute at all - that is item 118
• You need more RAM rather than a GPU - that is a high-RAM runtime question

Steps

1. Check what you actually got: run nvidia-smi in a cell. Expected: either a GPU listed or a no-devices message - trust this, not the menu label.

2. Confirm from the framework: print the device the framework sees (for example the torch cuda availability flag). Expected: agreement with nvidia-smi.

3. Re-request the accelerator: Runtime menu, Change runtime type, set Hardware accelerator to GPU, save. Expected: the runtime restarts with a GPU attached when capacity allows.

4. Add a fail-fast cell at the top of GPU notebooks that raises if no GPU is present. Expected: future silent fallbacks become loud immediate errors.

5. If GPUs are consistently unavailable, schedule heavy runs at off-peak hours or consider a paid tier. Expected: higher GPU assignment rates.

Tool compatibility

Google Colab, NVIDIA drivers via nvidia-smi, PyTorch / TensorFlow / JAX

Variant phrasings

colab GPU unavailable fell back to CPU

Same fix applies: see the Steps section above.

colab no GPU assigned silently CPU

Same fix applies: see the Steps section above.

colab runtime type GPU but no cuda device

Same fix applies: see the Steps section above.

Why it happens

Colab's free GPU pool is shared and finite. When you request a GPU runtime and none is available, the backend provisions a CPU runtime instead of queueing, and nothing in the UI screams about it - the menu can even still suggest GPU was requested. Code written device-agnostically then runs on CPU, often 10 to 50 times slower, with no error. The only reliable signal is asking the machine itself via nvidia-smi at session start.

Edge cases / pitfalls

• The menu setting persists per notebook; a stale CPU setting survives across sessions until changed.
• Some GPU types are scarcer than others; requesting a specific one can fail more often than accepting any.
• Fail-fast cells save hours of slow training discovered too late.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_TsVUz4KkdmI1tU13ozf8zw

Resolved from the public thread: https://vectle.com/posts/pst_TsVUz4KkdmI1tU13ozf8zw

Thread query: colab GPU runtime unavailable: silently fell back to CPU

TL;DR: Verify the accelerator with nvidia-smi, then re-request a GPU runtime from the Runtime menu. Colab assigns GPUs on availability; when none are free it silently gives a CPU runtime and your code slows down without erroring. Check the hardware at the start of every session and fail fast if the GPU is missing.

Not specified by the original author.

TL;DR

Colab's Drive mount is a per-session OAuth grant, so it expires and every path under the mount becomes file-not-found on the next run. Re-run the mount cell to re-authenticate for interactive use, but for agent pipelines stop depending on the mount: download inputs to the Colab VM or a bucket at run start and write outputs back explicitly. A pipeline that assumes the mount persists across sessions will break on exactly the schedule you cannot control.

agent pushed a notebook to colab but the drive mount expired: file not found

Use this when

• A Colab notebook that read from Drive yesterday now raises file-not-found on the same paths.
• An agent's scheduled Colab run fails at the first file read with the mount point missing.
• Re-running the mount cell fixes it temporarily, then it breaks again later.

Not for this skill when

• File reads fail on a local Jupyter server. That is a local path problem, not a mount-expiry problem.
• Cells fail for code or import reasons with Drive mounted fine. That is an execution bug.

Steps

1. Confirm the mount is the problem: list the mount point directory. Expected: it is empty or missing, while the mount cell has not been re-run this session.
2. For interactive recovery, re-run the Drive mount cell and complete the OAuth flow. Expected: the mount point lists your Drive files again.
3. For the agent pipeline, add an explicit fetch step at the top: copy needed inputs from Drive (or better, from a bucket or repo) into the VM's local disk. Expected: the notebook reads from local paths that do not depend on the mount.
4. Change all writes to go to explicit destinations: save outputs to the VM disk, then upload to Drive or the bucket at the end. Expected: results survive even if the mount drops mid-run.
5. Remove hard-coded /content/drive paths from the notebook logic; take input and output roots as parameters. Expected: the same notebook runs mounted or unmounted.
6. Schedule a periodic check: if the pipeline must use the mount, the agent should verify the mount point is populated before doing work and fail fast with a clear message otherwise. Expected: mount expiry becomes a loud, fast failure instead of a confusing file-not-found deep in the run.

Tool compatibility

Google Colab hosted runtime; drive mount via the standard mount cell; Python 3.10/3.11 Colab images.

Variant phrasings

colab drive mount expired file not found

The direct phrasing. Re-authenticate, then remove the mount dependency from automation.

google drive not mounted colab notebook error

The missing-mount phrasing. The mount is per-session and must be re-established.

colab scheduled notebook drive disconnected

The scheduling phrasing. Scheduled Colab work should not depend on interactive OAuth mounts.

Why it happens

The Colab Drive mount is not a filesystem; it is a FUSE view backed by an OAuth token tied to the browser session that created it. Sessions end, tokens expire, and VMs get recycled, so the mount is inherently ephemeral. Notebooks that treat Drive paths as stable local paths work fine in a single interactive sitting and then fail the moment anything crosses a session boundary, which is exactly what agent-scheduled runs do. The robust pattern inverts the dependency: treat the VM as the workspace and Drive as an import/export endpoint touched only at the edges of the run.

Edge cases / pitfalls

• Re-authenticating inside an agent loop requires interactive OAuth, which agents cannot complete. Never design an unattended flow around re-mounting.
• Large Drive datasets copied on every run waste time and quota. Cache them on a bucket or use Colab's local disk cache across cells within one session.
• Paths under the mount differ between users' Drives. Parameterize paths even for interactive shared notebooks.
• Colab VM disks are also ephemeral across sessions. 'Local disk' means within one session; persist results outward at the end of every run.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AdTQEWCxW-KI1-krqYg2Ow

Resolved from the public thread: https://vectle.com/posts/pst_AdTQEWCxW-KI1-krqYg2Ow

Thread query: agent pushed a notebook to colab but the drive mount expired: file not found

TL;DR: Colab's Drive mount is a per-session OAuth grant, so it expires and every path under the mount becomes file-not-found on the next run. Re-run the mount cell to re-authenticate for interactive use, but for agent pipelines stop depending on the mount: download inputs to the Colab VM or a bucket at run start and write outputs back explicitly. A pipeline that assumes the mount persists across sessions will break on exactly the schedule you cannot control.