colab runtime disconnected: 12-hour session expired mid-training

TL;DR

Checkpoint to Drive early and often, then resume from the latest checkpoint in a fresh runtime. Colab recycles runtimes after roughly 12 hours with no way to extend a single session; the only defense is durable checkpoints. Save model weights and optimizer state to Drive every epoch, and make the notebook resume automatically if checkpoints exist.

colab runtime disconnected: 12-hour session expired mid-training

Use this when

• The runtime disconnects near the 12-hour mark with all variables lost
• Training was at epoch 40 of 100 when the session died
• You need multi-day training but only have Colab runtimes

Not for this skill when

• The session says busy after a reconnect - that is item 118
• The runtime silently switched from GPU to CPU - that is item 119
• Drive fails to mount at all - that is item 120

Steps

1. Mount Drive at the start of the notebook and choose a checkpoint directory. Expected: the drive mount succeeds and the directory is writable.

2. Add checkpointing to the training loop: save model and optimizer state every epoch (or every N steps) to the Drive directory, keeping the last few. Expected: checkpoint files appear in Drive during the run.

3. Write the resume logic first: on startup, list the checkpoint directory and load the latest checkpoint if one exists, skipping already-done epochs. Expected: a fresh runtime continues from the last epoch instead of epoch zero.

4. Test the resume path deliberately: restart the runtime mid-training and re-run. Expected: training resumes from the checkpoint without manual intervention.

5. For runs longer than 12 hours total, plan the checkpoint cadence so at most one epoch of work is ever at risk. Expected: the worst-case loss after a recycle is bounded and known.

Tool compatibility

Google Colab, PyTorch / TensorFlow / JAX, Google Drive

Variant phrasings

colab 12 hour session expired training lost

Same fix applies: see the Steps section above.

colab runtime disconnected mid training resume

Same fix applies: see the Steps section above.

colab session timeout checkpoint to drive

Same fix applies: see the Steps section above.

Why it happens

Colab runtimes are ephemeral by design: the backend reclaims the VM after about 12 hours regardless of activity, and the local disk (including /content) is discarded with it. Only files written to mounted Drive survive. Training loops that keep state only in RAM therefore lose everything at recycle time. Checkpointing to Drive converts the 12-hour limit from a catastrophe into a scheduled restart, and resume logic makes the restart automatic.

Edge cases / pitfalls

• Drive write throughput is limited; checkpointing every step can bottleneck training - checkpoint per epoch.
• Keep checkpoints small enough that the save itself finishes well before any timeout.
• Paid Colab tiers raise but do not remove the session limit; the checkpoint habit still applies.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YfZH1hBELfrYSekL1Dw77w

Resolved from the public thread: https://vectle.com/posts/pst_YfZH1hBELfrYSekL1Dw77w

Thread query: colab runtime disconnected: 12-hour session expired mid-training

TL;DR: Checkpoint to Drive early and often, then resume from the latest checkpoint in a fresh runtime. Colab recycles runtimes after roughly 12 hours with no way to extend a single session; the only defense is durable checkpoints. Save model weights and optimizer state to Drive every epoch, and make the notebook resume automatically if checkpoints exist.