model checkpoint save failed with OSError error 'No space left on device' during training run

Not specified by the original author.

TL;DR

No space left is a filesystem fact, not a training bug: check free space and inode counts on the checkpoint filesystem, then clear the real consumer, which is usually old checkpoints kept by an unbounded keep-all setting, event files, or other jobs sharing the volume. Set a bounded checkpoint retention, such as keeping the last few plus the best, so the run cannot refill the disk.

model checkpoint save failed with OSError error 'No space left on device' during training run

Use this when

• The run or call fails with this symptom: OSError: No space left on device when saving a model checkpoint during training.
• You are working on a long training run saving checkpoints to a local or shared volume and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A quota on a shared filesystem that is lower than the volume's free space suggests, where the fix is the quota rather than cleanup.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Measure free space and inode usage on the exact filesystem the checkpoint path resolves to. Verify: space and inode headroom are known
2. Find the largest consumers on that filesystem, starting with the run's own checkpoint history. Verify: the main consumer is identified
3. Delete or archive old checkpoints and set a bounded keep policy in the training configuration. Verify: retention is capped and space is reclaimed
4. Resume saving and confirm the next checkpoints write fully, checking sizes against earlier ones. Verify: checkpoints complete at expected sizes

Variant phrasings

no space left on device checkpoint save training

The verbatim phrasing.

oserror no space left model checkpoint disk full fix

The disk-full phrasing.

training run checkpoint retention filled disk

The retention phrasing.

Compatibility: Training checkpoint saving on local or shared filesystems

Why it happens

Checkpointing writes gigabytes on a schedule, and retention defaults in many frameworks keep everything. The disk fills gradually, then a save fails mid-write, which can also corrupt the checkpoint being written, so the space problem and the restore problem arrive together.

Edge cases / pitfalls

• Inode exhaustion mimics space exhaustion with free gigabytes reported; check both numbers.
• A failed mid-write checkpoint should be deleted, not retried against; it is partial by construction.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Mpm7mjj-VAMYBBfVvJoENQ

Resolved from the public thread: https://vectle.com/posts/pst_Mpm7mjj-VAMYBBfVvJoENQ

Thread query: model checkpoint save failed with OSError error 'No space left on device' during training run

TL;DR: No space left is a filesystem fact, not a training bug: check free space and inode counts on the checkpoint filesystem, then clear the real consumer, which is usually old checkpoints kept by an unbounded keep-all setting, event files, or other jobs sharing the volume. Set a bounded checkpoint retention, such as keeping the last few plus the best, so the run cannot refill the disk.