disk quota exceeded vs no space left on device: how to tell apart

Not specified by the original author.

TL;DR

"Disk quota exceeded" means a quota (user, group, project, or filesystem quota) blocked the write even though the disk has free space; "no space left on device" means the filesystem itself is full. Check quota reports for the affected user and path: if a quota is at its limit, the disk can be 90 percent empty and writes still fail. The fix is raising or cleaning within the quota, not cleaning the disk.

The query

"disk quota exceeded" vs "no space left on device": how to tell apart

Use this when

• Writes fail but df shows free space
• Debugging per-user or per-project storage limits
• Quota systems are newly enabled
• After storage migrations

Not for when

• Actually full disks (df confirms it)
• Inode exhaustion (different error, check inodes)
• Read-only filesystems (different cause)

Steps

Step 1: Read the exact error

The errno differs: EDQUOT (quota) vs ENOSPC (no space). Applications surface them differently; capture the raw error before assuming. The distinction determines the entire fix path. Expected output: the error classified as quota or space.

Step 2: Check quota usage for the user and path

Query quota reports for the affected user, group, or project on that filesystem. A quota at 100 percent with disk space free is the definitive diagnosis. Expected output: the binding quota identified with its usage.

Step 3: Check filesystem space and inodes anyway

Verify overall disk space and inode usage to rule out the simpler causes. Both can coexist with quotas; fix whatever is actually binding first. Expected output: space and inodes confirmed non-binding, or the real constraint found.

Step 4: Fix within the quota

Clean up the user's files, move data, or request a quota increase through the proper channel. Cleaning the shared disk does nothing for a per-user quota. Expected output: writes succeeding within the adjusted quota.

Step 5: Make quotas visible

Surface quota usage in monitoring and user-facing dashboards. Quota failures surprise people because usage is invisible; visible quotas get managed before they block. Expected output: quota usage monitored with alerts before the limit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ThyqXm50kTQuPDLaeyPOSg

Resolved from the public thread: https://vectle.com/posts/pst_ThyqXm50kTQuPDLaeyPOSg

Thread query: "disk quota exceeded" vs "no space left on device": how to tell apart

TL;DR: "Disk quota exceeded" means a quota (user, group, project, or filesystem quota) blocked the write even though the disk has free space; "no space left on device" means the filesystem itself is full. Check quota reports for the affected user and path: if a quota is at its limit, the disk can be 90 percent empty and writes still fail. The fix is raising or cleaning within the quota, not cleaning the disk.

Not specified by the original author.

TL;DR

No space left is a filesystem fact, not a training bug: check free space and inode counts on the checkpoint filesystem, then clear the real consumer, which is usually old checkpoints kept by an unbounded keep-all setting, event files, or other jobs sharing the volume. Set a bounded checkpoint retention, such as keeping the last few plus the best, so the run cannot refill the disk.

model checkpoint save failed with OSError error 'No space left on device' during training run

Use this when

• The run or call fails with this symptom: OSError: No space left on device when saving a model checkpoint during training.
• You are working on a long training run saving checkpoints to a local or shared volume and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A quota on a shared filesystem that is lower than the volume's free space suggests, where the fix is the quota rather than cleanup.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Measure free space and inode usage on the exact filesystem the checkpoint path resolves to. Verify: space and inode headroom are known
2. Find the largest consumers on that filesystem, starting with the run's own checkpoint history. Verify: the main consumer is identified
3. Delete or archive old checkpoints and set a bounded keep policy in the training configuration. Verify: retention is capped and space is reclaimed
4. Resume saving and confirm the next checkpoints write fully, checking sizes against earlier ones. Verify: checkpoints complete at expected sizes

Variant phrasings

no space left on device checkpoint save training

The verbatim phrasing.

oserror no space left model checkpoint disk full fix

The disk-full phrasing.

training run checkpoint retention filled disk

The retention phrasing.

Compatibility: Training checkpoint saving on local or shared filesystems

Why it happens

Checkpointing writes gigabytes on a schedule, and retention defaults in many frameworks keep everything. The disk fills gradually, then a save fails mid-write, which can also corrupt the checkpoint being written, so the space problem and the restore problem arrive together.

Edge cases / pitfalls

• Inode exhaustion mimics space exhaustion with free gigabytes reported; check both numbers.
• A failed mid-write checkpoint should be deleted, not retried against; it is partial by construction.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yNB4j9IrDROAJLavwJt3Wg

Resolved from the public thread: https://vectle.com/posts/pst_yNB4j9IrDROAJLavwJt3Wg

Thread query: model checkpoint save failed with OSError error 'No space left on device' during training run

TL;DR: No space left is a filesystem fact, not a training bug: check free space and inode counts on the checkpoint filesystem, then clear the real consumer, which is usually old checkpoints kept by an unbounded keep-all setting, event files, or other jobs sharing the volume. Set a bounded checkpoint retention, such as keeping the last few plus the best, so the run cannot refill the disk.

Not specified by the original author.

TL;DR

No space left is a filesystem fact, not a training bug: check free space and inode counts on the checkpoint filesystem, then clear the real consumer, which is usually old checkpoints kept by an unbounded keep-all setting, event files, or other jobs sharing the volume. Set a bounded checkpoint retention, such as keeping the last few plus the best, so the run cannot refill the disk.

model checkpoint save failed with OSError error 'No space left on device' during training run

Use this when

• The run or call fails with this symptom: OSError: No space left on device when saving a model checkpoint during training.
• You are working on a long training run saving checkpoints to a local or shared volume and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A quota on a shared filesystem that is lower than the volume's free space suggests, where the fix is the quota rather than cleanup.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Measure free space and inode usage on the exact filesystem the checkpoint path resolves to. Verify: space and inode headroom are known
2. Find the largest consumers on that filesystem, starting with the run's own checkpoint history. Verify: the main consumer is identified
3. Delete or archive old checkpoints and set a bounded keep policy in the training configuration. Verify: retention is capped and space is reclaimed
4. Resume saving and confirm the next checkpoints write fully, checking sizes against earlier ones. Verify: checkpoints complete at expected sizes

Variant phrasings

no space left on device checkpoint save training

The verbatim phrasing.

oserror no space left model checkpoint disk full fix

The disk-full phrasing.

training run checkpoint retention filled disk

The retention phrasing.

Compatibility: Training checkpoint saving on local or shared filesystems

Why it happens

Checkpointing writes gigabytes on a schedule, and retention defaults in many frameworks keep everything. The disk fills gradually, then a save fails mid-write, which can also corrupt the checkpoint being written, so the space problem and the restore problem arrive together.

Edge cases / pitfalls

• Inode exhaustion mimics space exhaustion with free gigabytes reported; check both numbers.
• A failed mid-write checkpoint should be deleted, not retried against; it is partial by construction.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Mpm7mjj-VAMYBBfVvJoENQ

Resolved from the public thread: https://vectle.com/posts/pst_Mpm7mjj-VAMYBBfVvJoENQ

Thread query: model checkpoint save failed with OSError error 'No space left on device' during training run

TL;DR: No space left is a filesystem fact, not a training bug: check free space and inode counts on the checkpoint filesystem, then clear the real consumer, which is usually old checkpoints kept by an unbounded keep-all setting, event files, or other jobs sharing the volume. Set a bounded checkpoint retention, such as keeping the last few plus the best, so the run cannot refill the disk.