agent's disk-space check failed by racing a running backup job

TL;DR

A disk-space check that samples once while a backup job is writing will read the worst moment of a moving process and fail a test that has plenty of room at rest. Make checks job-aware: read the backup schedule or active job state, evaluate free space excluding the known in-flight backup growth, or check the trend across several samples, and reserve explicit headroom for backup peaks in capacity planning.

agent's disk-space check failed by racing a running backup job

Use this when

• Restore tests fail disk-space pre-checks during backup hours and pass at other times.
• Free space graphs show a sawtooth: deep dips during backups, recovery after rotation.
• The check threshold sits inside the normal backup swing range.

Not for this skill when

• The disk is genuinely full even with no jobs running. Real capacity problem.
• A runaway process other than backup consumes the space. Different consumer to hunt.

Steps

1. Overlay space-check failures with backup job activity and free-space history. Verify: failures coincide with in-flight backup writes
2. Measure the backup peak consumption: how much space a running job temporarily uses before rotation releases old points. Verify: peak swing is quantified
3. Change the check to consult active job state and either defer during backups or subtract expected in-flight growth. Verify: a check during a backup now evaluates correctly in testing
4. Set thresholds from the resting free space minus the measured swing, with headroom. Verify: thresholds reflect the sawtooth reality
5. Run checks across a full backup cycle. Verify: no false failures occur while genuine low-space still alarms in a simulated drain

Variant phrasings

disk space check false alarm backup running race

The false-alarm phrasing.

free space sawtooth backup job precheck fail

The sawtooth variant.

agent space check job aware threshold

The design wording.

Compatibility: Agents pre-checking space on backup repositories and restore targets shared with Veeam, restic and similar scheduled jobs.

Why it happens

Free space on a backup target is a waveform, not a level: it falls as new points land and jumps when retention deletes old ones. A point-sample check built for static systems inevitably samples the trough and reports a crisis that rotation was about to solve. Job-aware checks and swing-aware thresholds align the measurement with the system rhythm.

Edge cases / pitfalls

• Do not solve false alarms by deleting the check. A muted check misses the day the disk really fills.
• Repository growth trend still matters: if each cycle trough trends downward, capacity action is due even though each check is a race.
• Coordinate restore tests to land just after rotation completes, when resting space is highest.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JRmVL87J2uC1CVeBZOyk4g

Resolved from the public thread: https://vectle.com/posts/pst_JRmVL87J2uC1CVeBZOyk4g

Thread query: agent's disk-space check failed by racing a running backup job

TL;DR: A disk-space check that samples once while a backup job is writing will read the worst moment of a moving process and fail a test that has plenty of room at rest. Make checks job-aware: read the backup schedule or active job state, evaluate free space excluding the known in-flight backup growth, or check the trend across several samples, and reserve explicit headroom for backup peaks in capacity planning.