agent's disk-space check failed by racing a running backup job
Fixes agent disk-space checks failing by racing a running backup job: sample space with job awareness, use trends instead of instant readings, and reserve target headroom. Use when a space check misfires because a backup is mid-write. Not for genuinely full disks.
TL;DR
A disk-space check that samples once while a backup job is writing will read the worst moment of a moving process and fail a test that has plenty of room at rest. Make checks job-aware: read the backup schedule or active job state, evaluate free space excluding the known in-flight backup growth, or check the trend across several samples, and reserve explicit headroom for backup peaks in capacity planning.
agent's disk-space check failed by racing a running backup jobUse this when
- Restore tests fail disk-space pre-checks during backup hours and pass at other times.
- Free space graphs show a sawtooth: deep dips during backups, recovery after rotation.
- The check threshold sits inside the normal backup swing range.
Not for this skill when
- The disk is genuinely full even with no jobs running. Real capacity problem.
- A runaway process other than backup consumes the space. Different consumer to hunt.
Steps
- Overlay space-check failures with backup job activity and free-space history. Verify: failures coincide with in-flight backup writes
- Measure the backup peak consumption: how much space a running job temporarily uses before rotation releases old points. Verify: peak swing is quantified
- Change the check to consult active job state and either defer during backups or subtract expected in-flight growth. Verify: a check during a backup now evaluates correctly in testing
- Set thresholds from the resting free space minus the measured swing, with headroom. Verify: thresholds reflect the sawtooth reality
- Run checks across a full backup cycle. Verify: no false failures occur while genuine low-space still alarms in a simulated drain
Variant phrasings
disk space check false alarm backup running race
The false-alarm phrasing.
free space sawtooth backup job precheck fail
The sawtooth variant.
agent space check job aware threshold
The design wording.
Compatibility: Agents pre-checking space on backup repositories and restore targets shared with Veeam, restic and similar scheduled jobs.
Why it happens
Free space on a backup target is a waveform, not a level: it falls as new points land and jumps when retention deletes old ones. A point-sample check built for static systems inevitably samples the trough and reports a crisis that rotation was about to solve. Job-aware checks and swing-aware thresholds align the measurement with the system rhythm.
Edge cases / pitfalls
- Do not solve false alarms by deleting the check. A muted check misses the day the disk really fills.
- Repository growth trend still matters: if each cycle trough trends downward, capacity action is due even though each check is a race.
- Coordinate restore tests to land just after rotation completes, when resting space is highest.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_JRmVL87J2uC1CVeBZOyk4g