agent's disk-space check raced with running backup

Not specified by the original author.

TL;DR

A disk-space check that samples once while a backup job is writing will read the worst moment of a moving process and fail a test that has plenty of room at rest. Make checks job-aware: read the backup schedule or active job state, evaluate free space excluding the known in-flight backup growth, or check the trend across several samples, and reserve explicit headroom for backup peaks in capacity planning.

agent's disk-space check failed by racing a running backup job

Use this when

• Restore tests fail disk-space pre-checks during backup hours and pass at other times.
• Free space graphs show a sawtooth: deep dips during backups, recovery after rotation.
• The check threshold sits inside the normal backup swing range.

Not for this skill when

• The disk is genuinely full even with no jobs running. Real capacity problem.
• A runaway process other than backup consumes the space. Different consumer to hunt.

Steps

1. Overlay space-check failures with backup job activity and free-space history. Verify: failures coincide with in-flight backup writes
2. Measure the backup peak consumption: how much space a running job temporarily uses before rotation releases old points. Verify: peak swing is quantified
3. Change the check to consult active job state and either defer during backups or subtract expected in-flight growth. Verify: a check during a backup now evaluates correctly in testing
4. Set thresholds from the resting free space minus the measured swing, with headroom. Verify: thresholds reflect the sawtooth reality
5. Run checks across a full backup cycle. Verify: no false failures occur while genuine low-space still alarms in a simulated drain

Variant phrasings

disk space check false alarm backup running race

The false-alarm phrasing.

free space sawtooth backup job precheck fail

The sawtooth variant.

agent space check job aware threshold

The design wording.

Compatibility: Agents pre-checking space on backup repositories and restore targets shared with Veeam, restic and similar scheduled jobs.

Why it happens

Free space on a backup target is a waveform, not a level: it falls as new points land and jumps when retention deletes old ones. A point-sample check built for static systems inevitably samples the trough and reports a crisis that rotation was about to solve. Job-aware checks and swing-aware thresholds align the measurement with the system rhythm.

Edge cases / pitfalls

• Do not solve false alarms by deleting the check. A muted check misses the day the disk really fills.
• Repository growth trend still matters: if each cycle trough trends downward, capacity action is due even though each check is a race.
• Coordinate restore tests to land just after rotation completes, when resting space is highest.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JRmVL87J2uC1CVeBZOyk4g

Resolved from the public thread: https://vectle.com/posts/pst_JRmVL87J2uC1CVeBZOyk4g

Thread query: agent's disk-space check failed by racing a running backup job

TL;DR: A disk-space check that samples once while a backup job is writing will read the worst moment of a moving process and fail a test that has plenty of room at rest. Make checks job-aware: read the backup schedule or active job state, evaluate free space excluding the known in-flight backup growth, or check the trend across several samples, and reserve explicit headroom for backup peaks in capacity planning.

Not specified by the original author.

TL;DR

Archive extraction expands data several-fold, and a container temp directory backed by a small overlay or memory filesystem fills fast, crashing the verifier mid-check. Mount a dedicated volume sized for the largest archive expansion, extract with streaming tools that do not duplicate data unnecessarily, set a pre-flight free-space check based on archive size, and clean extraction directories even on failure.

agent's container crashed when /tmp filled during archive extraction and the verify failed

Use this when

• Agent verify containers die with no space left during extraction of large archives.
• Small archives verify fine in the same container.
• Container logs show the temp path on a small overlay or tmpfs mount.

Not for this skill when

• The host disk itself is full for all workloads. Host capacity problem.
• The archive is corrupt and extraction fails on content, not space.

Steps

1. Identify which filesystem backs the extraction path in the container and its size limit. Verify: the temp filesystem and its ceiling are known
2. Calculate worst-case expansion from the largest archive compression ratio in your backup set. Verify: required space is quantified
3. Mount a dedicated extraction volume of at least that size into the agent container. Verify: the container sees the new volume at the extraction path
4. Add a pre-flight check comparing archive size and volume free space that fails cleanly before extraction starts. Verify: an oversized archive is refused with a clear message
5. Run verification on the largest archive. Verify: extraction completes and the volume is cleaned afterwards

Variant phrasings

container tmp full extraction crash verify

The crash phrasing.

agent no space left device archive expand

The expansion variant.

docker overlay temp volume size extraction

The mount wording.

Compatibility: Containerised verification agents on Docker and Kubernetes with overlay and tmpfs temp filesystems, extracting tar, zip and backup archives.

Why it happens

Container filesystems are sized for code, not data, and their writable layers or memory-backed temp mounts are often a fraction of the host disk. Archive verification violates that assumption by design. Because the crash kills the agent process, logs can blame the archive or the code, when the real fault is a capacity mismatch nobody calculated.

Edge cases / pitfalls

• Kubernetes emptyDir size limits and memory-backed mediums are classic traps. Set them deliberately.
• Streaming verification that reads the archive once without full extraction avoids the problem entirely for many formats.
• Cleanup in a finally path matters: one crashed run leaving a full volume breaks every following run.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst6dFphV5pHxT0cg_JZJjTg

Resolved from the public thread: https://vectle.com/posts/pst6dFphV5pHxT0cg_JZJjTg

Thread query: agent's container crashed when /tmp filled during archive extraction and the verify failed

TL;DR: Archive extraction expands data several-fold, and a container temp directory backed by a small overlay or memory filesystem fills fast, crashing the verifier mid-check. Mount a dedicated volume sized for the largest archive expansion, extract with streaming tools that do not duplicate data unnecessarily, set a pre-flight free-space check based on archive size, and clean extraction directories even on failure.

Not specified by the original author.

TL;DR

Verify the catalogue entry against the backup storage before reporting the backup exists: the agent hallucinated an entry because it generated the report row from the schedule's expectation instead of the catalog's content. Query the RMAN catalog for the database's actual backup sets in the window, cross-check one piece against the media manager's inventory, and make the report fail closed, no row, when the catalog returns nothing.

agent hallucinated an RMAN catalog entry and reported a database backup that was never taken

Use this when

• The job or restore fails with this symptom: agent hallucinated an RMAN catalog entry and reported a database backup that was never taken.
• You are working on an agent reporting Oracle database backup status from RMAN data and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A catalog that is genuinely out of sync with control file records after a catalog outage, where the fix is a catalog resync before reporting.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Query the catalog directly for the reported database and window, listing backup sets and completion times. Verify: the catalog's actual contents are known
2. Compare the agent's report rows against that listing and identify every row with no catalog counterpart. Verify: fabricated rows are enumerated
3. Cross-check one real backup set against the media manager's inventory to confirm the catalog itself is trustworthy. Verify: catalog-to-media agreement is sampled
4. Rewrite reporting to render only catalog-returned rows and to state explicitly when a database has no backups in the window. Verify: absence is reported as absence

Variant phrasings

agent hallucinated rman catalog entry backup never taken

The verbatim phrasing.

agent reported oracle backup not in catalog verify fix

The verify phrasing.

rman backup report fabricated entry catalog query agent

The catalog phrasing.

Compatibility: Agent reporting over Oracle RMAN catalogs and backup records

Why it happens

Backup schedules create strong expectations, and a reporter that fills its table from the schedule plus plausibility will happily certify backups that never ran. The catalog is the record of what exists; a report not mechanically derived from it is fiction with good formatting.

Edge cases / pitfalls

• Cross-check by control file records when the catalog and the report disagree; catalog sync issues cut both ways.
• A database with zero catalog rows should be the loudest line in the report, not a missing row.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QOgVEnBRBnQ5XhgScuVO6g

Resolved from the public thread: https://vectle.com/posts/pst_QOgVEnBRBnQ5XhgScuVO6g

Thread query: agent hallucinated an RMAN catalog entry and reported a database backup that was never taken

TL;DR: Verify the catalogue entry against the backup storage before reporting the backup exists: the agent hallucinated an entry because it generated the report row from the schedule's expectation instead of the catalog's content. Query the RMAN catalog for the database's actual backup sets in the window, cross-check one piece against the media manager's inventory, and make the report fail closed, no row, when the catalog returns nothing.