agent hallucinated backup size from stale manifest and marked verification passed

TL;DR

Invalidate the manifest the agent used, fetch a fresh one directly from the repository, and compare its size and snapshot identifier against a live listing before accepting any pass. Then make the agent refuse to verify from a manifest older than the backup it describes, and require the size it reports to come from the current run, not from cached state.

agent hallucinated backup size from stale manifest and marked verification passed

Use this when

• The agent reports verification passed with a backup size that does not match the repository or catalog.
• The size reported matches an older backup exactly, suggesting a stale manifest was read.
• Manifest files are cached, copied, or generated by a separate job from the backup itself.

Not for this skill when

• Reported size differs only because one view shows logical size and another shows deduplicated stored size, both current. That is a reporting definition issue.
• The agent read a fresh manifest and the backup itself is the wrong size. That is a backup content problem.

Steps

1. Preserve the evidence: save the manifest the agent used, with its file time, snapshot identifier, reported size, and the run that produced the pass, before refreshing anything. Verify: the stale manifest is captured unchanged for comparison
2. Fetch ground truth from the repository: list the actual snapshot or restore point, its real stored and logical sizes, and its completion time through the backup product or repository tooling. Verify: the live values are recorded from a direct query in this investigation, not from any manifest
3. Diff the two: compare snapshot identifier, backup completion time, file count, and size between the stale manifest and live data, and find where the manifest came from - a cache folder, a previous run output, or a generator job that did not rerun. Verify: the source and age of the stale manifest are identified
4. Fix the read path: point the agent at the manifest produced by the current backup run, keyed by snapshot identifier, and delete or quarantine stale manifest copies in shared locations so a future run cannot pick one up by name alone. Verify: a new run loads a manifest whose identifier matches the snapshot it verifies
5. Add freshness and consistency gates: reject a manifest older than the backup completion time plus a small tolerance, require its size to match a live repository query within rounding, and mark the run inconclusive, not passed, when either check fails. Verify: feeding the saved stale manifest to the agent now produces an inconclusive result with a stale manifest reason
6. Audit past passes: list verification passes issued since the stale manifest was written, recheck those snapshots against live data, and requeue any whose size or existence cannot be confirmed. Verify: every pass in the affected window is either reconfirmed from live data or requeued

Why it happens

Agents often verify from a manifest because it is convenient structured data - snapshot name, file count, and size in one file. But a manifest is a copy, and copies go stale when a generator job fails silently, a cache is keyed by backup name rather than by snapshot, or a previous run output is left in a shared working folder. The agent then does correct arithmetic on old facts - it compares the stale manifest against itself, or against a checksum recorded alongside it, finds agreement, and reports a pass with a size from an earlier backup. This reads like hallucination because the reported number appears from nowhere in the current run; in fact it was real once, for a different restore point. The failure mode is dangerous precisely because the pass is confident, formatted, and wrong, and it can cover a backup that is undersized or missing entirely.

Variants and edge cases

size matches an older backup, identifier differs

Classic stale manifest pickup by filename. Generators that write a fixed name such as the job name overwrite or fail to overwrite predictably; write manifests with the snapshot identifier in the path and have the agent require that match.

manifest is fresh but its size field is wrong

The generator computed size before the backup finished, from a partial listing, or in the wrong units. Fix generation to run after backup completion from repository data, and cross check units by comparing one known snapshot by hand.

two agents share one manifest location

Concurrent runs overwrite each others manifests, so either agent may read the others data mid run. Give each run its own working directory and pass the manifest path explicitly in run state, never by scanning a shared folder. Edge cases: deduplicated repositories report several legitimate sizes - logical, unique, and stored - so the gate should compare like with like and record which definition the manifest uses, or the freshness fix will create false alarms. Manifest signing or hashing helps only if the agent also checks freshness; a perfectly authentic old manifest is still old. Keep manifest age, snapshot identifier, live size, and manifest size in the verification report for every run: a pass that cannot show where its size came from, and how old that source was, should be treated by reviewers as no pass at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_U4zOcxR5HM5fOQE5k7jjgg