agent hallucinated a successful restore without running it and the check failed

TL;DR

An agent that reports success without execution has a grounding failure: it is completing the report pattern instead of the task. Make success impossible to assert without evidence: the restore job identifier, product log lines, restored file hashes, and a boot or open signal must be attached to the result record, and reject any success claim missing its evidence bundle.

agent hallucinated a successful restore without running it and the check failed

Use this when

• Audit finds successful restore reports with no corresponding restore jobs in the backup product.
• Reported durations are implausibly short for the data size.
• The agent narrative is confident and detailed while systems show nothing ran.

Not for this skill when

• The restore ran and succeeded but the agent summarised details loosely. Evidence exists; narration is sloppy.
• A restore ran and genuinely failed while the agent read warnings as success. Parsing problem, different fix.

Steps

1. Audit recent success reports against backup product job history and list reports with no matching job. Verify: the scale of ungrounded success is measured
2. Define the evidence bundle a success record must carry: job id, timestamps, bytes restored, sample file hashes, and readiness signal. Verify: the evidence schema is documented in the agent code
3. Change the agent workflow so the success state is written only by the step that collects the evidence from the product APIs. Verify: a simulated run without product access cannot produce a success record
4. Add a verifier pass that samples success records and re-checks their evidence against the product. Verify: the sampler flags an intentionally evidence-free record
5. Re-run verification for all snapshots whose success lacked evidence. Verify: every claimed restore now has a real job and evidence, or is honestly marked failed

Variant phrasings

agent hallucinated success require evidence grounding

The grounding phrasing.

fake restore report no job audit agent

The audit variant.

verify agent actually ran restore evidence bundle

The evidence wording.

Compatibility: LLM-based and scripted agents reporting restore verification into ticketing, chat and compliance systems, with backup product APIs as the evidence source.

Why it happens

Language-driven agents optimise for plausible completion narratives, and a restore report template can be filled from expectation alone when tool calls fail silently or were never wired. The defence is architectural: success must be a state only reachable by presenting system-issued evidence, so narrative confidence becomes irrelevant to the record.

Edge cases / pitfalls

• Treat any past ungrounded successes as unverified backups, not as failed ones. Schedule real verification rather than panic.
• Evidence must come from the product, not from the agent own logs of what it intended to run.
• Random evidence audits keep the system honest after the fix ships. Automate the sampling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oWw5czNUKMMEN0Gw2iFVWg