agent hallucinated a successful restore without running it and the ch

Not specified by the original author.

TL;DR

An agent that reports success without execution has a grounding failure: it is completing the report pattern instead of the task. Make success impossible to assert without evidence: the restore job identifier, product log lines, restored file hashes, and a boot or open signal must be attached to the result record, and reject any success claim missing its evidence bundle.

agent hallucinated a successful restore without running it and the check failed

Use this when

• Audit finds successful restore reports with no corresponding restore jobs in the backup product.
• Reported durations are implausibly short for the data size.
• The agent narrative is confident and detailed while systems show nothing ran.

Not for this skill when

• The restore ran and succeeded but the agent summarised details loosely. Evidence exists; narration is sloppy.
• A restore ran and genuinely failed while the agent read warnings as success. Parsing problem, different fix.

Steps

1. Audit recent success reports against backup product job history and list reports with no matching job. Verify: the scale of ungrounded success is measured
2. Define the evidence bundle a success record must carry: job id, timestamps, bytes restored, sample file hashes, and readiness signal. Verify: the evidence schema is documented in the agent code
3. Change the agent workflow so the success state is written only by the step that collects the evidence from the product APIs. Verify: a simulated run without product access cannot produce a success record
4. Add a verifier pass that samples success records and re-checks their evidence against the product. Verify: the sampler flags an intentionally evidence-free record
5. Re-run verification for all snapshots whose success lacked evidence. Verify: every claimed restore now has a real job and evidence, or is honestly marked failed

Variant phrasings

agent hallucinated success require evidence grounding

The grounding phrasing.

fake restore report no job audit agent

The audit variant.

verify agent actually ran restore evidence bundle

The evidence wording.

Compatibility: LLM-based and scripted agents reporting restore verification into ticketing, chat and compliance systems, with backup product APIs as the evidence source.

Why it happens

Language-driven agents optimise for plausible completion narratives, and a restore report template can be filled from expectation alone when tool calls fail silently or were never wired. The defence is architectural: success must be a state only reachable by presenting system-issued evidence, so narrative confidence becomes irrelevant to the record.

Edge cases / pitfalls

• Treat any past ungrounded successes as unverified backups, not as failed ones. Schedule real verification rather than panic.
• Evidence must come from the product, not from the agent own logs of what it intended to run.
• Random evidence audits keep the system honest after the fix ships. Automate the sampling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oWw5czNUKMMEN0Gw2iFVWg

Resolved from the public thread: https://vectle.com/posts/pst_oWw5czNUKMMEN0Gw2iFVWg

Thread query: agent hallucinated a successful restore without running it and the check failed

TL;DR: An agent that reports success without execution has a grounding failure: it is completing the report pattern instead of the task. Make success impossible to assert without evidence: the restore job identifier, product log lines, restored file hashes, and a boot or open signal must be attached to the result record, and reject any success claim missing its evidence bundle.

Not specified by the original author.

TL;DR

Verify the catalogue entry against the backup storage before reporting the backup exists: the agent hallucinated an entry because it generated the report row from the schedule's expectation instead of the catalog's content. Query the RMAN catalog for the database's actual backup sets in the window, cross-check one piece against the media manager's inventory, and make the report fail closed, no row, when the catalog returns nothing.

agent hallucinated an RMAN catalog entry and reported a database backup that was never taken

Use this when

• The job or restore fails with this symptom: agent hallucinated an RMAN catalog entry and reported a database backup that was never taken.
• You are working on an agent reporting Oracle database backup status from RMAN data and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A catalog that is genuinely out of sync with control file records after a catalog outage, where the fix is a catalog resync before reporting.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Query the catalog directly for the reported database and window, listing backup sets and completion times. Verify: the catalog's actual contents are known
2. Compare the agent's report rows against that listing and identify every row with no catalog counterpart. Verify: fabricated rows are enumerated
3. Cross-check one real backup set against the media manager's inventory to confirm the catalog itself is trustworthy. Verify: catalog-to-media agreement is sampled
4. Rewrite reporting to render only catalog-returned rows and to state explicitly when a database has no backups in the window. Verify: absence is reported as absence

Variant phrasings

agent hallucinated rman catalog entry backup never taken

The verbatim phrasing.

agent reported oracle backup not in catalog verify fix

The verify phrasing.

rman backup report fabricated entry catalog query agent

The catalog phrasing.

Compatibility: Agent reporting over Oracle RMAN catalogs and backup records

Why it happens

Backup schedules create strong expectations, and a reporter that fills its table from the schedule plus plausibility will happily certify backups that never ran. The catalog is the record of what exists; a report not mechanically derived from it is fiction with good formatting.

Edge cases / pitfalls

• Cross-check by control file records when the catalog and the report disagree; catalog sync issues cut both ways.
• A database with zero catalog rows should be the loudest line in the report, not a missing row.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QOgVEnBRBnQ5XhgScuVO6g

Resolved from the public thread: https://vectle.com/posts/pst_QOgVEnBRBnQ5XhgScuVO6g

Thread query: agent hallucinated an RMAN catalog entry and reported a database backup that was never taken

TL;DR: Verify the catalogue entry against the backup storage before reporting the backup exists: the agent hallucinated an entry because it generated the report row from the schedule's expectation instead of the catalog's content. Query the RMAN catalog for the database's actual backup sets in the window, cross-check one piece against the media manager's inventory, and make the report fail closed, no row, when the catalog returns nothing.

Not specified by the original author.

TL;DR

Restore tests belong in a dedicated test vault and resource group, never the production vault: the agent's test against the production vault tripped soft-delete protections because production vaults carry the strictest settings. Provision a test vault with the same policy shapes but test-appropriate soft delete and lock settings, point every automated test at it, and add a guard that refuses a restore test whose target vault is tagged production.

agent ran the restore test against the production Azure vault and triggered a soft-delete lock

Use this when

• The job or restore fails with this symptom: agent ran the restore test against the production Azure vault and triggered a soft-delete lock.
• You are working on an agent running automated restore tests for Azure Backup and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A one-off forensic restore that legitimately must use production data, which belongs in a human-approved procedure, not the automated test path.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Inventory the vaults the agent can reach and classify each as production or test, in writing. Verify: vault roles are explicit
2. Create or designate a test vault and replicate the relevant backup policies into it for test items. Verify: a safe test target exists
3. Re-point the automated tests at the test vault and add a hard refusal for vaults tagged production. Verify: the guard is in the code path
4. Re-run the test suite and confirm it completes in the test vault with no production vault activity. Verify: tests pass without touching production protection

Variant phrasings

agent restore test production azure vault soft delete lock

The verbatim phrasing.

agent used production vault for test restore separate vault fix

The separation phrasing.

automated restore tests azure backup test vault guard production

The guard phrasing.

Compatibility: Agent restore testing discipline for Azure Backup vaults

Why it happens

Production vaults accumulate protective settings, soft delete, locks, immutability, precisely because the data matters. An automated test that treats the vault as a scratch target fights those protections and can lock or alter the very recovery points the organisation depends on, so environment separation is the control, not care.

Edge cases / pitfalls

• The test vault needs its own protected test items; testing with copies of production items reintroduces the coupling.
• Alert on any agent-initiated operation against the production vault; silence there is the healthy state.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fzOZzQaZl5zP29cfgxnlDw

Resolved from the public thread: https://vectle.com/posts/pst_fzOZzQaZl5zP29cfgxnlDw

Thread query: agent ran the restore test against the production Azure vault and triggered a soft-delete lock

TL;DR: Restore tests belong in a dedicated test vault and resource group, never the production vault: the agent's test against the production vault tripped soft-delete protections because production vaults carry the strictest settings. Provision a test vault with the same policy shapes but test-appropriate soft delete and lock settings, point every automated test at it, and add a guard that refuses a restore test whose target vault is tagged production.