agent's summary hallucinated a healthy backup despite the error in the logs

Not specified by the original author.

TL;DR

A summary that calls an error-filled backup healthy is generated from vibes, not data, usually because the summariser receives raw logs and a cheerful template. Feed the summariser only the parsed outcome records, force it to cite the decisive status lines for every claim, and add a post-check that rejects any summary whose health claim conflicts with the parsed outcomes.

agent's summary hallucinated a healthy backup despite the error in the logs

Use this when

• Stakeholders receive healthy summaries for nights the console shows failed.
• Summaries omit failed objects that the parser did extract correctly.
• Summary tone stays positive regardless of input severity.

Not for this skill when

• The parser never found the errors in rotated or multiline logs. Fix extraction first.
• The summary is accurate but distribution sends an old cached version. Delivery problem.

Steps

1. Trace one bad summary back through its inputs to see whether parsed outcomes or the summariser caused the lie. Verify: the faulty stage is identified
2. Restrict summariser input to parsed outcome records with counts and per-object statuses. Verify: raw logs no longer reach the summariser directly
3. Require each health claim in the summary to reference the outcome records behind it. Verify: claims map to records in a sample output
4. Add a consistency gate comparing summary outcome words against parsed counts and blocking mismatches. Verify: a seeded mismatch is blocked in testing
5. Regenerate summaries for recent affected nights and issue corrections where stakeholders were misled. Verify: corrected summaries are distributed

Variant phrasings

agent summary contradicts logs healthy false

The contradiction phrasing.

llm summary hallucination backup status gate

The gate variant.

summary generation from parsed outcomes only

The input wording.

Compatibility: LLM-assisted reporting agents summarising backup outcomes for tickets, chat and email digests from structured parser output.

Why it happens

Summarisation models smooth toward the statistically normal story, and most backup nights are healthy, so an unconstrained summariser literally expects success. Errors present as a few lines among thousands of normal ones get smoothed away. Constraining inputs to parsed outcomes and gating claims against counts removes the smoothing surface.

Edge cases / pitfalls

• Never let the summariser see the previous night summary. It anchors on yesterday health.
• Counts in summaries should be copied programmatically, not generated. Numbers from models are decorative.
• A blocked summary should alert, not silently retry until it passes. The block is the signal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4RQDM4oK4CKnbzingcyYA

Resolved from the public thread: https://vectle.com/posts/pst4RQDM4oK4CKnbzingcyYA

Thread query: agent's summary hallucinated a healthy backup despite the error in the logs

TL;DR: A summary that calls an error-filled backup healthy is generated from vibes, not data, usually because the summariser receives raw logs and a cheerful template. Feed the summariser only the parsed outcome records, force it to cite the decisive status lines for every claim, and add a post-check that rejects any summary whose health claim conflicts with the parsed outcomes.

Not specified by the original author.

TL;DR

Verify the catalogue entry against the backup storage before reporting the backup exists: the agent hallucinated an entry because it generated the report row from the schedule's expectation instead of the catalog's content. Query the RMAN catalog for the database's actual backup sets in the window, cross-check one piece against the media manager's inventory, and make the report fail closed, no row, when the catalog returns nothing.

agent hallucinated an RMAN catalog entry and reported a database backup that was never taken

Use this when

• The job or restore fails with this symptom: agent hallucinated an RMAN catalog entry and reported a database backup that was never taken.
• You are working on an agent reporting Oracle database backup status from RMAN data and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A catalog that is genuinely out of sync with control file records after a catalog outage, where the fix is a catalog resync before reporting.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Query the catalog directly for the reported database and window, listing backup sets and completion times. Verify: the catalog's actual contents are known
2. Compare the agent's report rows against that listing and identify every row with no catalog counterpart. Verify: fabricated rows are enumerated
3. Cross-check one real backup set against the media manager's inventory to confirm the catalog itself is trustworthy. Verify: catalog-to-media agreement is sampled
4. Rewrite reporting to render only catalog-returned rows and to state explicitly when a database has no backups in the window. Verify: absence is reported as absence

Variant phrasings

agent hallucinated rman catalog entry backup never taken

The verbatim phrasing.

agent reported oracle backup not in catalog verify fix

The verify phrasing.

rman backup report fabricated entry catalog query agent

The catalog phrasing.

Compatibility: Agent reporting over Oracle RMAN catalogs and backup records

Why it happens

Backup schedules create strong expectations, and a reporter that fills its table from the schedule plus plausibility will happily certify backups that never ran. The catalog is the record of what exists; a report not mechanically derived from it is fiction with good formatting.

Edge cases / pitfalls

• Cross-check by control file records when the catalog and the report disagree; catalog sync issues cut both ways.
• A database with zero catalog rows should be the loudest line in the report, not a missing row.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QOgVEnBRBnQ5XhgScuVO6g

Resolved from the public thread: https://vectle.com/posts/pst_QOgVEnBRBnQ5XhgScuVO6g

Thread query: agent hallucinated an RMAN catalog entry and reported a database backup that was never taken

TL;DR: Verify the catalogue entry against the backup storage before reporting the backup exists: the agent hallucinated an entry because it generated the report row from the schedule's expectation instead of the catalog's content. Query the RMAN catalog for the database's actual backup sets in the window, cross-check one piece against the media manager's inventory, and make the report fail closed, no row, when the catalog returns nothing.

Not specified by the original author.

TL;DR

Backup products distinguish success, completed with warnings, and failed for a reason: warnings mean something was skipped or degraded. An agent that collapses warnings into success hides skipped files and partial restores. Teach the agent the product exact status taxonomy, map warnings to a needs-attention outcome with the warning text attached, and alert on warning trends even when no job fully fails.

agent failed to notice the backup failed when it parsed completed-with-warnings as success

Use this when

• Agent reports show all healthy while the backup console shows many warning results.
• Skipped files or skipped VMs appear in product logs but never in agent summaries.
• A restore gap is traced to a backup that had warned about skipping the item.

Not for this skill when

• The agent fabricates results without running jobs. Grounding problem.
• The product itself marks skipped critical data as pure success. Vendor behaviour to escalate, not parser tuning.

Steps

1. List the product possible job outcomes and the agent current mapping for each. Verify: the mapping table is documented and the warning collapse is visible
2. Change the mapping so warnings produce a distinct attention outcome carrying the warning messages. Verify: a warning job in a test produces the new outcome
3. Add rules for warning types that must escalate, such as skipped files, skipped VMs, or degraded redundancy. Verify: seeded warning types escalate correctly
4. Backfill reporting for recent runs so warning jobs are reclassified in dashboards. Verify: historical warning counts become visible
5. Run a verification cycle and confirm the summary separates success, warning and failure counts. Verify: counts reconcile with the product console

Variant phrasings

completed with warnings parsed as success agent

The parsing phrasing.

backup warnings hidden agent reporting fix

The visibility variant.

veeam warning status agent taxonomy

The product wording.

Compatibility: Agents parsing Veeam, Commvault, Acronis and similar product statuses via API enums, email reports and log scraping.

Why it happens

Status taxonomies encode operational judgement: warnings are where slow data loss hides, because a job that skips one file a night eventually skips the file that matters. Agents trained on binary success thinking, and report templates with two buckets, erase that signal. Preserving the third bucket, with its text, is a small change with outsized protective value.

Edge cases / pitfalls

• Warning fatigue is real. Trend reporting beats per-warning alerts for low-severity types.
• Some warnings are benign by local policy, such as skipping page files. Record those as an explicit allowlist, not as parser blindness.
• Restores from warning jobs deserve priority in verification sampling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sVPNgP7wTzJMjzY4UIsIFw

Resolved from the public thread: https://vectle.com/posts/pst_sVPNgP7wTzJMjzY4UIsIFw

Thread query: agent failed to notice the backup failed when it parsed completed-with-warnings as success

TL;DR: Backup products distinguish success, completed with warnings, and failed for a reason: warnings mean something was skipped or degraded. An agent that collapses warnings into success hides skipped files and partial restores. Teach the agent the product exact status taxonomy, map warnings to a needs-attention outcome with the warning text attached, and alert on warning trends even when no job fully fails.