agent's summary hallucinated a healthy backup despite the error in the logs

TL;DR

A summary that calls an error-filled backup healthy is generated from vibes, not data, usually because the summariser receives raw logs and a cheerful template. Feed the summariser only the parsed outcome records, force it to cite the decisive status lines for every claim, and add a post-check that rejects any summary whose health claim conflicts with the parsed outcomes.

agent's summary hallucinated a healthy backup despite the error in the logs

Use this when

• Stakeholders receive healthy summaries for nights the console shows failed.
• Summaries omit failed objects that the parser did extract correctly.
• Summary tone stays positive regardless of input severity.

Not for this skill when

• The parser never found the errors in rotated or multiline logs. Fix extraction first.
• The summary is accurate but distribution sends an old cached version. Delivery problem.

Steps

1. Trace one bad summary back through its inputs to see whether parsed outcomes or the summariser caused the lie. Verify: the faulty stage is identified
2. Restrict summariser input to parsed outcome records with counts and per-object statuses. Verify: raw logs no longer reach the summariser directly
3. Require each health claim in the summary to reference the outcome records behind it. Verify: claims map to records in a sample output
4. Add a consistency gate comparing summary outcome words against parsed counts and blocking mismatches. Verify: a seeded mismatch is blocked in testing
5. Regenerate summaries for recent affected nights and issue corrections where stakeholders were misled. Verify: corrected summaries are distributed

Variant phrasings

agent summary contradicts logs healthy false

The contradiction phrasing.

llm summary hallucination backup status gate

The gate variant.

summary generation from parsed outcomes only

The input wording.

Compatibility: LLM-assisted reporting agents summarising backup outcomes for tickets, chat and email digests from structured parser output.

Why it happens

Summarisation models smooth toward the statistically normal story, and most backup nights are healthy, so an unconstrained summariser literally expects success. Errors present as a few lines among thousands of normal ones get smoothed away. Constraining inputs to parsed outcomes and gating claims against counts removes the smoothing surface.

Edge cases / pitfalls

• Never let the summariser see the previous night summary. It anchors on yesterday health.
• Counts in summaries should be copied programmatically, not generated. Numbers from models are decorative.
• A blocked summary should alert, not silently retry until it passes. The block is the signal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4RQDM4oK4CKnbzingcyYA