churn agent hallucinating save-play outcomes not in the ticket history: how to fix

TL;DR

Hallucinated outcomes appear when the agent is asked to report results it never retrieved, and fills the gap fluently. Ground the workflow: outcomes may only be stated from ticket or CRM records fetched in the run, every reported outcome carries its source record id, and a verification pass rejects any outcome statement without a matching record before the report ships.

churn agent hallucinating save-play outcomes not in the ticket history: how to fix

Use this when

• Agent reports claim calls happened, credits were issued, or tickets resolved that records do not show
• The agent's task includes reporting on data it was not given
• Fabrications are plausible and specific, making them hard to spot

Not for this skill when

• The records exist but live in a system the agent cannot access, an access problem first
• The agent misread a real record, a comprehension error rather than invention

Steps

1. Audit one report: list every outcome claim and search the ticket and CRM history for its record. Verify: each claim is marked grounded or invented with counts
2. Find the step where the agent reports beyond its retrieved data (missing tool call, skipped fetch, or a prompt asking for a narrative). Verify: the invention point in the workflow is located
3. Require source record ids on every outcome statement and validate them against the systems before sending. Verify: a test report with a fabricated outcome is rejected by validation
4. Change the task framing: the agent must state not found when records are absent, and that answer is treated as success. Verify: the incentive to fill gaps is removed
5. Sample-audit reports weekly against records and track the grounded rate. Verify: grounding quality is measured over time, not assumed after one fix

Variant phrasings

agent hallucinating save play outcomes

A related phrasing for the same underlying problem.

ungrounded claims in agent report

A related phrasing for the same underlying problem.

require source records for outcomes

A related phrasing for the same underlying problem.

Compatibility: Compatibility: LLM agents with tool access to helpdesk and CRM records, current versions.

Why it happens

Models complete patterns; a report with a missing outcome looks incomplete, so the pattern completes it. The cure is structural, not exhortation: only retrieved records may be cited, citations are validated mechanically, and saying not found is rewarded as a correct result.

Edge cases / pitfalls

• Validation should check the record supports the claim, not merely that the id exists; a real ticket can be cited for a claim it does not make.
• Temperature and prompt tweaks reduce but never eliminate invention. Keep the mechanical check regardless.
• Fabricated outcomes already sent to customers need a correction pass; find them via the same audit and correct them proactively.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UOqoRJntQ2ZsTczdFMPXBw

Resolved from the public thread: https://vectle.com/posts/pst_UOqoRJntQ2ZsTczdFMPXBw

Thread query: churn agent hallucinating save-play outcomes not in the ticket history: how to fix

TL;DR: Hallucinated outcomes appear when the agent is asked to report results it never retrieved, and fills the gap fluently. Ground the workflow: outcomes may only be stated from ticket or CRM records fetched in the run, every reported outcome carries its source record id, and a verification pass rejects any outcome statement without a matching record before the report ships.