agent's reconciliation summary cited transactions that were not in the source file: hallucinated matches in the...

TL;DR

Treat the whole report as untrusted, not just the invented lines: if the agent cited transactions that do not exist in the source file, its matching narrative was generated from plausibility rather than from the data, and the real exceptions may be misdescribed or missing too. Quarantine the report so nobody acts on it, then rebuild the exception list mechanically from the source file and the ledger - set differences computed by query, not narrated by the agent. Every transaction the new report cites must be verified to exist in the source by identifier lookup at generation time. Then change the agent workflow so narrative is written only over verified result sets: the agent receives the computed exceptions as its only material, and a post-generation check confirms every cited identifier resolves to a real record before the report can be published.

agent's reconciliation summary cited transactions that were not in the source file: hallucinated matches in the exception report

Use this when

• An agent-produced reconciliation summary or exception report references transactions absent from the source statement or feed.
• Cited amounts, dates or counterparties look plausible but cannot be found in the source data.
• Decisions or follow-ups were about to be based on the report.

Not for this skill when

• The report cites real transactions but draws a wrong conclusion about them. That is an analysis error; correct the analysis, not the pipeline's grounding.
• Transactions are missing from the source file itself. That is a feed completeness problem upstream of reporting.

Steps

1. Withdraw or clearly mark the faulty report wherever it was published or sent, and notify anyone who received it. Verify: no consumer of the report can act on it unknowingly
2. Recompute the exceptions directly from the source file and the ledger with a query or script, producing a verifiable list with identifiers. Verify: the true exception list exists independent of the agent's narrative
3. Diff the agent's cited transactions against the source file identifiers and document which citations were fabricated. Verify: the fabrication scope is measured and recorded in the incident log
4. Regenerate the summary from the verified exception set only, then programmatically check that every cited identifier exists in the source. Verify: the new report passes a full citation-existence check
5. Add the grounding controls to the workflow: source-bound result sets for narrative generation and a publish gate that rejects unverifiable citations. Verify: a test run with a planted unverifiable citation is blocked at the gate

Variant phrasings

agent reconciliation summary cited transactions not in source hallucinated matches

The fabricated citation symptom.

verify agent report citations exist source data publish gate

The verification-gate phrasing.

hallucinated exception report agent rebuild exceptions from source query

The rebuild procedure search.

Compatibility: Agent reporting workflows over bank feeds and ledgers in any ERP or reconciliation platform, where summaries are generated from computed result sets.

Why it happens

A language model asked to summarise matching results will produce fluent specifics even when its input was truncated, empty or partially loaded, because fluency is its default and grounding is a workflow property, not a model property. If the pipeline hands the agent a vague brief instead of the actual exception rows, invented transaction details fill the vacuum and read exactly like real ones. The failure therefore belongs to the workflow design: narrative generation must be the last step over verified data, with citation checks the agent cannot talk its way past.

Edge cases / pitfalls

• Partial fabrication is the dangerous case: mostly real reports with a few invented items evade casual review, so checks must cover every citation.
• If the fabricated report reached auditors or a regulator, follow your disclosure process; silent replacement compounds the problem.
• Keep the faulty report in the incident file as the test case for the publish gate.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstZN3mGuoQPhlzvUhuyikKA