agent review summary listed files not in the diff: hallucination
Fixes review agents hallucinating files in summaries by grounding the summary in the actual diff file list. Use when the summary names files the PR never touched. Not for wrong-line attributions.
TL;DR
The agent's summary mentioned files that are not in the PR diff - a hallucination. Ground summary generation in the real file list from the diff, and validate every named file against it before posting.
Error
Agent summary: "Changes to auth.py, billing.py, and cache.py look good."
PR files changed: auth.py, billing.py (cache.py not in the diff).Steps
- Confirm the hallucination: compare the summary's file list with the PR's files-changed list. Expected: at least one named file is absent from the diff.
- Fix the prompt/pipeline: feed the agent the exact file list from the diff API and instruct it to mention only those files. Expected: the agent cannot invent files.
- Add a post-generation validator: extract filenames from the summary and check each against the diff file list; drop or flag mismatches. Expected: hallucinations are caught before posting.
- Regenerate the summary. Expected: every named file exists in the diff.
When to use
- The agent's review summary names files not present in the PR diff.
- The summary is LLM-generated.
When not to use
- Files are in the diff but the comments are on wrong lines - that is a position-mapping bug.
- A human wrote the summary; this is an agent-pipeline fix.
Tool compatibility
- Any LLM-based review agent.
- GitHub / GitLab diff APIs.
Variant phrasings
Summary invents functions, not files
Same fix at symbol level: ground in the parsed diff hunks.
Summary describes reverted changes
Summarize the final diff, not the commit sequence - rebase the view onto the head diff.
Why it happens
Language models complete patterns fluently, including plausible-but-wrong filenames. Without grounding, the summary reads convincingly while describing code that was never changed - worse than no summary.
Edge cases
- The validator should be strict: a summary mentioning an absent file is always wrong, never stylistic.
- Filenames with similar prefixes (auth.py vs auth_test.py) are the most common confabulation - the validator catches these cheaply.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_7F5xEN2KcVhuJjayZw5UVw