## TL;DR

The agent's summary mentioned files that are not in the PR diff  -  a hallucination. Ground summary generation in the real file list from the diff, and validate every named file against it before posting.

## Error

```text
Agent summary: "Changes to auth.py, billing.py, and cache.py look good."
PR files changed: auth.py, billing.py (cache.py not in the diff).
```

## Steps

1. Confirm the hallucination: compare the summary's file list with the PR's files-changed list. Expected: at least one named file is absent from the diff.
2. Fix the prompt/pipeline: feed the agent the exact file list from the diff API and instruct it to mention only those files. Expected: the agent cannot invent files.
3. Add a post-generation validator: extract filenames from the summary and check each against the diff file list; drop or flag mismatches. Expected: hallucinations are caught before posting.
4. Regenerate the summary. Expected: every named file exists in the diff.

## When to use

- The agent's review summary names files not present in the PR diff.
- The summary is LLM-generated.

## When not to use

- Files are in the diff but the comments are on wrong lines  -  that is a position-mapping bug.
- A human wrote the summary; this is an agent-pipeline fix.

## Tool compatibility

- Any LLM-based review agent.
- GitHub / GitLab diff APIs.

## Variant phrasings

### Summary invents functions, not files

Same fix at symbol level: ground in the parsed diff hunks.

### Summary describes reverted changes

Summarize the final diff, not the commit sequence  -  rebase the view onto the head diff.

## Why it happens

Language models complete patterns fluently, including plausible-but-wrong filenames. Without grounding, the summary reads convincingly while describing code that was never changed  -  worse than no summary.

## Edge cases

- The validator should be strict: a summary mentioning an absent file is always wrong, never stylistic.
- Filenames with similar prefixes (auth.py vs auth_test.py) are the most common confabulation  -  the validator catches these cheaply.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7F5xEN2KcVhuJjayZw5UVw