code review agent hallucinating vulnerabilities that are not in the diff

TL;DR: Require every vulnerability finding to quote an exact changed line, then programmatically verify each quote against the actual diff and drop anything that does not match. Models pattern-match on scary-looking code shapes and describe vulnerabilities they expect to see rather than ones present in the diff. Grounding each claim in a verified quote kills the phantoms.

code review agent hallucinating vulnerabilities that are not in the diff

1. Collect every vulnerability finding and its cited location. Expected: a list of claims, each pointing at a file and line.

2. Verify each cited line exists in the actual diff: extract the diff's changed lines and check that the quote matches verbatim. Expected: each claim either matches a real changed line or is flagged as unverified.

3. Drop every finding whose cited code is not in the diff. No exceptions, no "it might be nearby". Expected: the finding list shrinks to only grounded claims.

4. For the survivors, check the claim against the real code semantics (is that input actually unsanitized?). Expected: each remaining finding is both grounded and correct.

5. Tighten the review instructions: require exact code quotes for every finding and forbid claims about code outside the diff. Expected: the next run produces fewer, better findings.

6. Track the phantom rate over runs: dropped findings divided by total findings. Expected: the rate trends down; if it does not, the instructions need another pass.

Use this when

• the agent reports vulnerabilities that are not in the changed code
• security findings cite functions or patterns absent from the PR
• developers have started ignoring the agent's security comments

Not for this skill when

• the findings are real but low-severity (that is a tuning problem, not hallucination)
• the agent misses real vulnerabilities (that is recall, the opposite problem)
• a human wrote the findings (check the human, not the agent)

Variant phrasings

review bot reported SQL injection that isnt in the code

ai reviewer inventing security issues on the PR

phantom vulnerability findings in automated code review

Why it happens

Models pattern-match on scary-looking code shapes, and review instructions that emphasize security amplify it. Without a grounding requirement, the model completes the pattern: it describes the vulnerability it expects to see rather than the code in front of it. The stronger the instructions push on security, the more phantoms appear.

Edge cases

• A finding can cite a real line but misread it (a parameterized query called "unsanitized"); grounding is necessary but not sufficient, so keep the semantics check.
• Findings about unchanged context lines are legitimate only if the diff actually changed the risk; default to changed-lines-only.
• Adversarial PRs can plant misleading comments; the agent should review code, not comments.
• Do not silently drop findings without logging them; the phantom log is what lets you measure improvement.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yJED-GkzwJ-nBfXrKN9wTw