review agent failed to verify: cited a nonexistent style guide rule in its comment: how to fix

TL;DR

Treat every cited rule as a claim to verify: find the actual guide, search it for the exact quoted text, and strip any citation you cannot ground. A finding with a verified citation is twice as convincing; a finding with an invented one poisons the whole review.

The problem

review agent failed to verify: cited a nonexistent style guide rule in its comment: how to fix

When to use

• Comments cite numbered rules, sections, or pages you cannot locate
• Guide links in comments 404 or point at unrelated sections
• Reviewers say "we have no such rule" about bot citations
• The bot quotes rule text that does not appear in the guide

When not to use

• The rule exists but the team disagrees with it (a policy debate, not a hallucination)
• The citation is vague ("per our style guide") without a specific invented rule
• The guide genuinely changed and the citation is merely outdated

Steps

1. Locate the actual guide and record its path

Find the guide in the repo docs, CONTRIBUTING file, or linked wiki page. Record the exact path or URL in the run config so verification is repeatable.

Expected output: one canonical guide location; no more "which guide did the bot mean" ambiguity.

2. Search the guide for the exact quoted rule text

rg -n "rule 4.2|4\.2" docs/style-guide.md CONTRIBUTING.md

Expected output: either the rule text with its line number (citation verified) or no match (citation invented).

3. Strip unverifiable citations before posting

If the quote is not in the guide, remove the citation and keep only the concrete suggestion ("consider renaming to..."), or drop the finding if the citation was its whole basis.

Expected output: posted comments contain only citations that exist; nothing invented ships.

4. Require the agent to quote the rule text it relied on

Change the agent's output format so every rule citation includes the verbatim quoted rule text, then run the step-2 search automatically before posting.

Expected output: hallucinations get caught mechanically, before a human ever sees them.

5. Keep an allowlist of real rule identifiers per repo

Extract the real numbered rules or section anchors once, store them, and reject any citation not on the list. Expected output: a cheap pre-post gate that blocks invented rules even when the quote search is inconclusive.

Tool compatibility

• ripgrep or git grep: for the verbatim quote search; any recent version works.
• Any review pipeline: the verify-before-post gate is client-side and needs no API changes.
• Wiki-hosted guides: fetch the page and search the rendered text the same way.

Variant phrasings

bot invented a style guide rule

Same fix. Step 4 is the durable defense; step 3 is the triage for comments already posted (edit or delete them and note the correction).

review comment links to a style guide section that does not exist

Verify the anchor as well as the rule: guides get reorganized, and a real rule can live under a dead link. Fix the link rather than dropping a valid finding.

how to stop the agent hallucinating citations

Grounding (steps 2 and 4) beats prompt pleading. Models instructed to "only cite real rules" still invent them; models forced to quote-then-verify cannot.

Why it happens

The model has seen thousands of style guides in training and pattern-completes a plausible-sounding rule ("rule 4.2") for the situation at hand. It is not lying; it is generating the most likely citation-shaped text. Without a grounding step, there is nothing connecting that text to your actual guide.

Edge cases

• Guides in wikis outside the repo: fetch and cache the page; verification still works, just not with local grep.
• Monorepos with multiple guides: scope citations per directory; a rule real in one package may not exist in another.
• Guides newer than the model's knowledge: the model cannot know them at all, so grounding is the only viable citation path.
• Deliberately uncited conventions ("we just do it this way"): allow a "team convention, uncited" label rather than forcing a fake citation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_E1BZZpuue9xE6afiw7YXtA