agent invented API endpoints in review comments: hallucination guard

TL;DR: Build a ground-truth endpoint inventory from the router definitions or the OpenAPI spec, extract every endpoint mention from the draft comments, and strip or correct anything not in the inventory before posting. Models know REST conventions and happily extrapolate - if the code has a user route, the model may discuss a per-user permissions route without checking. Endpoint-shaped text feels authoritative, so the invention survives unless something verifies it.

agent invented API endpoints in review comments: hallucination guard

1. Build the ground-truth endpoint inventory: extract routes from the codebase's router definitions or parse the OpenAPI spec. Expected: a complete list of real endpoints with methods and paths.

2. Extract every endpoint mention from the agent's draft review comments. Expected: a list of claimed endpoints.

3. Check each claim against the inventory. Expected: each claim marked real or invented.

4. Strip or correct invented endpoints before posting; keep the rest of the comment if it still makes sense without them. Expected: posted comments reference only real endpoints.

5. Tighten the review instructions: the agent must quote the route definition or spec entry for any endpoint it discusses. Expected: future drafts cite sources for their endpoint claims.

6. Re-run the inventory extraction when the PR itself adds endpoints, so new routes are not flagged as invented. Expected: the guard knows about the PR's own new endpoints.

Use this when

• review comments mention API endpoints not present in the code
• the agent discusses versioned or renamed endpoints that never existed
• endpoint claims appear in security or breaking-change findings

Not for this skill when

• the endpoints are real but the concern is wrong (tune the review, not the guard)
• there is no route inventory to check against (build one first)
• a human wrote the comments (the guard is for agent output)

Variant phrasings

review bot cited an API route that doesnt exist

ai reviewer inventing REST endpoints in PR comments

hallucinated API paths in automated code review

Why it happens

Models absorb common REST conventions from training and extrapolate freely. An endpoint that "should" exist by convention gets discussed as if it does exist. Because endpoint paths look precise and technical, neither the agent nor a skimming human questions them, and the invention lands in a posted comment.

Edge cases

• Endpoints defined in config or annotations rather than code need a parser that understands that framework.
• The PR under review may rename endpoints; build the inventory from the PR's version of the code, not main.
• External third-party APIs mentioned in comments cannot be verified locally; mark them unverifiable rather than invented.
• GraphQL schemas need the same treatment as REST route tables.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0B6kxlQxkBXlqcoSExSEbQ