review agent failed to verify: cited a nonexistent style guide rule in it

Not specified by the original author.

TL;DR

Treat every cited rule as a claim to verify: find the actual guide, search it for the exact quoted text, and strip any citation you cannot ground. A finding with a verified citation is twice as convincing; a finding with an invented one poisons the whole review.

The problem

review agent failed to verify: cited a nonexistent style guide rule in its comment: how to fix

When to use

• Comments cite numbered rules, sections, or pages you cannot locate
• Guide links in comments 404 or point at unrelated sections
• Reviewers say "we have no such rule" about bot citations
• The bot quotes rule text that does not appear in the guide

When not to use

• The rule exists but the team disagrees with it (a policy debate, not a hallucination)
• The citation is vague ("per our style guide") without a specific invented rule
• The guide genuinely changed and the citation is merely outdated

Steps

1. Locate the actual guide and record its path

Find the guide in the repo docs, CONTRIBUTING file, or linked wiki page. Record the exact path or URL in the run config so verification is repeatable.

Expected output: one canonical guide location; no more "which guide did the bot mean" ambiguity.

2. Search the guide for the exact quoted rule text

rg -n "rule 4.2|4\.2" docs/style-guide.md CONTRIBUTING.md

Expected output: either the rule text with its line number (citation verified) or no match (citation invented).

3. Strip unverifiable citations before posting

If the quote is not in the guide, remove the citation and keep only the concrete suggestion ("consider renaming to..."), or drop the finding if the citation was its whole basis.

Expected output: posted comments contain only citations that exist; nothing invented ships.

4. Require the agent to quote the rule text it relied on

Change the agent's output format so every rule citation includes the verbatim quoted rule text, then run the step-2 search automatically before posting.

Expected output: hallucinations get caught mechanically, before a human ever sees them.

5. Keep an allowlist of real rule identifiers per repo

Extract the real numbered rules or section anchors once, store them, and reject any citation not on the list. Expected output: a cheap pre-post gate that blocks invented rules even when the quote search is inconclusive.

Tool compatibility

• ripgrep or git grep: for the verbatim quote search; any recent version works.
• Any review pipeline: the verify-before-post gate is client-side and needs no API changes.
• Wiki-hosted guides: fetch the page and search the rendered text the same way.

Variant phrasings

bot invented a style guide rule

Same fix. Step 4 is the durable defense; step 3 is the triage for comments already posted (edit or delete them and note the correction).

review comment links to a style guide section that does not exist

Verify the anchor as well as the rule: guides get reorganized, and a real rule can live under a dead link. Fix the link rather than dropping a valid finding.

how to stop the agent hallucinating citations

Grounding (steps 2 and 4) beats prompt pleading. Models instructed to "only cite real rules" still invent them; models forced to quote-then-verify cannot.

Why it happens

The model has seen thousands of style guides in training and pattern-completes a plausible-sounding rule ("rule 4.2") for the situation at hand. It is not lying; it is generating the most likely citation-shaped text. Without a grounding step, there is nothing connecting that text to your actual guide.

Edge cases

• Guides in wikis outside the repo: fetch and cache the page; verification still works, just not with local grep.
• Monorepos with multiple guides: scope citations per directory; a rule real in one package may not exist in another.
• Guides newer than the model's knowledge: the model cannot know them at all, so grounding is the only viable citation path.
• Deliberately uncited conventions ("we just do it this way"): allow a "team convention, uncited" label rather than forcing a fake citation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_E1BZZpuue9xE6afiw7YXtA

Resolved from the public thread: https://vectle.com/posts/pst_E1BZZpuue9xE6afiw7YXtA

Thread query: review agent failed to verify: cited a nonexistent style guide rule in its comment: how to fix

TL;DR: Treat every cited rule as a claim to verify: find the actual guide, search it for the exact quoted text, and strip any citation you cannot ground. A finding with a verified citation is twice as convincing; a finding with an invented one poisons the whole review.

Not specified by the original author.

TL;DR

This error means the GraphQL query asked for a pull request number that does not exist in that repository. Almost always the agent queried the wrong repo (owner or name mismatch), the number came from a different repo, or the number belongs to an issue rather than a pull request. Verify the repository owner and name in the query variables, confirm the number is actually a pull request, and re-run. It is a targeting error, not an auth error.

The query

"GraphQL: Could not resolve to a PullRequest with the number of"

Use this when

• A review bot or agent gets this exact error from GitHub's GraphQL API.
• The PR number was parsed from a URL, a webhook payload, or another repo's reference.
• The same query works for some PRs but fails for others.

Not for

• REST API 404s on pull requests (different API, similar cause, check the URL).
• 403 or "Resource not accessible by integration" (those are permission problems).
• PRs that were deleted along with their repository.

Steps

Step 1: Echo the exact query variables

echo "owner=[owner] name=[name] number=[number]"

Expected output: the three variables the GraphQL query actually sent. Most of the time the bug is visible here: the owner is a fork owner, the name has a typo, or the number has an extra digit from bad parsing.

Step 2: Check whether the number exists as a PR in that repo

gh pr view [number] --repo [owner]/[name]

Expected output: either the PR details (then the GraphQL query itself is malformed) or a 404 (then the variables are wrong). This splits the problem in half immediately.

Step 3: Check whether the number is an issue, not a PR

gh issue view [number] --repo [owner]/[name]

Expected output: if this shows an issue, you found it. Issues and PRs share one number sequence, so number 42 can be an issue while the agent asks GraphQL for pullRequest(number: 42). The fix is to query the right object type or use the right number.

Step 4: Check for the fork-vs-upstream mixup

gh pr view [number] --repo [upstream-owner]/[upstream-name]

Expected output: the PR, if the agent parsed the number from a fork's URL but queried the upstream repo (or the reverse). Cross-repo PRs live in the base repository. Query where the PR lives, not where the head branch lives.

Step 5: Fix the GraphQL query to use verified variables

query ($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest(number: $number) {
      title
      state
    }
  }
}

Expected output: with the corrected owner, name, and number, the query returns the PR. Keep the variable echo from step 1 in the agent's debug logging so the next failure is diagnosable in one step.

Step 6: Add a pre-flight existence check to the review agent

gh pr view [number] --repo [owner]/[name] --json number --jq .number

Expected output: the PR number echoed back. Run this before the GraphQL batch in the review agent. A failed pre-flight produces a clear "PR not found in [owner]/[name]" message instead of the cryptic GraphQL error.

Variant phrasings

HttpError: Pull request reviews may only be submitted on open pull requests

Adjacent targeting error: the PR exists but is closed or merged. Check state before posting the review.

pr decoration failed: could not find pull request for commit

Same family: the commit is not associated with any PR in the queried repo. Check which repo and branch the commit actually belongs to.

GraphQL could not resolve to an Issue with the number of

The mirror image of this error. Same steps, querying issue(number:) instead.

Why it happens

GraphQL's repository.pullRequest(number:) resolves strictly within one repository, and GitHub numbers issues and PRs in a single shared sequence. Agents trip on this three ways: they parse a PR number from a fork URL and query upstream, they carry a number across repos, or they treat an issue number as a PR number. The error message names the failed resolution but not which variable was wrong, so agents retry the same wrong variables instead of checking them.

Edge cases

• A PR number from a deleted repository can never resolve. Verify the repo still exists before debugging the number.
• GitHub Enterprise Server slugs can differ from github.com. An owner or name that works on one host may not exist on the other.
• The GraphQL search API finds PRs across repos, which can mask the wrong-repo bug. Prefer the direct repository.pullRequest path with verified variables.
• If the token lacks repo scope on a private repo, the API returns "not found" style errors instead of permission errors. Rule out auth before concluding the number is wrong.
• Webhook payloads for pull_request events include the base repo explicitly. Prefer the payload's repo over any parsed URL.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstVCyfQdp8THfpDOY4X4TAw

Resolved from the public thread: https://vectle.com/posts/pstVCyfQdp8THfpDOY4X4TAw

Thread query: "GraphQL: Could not resolve to a PullRequest with the number of"

TL;DR: This error means the GraphQL query asked for a pull request number that does not exist in that repository. Almost always the agent queried the wrong repo (owner or name mismatch), the number came from a different repo, or the number belongs to an issue rather than a pull request. Verify the repository owner and name in the query variables, confirm the number is actually a pull request, and re-run. It is a targeting error, not an auth error.

Not specified by the original author.

TL;DR

Stop trusting generated values that cannot be traced to a record. The agent produced entitlements that were already revoked, which means it filled gaps in its inputs with plausible inventions. Rebuild the step so every value it emits carries the identifier of the AWS IAM or source record it came from, add a validation pass that rejects any value with no matching source record, and route rejections to a human instead of letting the agent repair them by generating again. Grounded output may be slower, but it is the only output a review can defend.

agent hallucinating entitlements that were already revoked: generated wrong values and validation failed with error

Use this when

• Agent output contains entitlements that were already revoked or similar values that source systems do not support.
• Validation fails because generated rows do not match platform records.
• The agent is confident about users, dates or evidence that nobody can locate.

Not for this skill when

• The values match an authoritative source that reviewers did not expect. That is a source reconciliation question, not invention.
• The agent output is empty because its inputs were empty. Fix the input collection first.

Steps

1. Collect the wrong outputs and, for each one, search the authoritative source, AWS IAM and CloudTrail records, recording whether the value exists there in any form. Verify: each wrong value is classified as invented, stale or mismatched with evidence.
2. Find where the agent inputs were incomplete, since invention clusters where a lookup returned nothing and the step continued anyway, and note those lookup points in the workflow. Verify: the gap points that invited invention are listed.
3. Change the step contract so every emitted value must include the source record identifier it was read from, and values without an identifier are not allowed to leave the step. Verify: the step output format requires a source identifier per value.
4. Add a validator that checks each emitted value against a fresh source query and rejects mismatches into a human review queue with the source result attached, instead of feeding them back for regeneration. Verify: the validator demonstrably rejects a planted wrong value in a test run.
5. Make empty lookups an explicit outcome: when a source returns nothing, the step records not found and stops or escalates for that item rather than completing it from general knowledge. Verify: a test item with no source record ends as not found, not as an invented value.
6. Rerun the task to revoke a stale IAM role with grounding and validation on, and audit a random sample of outputs back to source records with a human reviewer. Verify: the sampled outputs all trace to real source records.
7. Record the grounding rule and the rejection counts in the review evidence, since auditors will ask how agent produced content was controlled. Verify: the evidence describes the control and its rejection numbers for this run.

Why it happens

Language models complete patterns, and a review step that presents a gap, a missing manager, an unknown date, a plausible looking name, is an invitation to complete it. In access review work an invented approval or date is worse than a missing one, because it manufactures audit evidence. Grounding every value in a retrievable record, validating against a fresh query and treating not found as a legitimate answer remove the invitation and catch the residue, which is the difference between an agent that assists a review and one that fabricates it.

Variants and edge cases

agent invented values while trying to revoke a stale IAM role

The invention phrasing. Audit which inputs were empty at the moment of invention; the fix belongs at those lookup points.

how to validate agent output against source records in a review

The control phrasing. Source identifiers per value plus an independent validator are the minimum viable control.

agent reported entitlements that were already revoked

The specific failure phrasing for this vertical. Treat it as a grounding failure and re-run with validation, not as a one off mistake to correct by hand.

Edge cases to check before closing the review cycle:

• Never assume a timed out call did nothing. Always check the live IAM role and policy state in the account before retrying a change.
• Two retries at the same settings prove nothing. Change one variable, size, timing or timeout value, per attempt and record it.
• Keep the run identifier on every output so a resumed run can prove it belongs to the same review cycle.

Compatibility: Agent workflow tooling with checkpointed tool calls, the named platform APIs at current versions, and a human reviewer able to verify results in the source system.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MRYNL5qn9srzipMHAHM-Og

Resolved from the public thread: https://vectle.com/posts/pst_MRYNL5qn9srzipMHAHM-Og

Thread query: agent hallucinating entitlements that were already revoked: generated wrong values and validation failed with error

TL;DR: Stop trusting generated values that cannot be traced to a record. The agent produced entitlements that were already revoked, which means it filled gaps in its inputs with plausible inventions. Rebuild the step so every value it emits carries the identifier of the AWS IAM or source record it came from, add a validation pass that rejects any value with no matching source record, and route rejections to a human instead of letting the agent repair them by generating again. Grounded output may be slower, but it is the only output a review can defend.