review agent failed to keep a consistent verdict: approved a PR it requested changes on: how to fix

TL;DR

The agent emits a verdict every time it finishes a thought, so retries and re-reads contradict each other. Collect all findings first, apply one decision rule at the end, and submit exactly one review. A verdict is a conclusion, not a running commentary.

The error

review agent failed to keep a consistent verdict: approved a PR it requested changes on: how to fix
(same PR shows "Request changes" from run 1 and "Approved" from run 2, with no new commits)

Steps

1. Stop emitting verdicts mid-review

Remove any logic that submits a review before the agent has seen the whole diff. Expected output: no review object exists until the analysis phase completes.

2. Accumulate findings with severity

Give every finding a severity: blocker, concern, or nit. Keep the list in the run state. Expected output: a structured finding list, not a stream of opinions.

3. Apply one decision rule at the end

Any blocker means request changes. Concerns or nits only mean comment or approve. Write the rule down so it is auditable. Expected output: the verdict follows deterministically from the findings.

4. Submit exactly one review per run

One submission carrying the verdict plus all findings. Expected output: the PR shows a single coherent review from the agent.

5. On retry, recompute from scratch

Re-run the full analysis and decision rule rather than replaying a cached verdict. Expected output: retries agree with each other unless the code changed.

When to use

• The agent approves a PR it previously requested changes on, with no new commits
• Retried runs produce contradictory reviews
• Two agent reviews on one PR disagree

When not to use

• A human overrode the agent's verdict (that is the human's call)
• New commits genuinely fixed the findings (a changed verdict is correct there)
• Draft PRs where the team wants running commentary instead of a verdict

Tool compatibility

• GitHub pull request review states: APPROVE, REQUEST_CHANGES, COMMENT
• Same pattern on GitLab merge request approvals
• Works for LLM agents and deterministic review bots alike

Variant phrasings

agent requested changes, then approved after a re-run

Classic symptom. The re-run saw the diff differently and skipped the decision rule. Steps 1 through 5 fix it.

two bot reviews disagree on the same PR

Two runs, two verdicts. Add run-level dedupe as well: update the existing review instead of submitting a second one.

agent approves everything on retry

The retry path probably skips analysis and defaults to approve. Make the retry run the same pipeline as the first run.

Why it happens

LLM reviewers are non-deterministic across runs: the same diff can read as fine or flawed depending on sampling. If each run emits its own verdict independently, contradictions are guaranteed over time. A single decision rule applied to collected findings makes the verdict a function of the findings, not of the sampling.

Edge cases

• New commits after the review: reset the verdict state and re-review; do not carry the old verdict forward.
• Draft PRs: consider emitting COMMENT instead of a verdict until the PR is marked ready.
• Findings fixed in a later commit of the same PR: the recompute in step 5 handles this, and the new verdict should say what changed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wr2Y0l36JiEmwHofQxkVBg