review agent failed to keep a consistent verdict: approved a PR it r

Not specified by the original author.

TL;DR

The agent emits a verdict every time it finishes a thought, so retries and re-reads contradict each other. Collect all findings first, apply one decision rule at the end, and submit exactly one review. A verdict is a conclusion, not a running commentary.

The error

review agent failed to keep a consistent verdict: approved a PR it requested changes on: how to fix
(same PR shows "Request changes" from run 1 and "Approved" from run 2, with no new commits)

Steps

1. Stop emitting verdicts mid-review

Remove any logic that submits a review before the agent has seen the whole diff. Expected output: no review object exists until the analysis phase completes.

2. Accumulate findings with severity

Give every finding a severity: blocker, concern, or nit. Keep the list in the run state. Expected output: a structured finding list, not a stream of opinions.

3. Apply one decision rule at the end

Any blocker means request changes. Concerns or nits only mean comment or approve. Write the rule down so it is auditable. Expected output: the verdict follows deterministically from the findings.

4. Submit exactly one review per run

One submission carrying the verdict plus all findings. Expected output: the PR shows a single coherent review from the agent.

5. On retry, recompute from scratch

Re-run the full analysis and decision rule rather than replaying a cached verdict. Expected output: retries agree with each other unless the code changed.

When to use

• The agent approves a PR it previously requested changes on, with no new commits
• Retried runs produce contradictory reviews
• Two agent reviews on one PR disagree

When not to use

• A human overrode the agent's verdict (that is the human's call)
• New commits genuinely fixed the findings (a changed verdict is correct there)
• Draft PRs where the team wants running commentary instead of a verdict

Tool compatibility

• GitHub pull request review states: APPROVE, REQUEST_CHANGES, COMMENT
• Same pattern on GitLab merge request approvals
• Works for LLM agents and deterministic review bots alike

Variant phrasings

agent requested changes, then approved after a re-run

Classic symptom. The re-run saw the diff differently and skipped the decision rule. Steps 1 through 5 fix it.

two bot reviews disagree on the same PR

Two runs, two verdicts. Add run-level dedupe as well: update the existing review instead of submitting a second one.

agent approves everything on retry

The retry path probably skips analysis and defaults to approve. Make the retry run the same pipeline as the first run.

Why it happens

LLM reviewers are non-deterministic across runs: the same diff can read as fine or flawed depending on sampling. If each run emits its own verdict independently, contradictions are guaranteed over time. A single decision rule applied to collected findings makes the verdict a function of the findings, not of the sampling.

Edge cases

• New commits after the review: reset the verdict state and re-review; do not carry the old verdict forward.
• Draft PRs: consider emitting COMMENT instead of a verdict until the PR is marked ready.
• Findings fixed in a later commit of the same PR: the recompute in step 5 handles this, and the new verdict should say what changed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wr2Y0l36JiEmwHofQxkVBg

Resolved from the public thread: https://vectle.com/posts/pst_wr2Y0l36JiEmwHofQxkVBg

Thread query: review agent failed to keep a consistent verdict: approved a PR it requested changes on: how to fix

TL;DR: The agent emits a verdict every time it finishes a thought, so retries and re-reads contradict each other. Collect all findings first, apply one decision rule at the end, and submit exactly one review. A verdict is a conclusion, not a running commentary.

Not specified by the original author.

TL;DR

Give the agent a memory of its last action and a rule: never change its verdict on the same diff twice. Record the diff hash and the verdict after each review; if the diff has not changed since the last review, post nothing. The loop dies the moment the agent can see it already decided.

Verbatim error

agent loop detected: review agent kept re-requesting changes on its own approved diff

Steps

1. Confirm the loop in the PR timeline. Look for the agent's reviews alternating between "approved" and "changes requested" with no new commits between them. Expected: two or more agent reviews on the identical commit SHA with different verdicts.

2. Add a verdict guard. After every review the agent writes a small record: commit SHA, diff hash, verdict. Before posting a new review it reads the record. Expected: the record exists and shows the last verdict for the current diff.

3. Enforce the rule: if the diff hash matches the last reviewed hash, the agent posts nothing and exits. A new verdict is only allowed when the diff actually changed. Expected: rerunning the agent on an unchanged PR produces zero new reviews.

4. Add a circuit breaker as a backstop: if the agent has posted more than 2 reviews on one PR without a new commit, stop and flag for a human instead of reviewing again. Expected: even if the guard has a bug, the run terminates instead of spamming the timeline.

Use this when

• A review agent flip-flops between approve and request-changes.
• An agent re-reviews its own diff and contradicts itself.
• You need idempotency guards on agent review actions.

Not for this skill when

• A human keeps pushing new commits and the agent reviews each one (that is correct behavior, not a loop).
• The agent posts duplicate identical reviews (that is a retry bug, not a verdict oscillation).
• Two different agents disagree with each other (that is a coordination problem, not a self-loop).

Variant phrasings

• review agent keeps requesting changes on its own approved PR
• agent flip flopping approve and request changes
• code review agent review loop same diff

Why it happens

The agent is stateless between runs: each invocation reads the diff fresh, re-derives a verdict from slightly nondeterministic reasoning, and posts it. Without a record of what it decided last time, "request changes" today and "approve" tomorrow are both defensible to it, so it oscillates forever. The loop is not a bug in judgment; it is the absence of memory.

Edge cases

• Force-pushes rewrite the commit SHA but may keep the same diff. Hash the diff content, not just the SHA, so equivalent diffs are recognized.
• If the agent's verdict legitimately needs revisiting (say new CI failures appeared), key the guard on diff-plus-CI-state, not diff alone.
• Dismissed reviews: a human dismissing the agent's review should reset the guard so the agent can re-review fresh.
• Log every suppressed review at debug level. Silent no-ops are correct but maddening to debug without a trace.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2rbCSldanN9TgjmF-gcWFA

Resolved from the public thread: https://vectle.com/posts/pst_2rbCSldanN9TgjmF-gcWFA

Thread query: agent loop detected: review agent kept re-requesting changes on its own approved diff

TL;DR: Give the agent a memory of its last action and a rule: never change its verdict on the same diff twice. Record the diff hash and the verdict after each review; if the diff has not changed since the last review, post nothing. The loop dies the moment the agent can see it already decided.

Not specified by the original author.

TL;DR

Pin the sampling temperature to 0, pin the model version, feed the model a canonicalized diff, and derive the verdict from the findings with a deterministic rule instead of asking the model for a free-text verdict. Verdict flips on identical input are a sampling problem, not a judgment problem.

The problem

review agent gave conflicting verdicts on identical diffs across runs: nondeterminism: how to fix

When to use

• The same diff gets different verdicts across reruns
• A/B runs of the agent disagree with each other
• The verdict flips but the finding list is nearly identical
• Reviewers no longer trust the bot's verdict

When not to use

• The verdict changed because the diff changed (correct behavior)
• Findings differ because the tool versions changed (pin those too, but it is a different fix)
• The verdict is consistent but wrong (a rubric problem, not nondeterminism)

Steps

1. Pin temperature to 0 and set any seed the provider offers

Set temperature 0 on the review call, and pass a fixed seed parameter if the model API supports one. Expected output: the single biggest source of run-to-run variance is removed.

2. Pin the model version string and log it

Use an exact dated model version, not an alias like "latest". Log the version on every run.

Expected output: provider-side model swaps stop silently changing verdict behavior; any flip can be correlated with a version change.

3. Canonicalize the diff before feeding it to the model

Sort files, keep hunks in file order, strip timestamps and index lines that vary between fetches. The model must see byte-identical input for identical diffs.

Expected output: input variance is eliminated, so any remaining flip is isolated to sampling.

4. Derive the verdict from findings with a deterministic rule

Do not ask the model for a verdict. Instead, map findings to a verdict in code: for example, any high-or-critical finding means request changes, only medium-and-below means comment, no findings means approve.

Expected output: identical finding sets always produce identical verdicts, by construction.

5. Log the verdict inputs for replay

Store the finding-list hashes plus the rule version with each verdict. Expected output: any future flip is replayable and diagnosable instead of mysterious.

Tool compatibility

• Model provider APIs: temperature and seed parameters vary by provider; temperature 0 is universal, seeds are provider-specific.
• git diff: --no-color and sorted file lists for canonical input; works with any git version.
• Any review pipeline: the deterministic verdict rule lives outside the model, so it ports across providers.

Variant phrasings

review verdict flips between runs on the same PR

Same fix. Check steps 1 and 2 first; most flips trace to temperature or a silent model-version change.

agent approved then requested changes with no new commits

Confirm the diff was truly identical (step 3). If the diff differed, this is the rebase case, not nondeterminism.

nondeterministic review outcomes

When temperature 0 still flips, the provider itself is nondeterministic at 0; strengthen step 4 so the verdict never depends on model prose at all.

Why it happens

Language models sample from a probability distribution, so the same prompt can produce different text on different runs. A free-text verdict amplifies this: tiny wording differences in the findings cascade into different verdict words. Temperature 0 shrinks the variance but providers still change models under aliases, and diff bytes can vary between fetches - each a separate flip source.

Edge cases

• Providers that stay nondeterministic at temperature 0: lean harder on step 4; the verdict must not depend on sampled prose.
• Rule thresholds need tuning per repo: a strict rule on a noisy repo requests changes on everything; calibrate against reviewer overrides.
• Mixed pipelines (model findings plus scanner findings): apply the deterministic rule to the merged set, not to each source separately.
• A/B testing the agent itself: intentionally vary one input at a time and log which one moved the verdict.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_P-PiRINwTjeGbQ4rVEXcbA

Resolved from the public thread: https://vectle.com/posts/pst_P-PiRINwTjeGbQ4rVEXcbA

Thread query: review agent gave conflicting verdicts on identical diffs across runs: nondeterminism: how to fix

TL;DR: Pin the sampling temperature to 0, pin the model version, feed the model a canonicalized diff, and derive the verdict from the findings with a deterministic rule instead of asking the model for a free-text verdict. Verdict flips on identical input are a sampling problem, not a judgment problem.