review agent gave conflicting verdicts on identical diffs across runs: nondeterminism: how to fix

TL;DR

Pin the sampling temperature to 0, pin the model version, feed the model a canonicalized diff, and derive the verdict from the findings with a deterministic rule instead of asking the model for a free-text verdict. Verdict flips on identical input are a sampling problem, not a judgment problem.

The problem

review agent gave conflicting verdicts on identical diffs across runs: nondeterminism: how to fix

When to use

• The same diff gets different verdicts across reruns
• A/B runs of the agent disagree with each other
• The verdict flips but the finding list is nearly identical
• Reviewers no longer trust the bot's verdict

When not to use

• The verdict changed because the diff changed (correct behavior)
• Findings differ because the tool versions changed (pin those too, but it is a different fix)
• The verdict is consistent but wrong (a rubric problem, not nondeterminism)

Steps

1. Pin temperature to 0 and set any seed the provider offers

Set temperature 0 on the review call, and pass a fixed seed parameter if the model API supports one. Expected output: the single biggest source of run-to-run variance is removed.

2. Pin the model version string and log it

Use an exact dated model version, not an alias like "latest". Log the version on every run.

Expected output: provider-side model swaps stop silently changing verdict behavior; any flip can be correlated with a version change.

3. Canonicalize the diff before feeding it to the model

Sort files, keep hunks in file order, strip timestamps and index lines that vary between fetches. The model must see byte-identical input for identical diffs.

Expected output: input variance is eliminated, so any remaining flip is isolated to sampling.

4. Derive the verdict from findings with a deterministic rule

Do not ask the model for a verdict. Instead, map findings to a verdict in code: for example, any high-or-critical finding means request changes, only medium-and-below means comment, no findings means approve.

Expected output: identical finding sets always produce identical verdicts, by construction.

5. Log the verdict inputs for replay

Store the finding-list hashes plus the rule version with each verdict. Expected output: any future flip is replayable and diagnosable instead of mysterious.

Tool compatibility

• Model provider APIs: temperature and seed parameters vary by provider; temperature 0 is universal, seeds are provider-specific.
• git diff: --no-color and sorted file lists for canonical input; works with any git version.
• Any review pipeline: the deterministic verdict rule lives outside the model, so it ports across providers.

Variant phrasings

review verdict flips between runs on the same PR

Same fix. Check steps 1 and 2 first; most flips trace to temperature or a silent model-version change.

agent approved then requested changes with no new commits

Confirm the diff was truly identical (step 3). If the diff differed, this is the rebase case, not nondeterminism.

nondeterministic review outcomes

When temperature 0 still flips, the provider itself is nondeterministic at 0; strengthen step 4 so the verdict never depends on model prose at all.

Why it happens

Language models sample from a probability distribution, so the same prompt can produce different text on different runs. A free-text verdict amplifies this: tiny wording differences in the findings cascade into different verdict words. Temperature 0 shrinks the variance but providers still change models under aliases, and diff bytes can vary between fetches - each a separate flip source.

Edge cases

• Providers that stay nondeterministic at temperature 0: lean harder on step 4; the verdict must not depend on sampled prose.
• Rule thresholds need tuning per repo: a strict rule on a noisy repo requests changes on everything; calibrate against reviewer overrides.
• Mixed pipelines (model findings plus scanner findings): apply the deterministic rule to the merged set, not to each source separately.
• A/B testing the agent itself: intentionally vary one input at a time and log which one moved the verdict.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_P-PiRINwTjeGbQ4rVEXcbA