review agent failed to keep summary and comments consistent: severity mismatch: how to fix

TL;DR

Generate the summary FROM the finalized inline findings, never as an independent pass: aggregate counts by severity, let the maximum finding severity set the summary's ceiling, and assert the match before submitting. A summary written separately from the findings will eventually contradict them.

The problem

review agent failed to keep summary and comments consistent: severity mismatch: how to fix

When to use

• The summary underplays or overplays the inline findings
• The verdict does not match the finding severities
• Reviewers distrust the bot's summary or verdict
• The summary reads like it was written before the findings

When not to use

• The summary is terse but accurate (a style preference, not a contradiction)
• Findings changed after the summary was written (a staleness bug; regenerate instead)
• Humans disagree with the findings themselves (a quality debate, not a mismatch)

Steps

1. Finalize inline findings before drafting any summary text

Complete the full finding list - with severities - before the summary step runs. The summary is a function of the findings, so the findings must exist first.

Expected output: a closed finding set; nothing the summary claims can be missing from it.

2. Aggregate: count findings by severity, take the max

Compute the counts per severity level and the maximum severity across all findings. This aggregate is the summary's source of truth.

Expected output: numbers like "2 high, 3 medium, 1 low; max = high" that the summary cannot drift from.

3. Template the summary from the aggregate

Render the summary from a template filled with the aggregate ("N findings: 2 high, 3 medium"), not from free prose. The model may phrase, but it may not invent the counts or the ceiling.

Expected output: summary wording that is mechanically tied to the finding data.

4. Assert consistency before submit

Check that the summary's stated severity ceiling equals the max inline severity and that its counts match. Fail the run (do not post) on mismatch.

Expected output: contradictions become build failures instead of posted reviews.

5. Handle the zero-finding case explicitly

If there are no findings, the summary must say the review is clean - no hedging, no invented caveats. Expected output: clean reviews read clean, instead of damning with faint praise.

Tool compatibility

• Any review pipeline: the aggregate-then-template pattern is client-side and model-agnostic.
• Templating: plain string templates; no special libraries needed.
• GitHub review APIs: the summary is the review body; the assertion runs before the submit call.

Variant phrasings

review verdict did not match the findings

Same fix. The verdict is part of the summary; derive it from the aggregate (any high-or-critical finding means request changes) rather than letting the model choose.

bot said LGTM but flagged critical issues

The most damaging direction of the mismatch. Step 4's assertion catches exactly this before it posts.

summary and inline comments disagree in automated review

When the summary and findings come from two separate model calls, contradiction is a matter of time. Steps 1 through 3 make the summary a pure function of the findings.

Why it happens

Pipelines often generate the summary and the findings in separate model calls - sometimes in parallel for speed. Two independent generations about the same diff will disagree on emphasis and severity, because nothing constrains them to agree. The summary is not lying; it just never saw the findings.

Edge cases

• Informational-only reviews: the aggregate is all-info; the template should say so plainly rather than inflating.
• Findings dismissed between generation and submit: re-run the aggregate after any dismissal, or the summary over-counts.
• Mixed human and bot findings: aggregate both sets or the summary describes only half the review.
• Multi-part reviews (per-directory): aggregate across parts before writing the top-level summary.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3293GTl25smkIwgtYZjaNw