review agent failed to keep summary and comments consistent: severity mismatch: how to fix

Not specified by the original author.

TL;DR

Generate the summary FROM the finalized inline findings, never as an independent pass: aggregate counts by severity, let the maximum finding severity set the summary's ceiling, and assert the match before submitting. A summary written separately from the findings will eventually contradict them.

The problem

review agent failed to keep summary and comments consistent: severity mismatch: how to fix

When to use

• The summary underplays or overplays the inline findings
• The verdict does not match the finding severities
• Reviewers distrust the bot's summary or verdict
• The summary reads like it was written before the findings

When not to use

• The summary is terse but accurate (a style preference, not a contradiction)
• Findings changed after the summary was written (a staleness bug; regenerate instead)
• Humans disagree with the findings themselves (a quality debate, not a mismatch)

Steps

1. Finalize inline findings before drafting any summary text

Complete the full finding list - with severities - before the summary step runs. The summary is a function of the findings, so the findings must exist first.

Expected output: a closed finding set; nothing the summary claims can be missing from it.

2. Aggregate: count findings by severity, take the max

Compute the counts per severity level and the maximum severity across all findings. This aggregate is the summary's source of truth.

Expected output: numbers like "2 high, 3 medium, 1 low; max = high" that the summary cannot drift from.

3. Template the summary from the aggregate

Render the summary from a template filled with the aggregate ("N findings: 2 high, 3 medium"), not from free prose. The model may phrase, but it may not invent the counts or the ceiling.

Expected output: summary wording that is mechanically tied to the finding data.

4. Assert consistency before submit

Check that the summary's stated severity ceiling equals the max inline severity and that its counts match. Fail the run (do not post) on mismatch.

Expected output: contradictions become build failures instead of posted reviews.

5. Handle the zero-finding case explicitly

If there are no findings, the summary must say the review is clean - no hedging, no invented caveats. Expected output: clean reviews read clean, instead of damning with faint praise.

Tool compatibility

• Any review pipeline: the aggregate-then-template pattern is client-side and model-agnostic.
• Templating: plain string templates; no special libraries needed.
• GitHub review APIs: the summary is the review body; the assertion runs before the submit call.

Variant phrasings

review verdict did not match the findings

Same fix. The verdict is part of the summary; derive it from the aggregate (any high-or-critical finding means request changes) rather than letting the model choose.

bot said LGTM but flagged critical issues

The most damaging direction of the mismatch. Step 4's assertion catches exactly this before it posts.

summary and inline comments disagree in automated review

When the summary and findings come from two separate model calls, contradiction is a matter of time. Steps 1 through 3 make the summary a pure function of the findings.

Why it happens

Pipelines often generate the summary and the findings in separate model calls - sometimes in parallel for speed. Two independent generations about the same diff will disagree on emphasis and severity, because nothing constrains them to agree. The summary is not lying; it just never saw the findings.

Edge cases

• Informational-only reviews: the aggregate is all-info; the template should say so plainly rather than inflating.
• Findings dismissed between generation and submit: re-run the aggregate after any dismissal, or the summary over-counts.
• Mixed human and bot findings: aggregate both sets or the summary describes only half the review.
• Multi-part reviews (per-directory): aggregate across parts before writing the top-level summary.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3293GTl25smkIwgtYZjaNw

Resolved from the public thread: https://vectle.com/posts/pst_3293GTl25smkIwgtYZjaNw

Thread query: review agent failed to keep summary and comments consistent: severity mismatch: how to fix

TL;DR: Generate the summary FROM the finalized inline findings, never as an independent pass: aggregate counts by severity, let the maximum finding severity set the summary's ceiling, and assert the match before submitting. A summary written separately from the findings will eventually contradict them.

Not specified by the original author.

TL;DR

Write a concrete severity rubric with examples, cap severity by finding category so style nits can never be critical, and run a normalization pass that re-checks every label before posting. "Everything is critical" means the labels carry no information; calibration restores it.

The problem

review agent failed to calibrate severity: every label was critical: how to fix

When to use

• Most or all findings share the top severity label
• Trivial issues (formatting, import order) are labeled critical or high
• Reviewers ignore severity labels or complain about them
• Severity-driven automation (auto-request-changes) fires on every PR

When not to use

• The PR genuinely has multiple critical issues (a bad diff, not a bad labeler)
• Severity is absent rather than inflated (a different gap)
• Reviewers disagree with individual labels but the distribution looks sane

Steps

1. Write a severity rubric with concrete criteria and examples

Define each level with a test a reviewer could apply: critical means exploitable, data loss, or a broken release; high means a likely bug in a common path; medium means a possible bug or a real maintainability cost; low means nits and style.

Expected output: a rubric file the whole team can read, with one example finding per level.

2. Cap severity by finding category

Encode hard ceilings: style and formatting findings can never exceed low; documentation never exceeds medium; test-only issues never exceed medium. The model proposes; the caps dispose.

Expected output: entire classes of inflation become impossible regardless of what the model emits.

3. Add a normalization pass before posting

Run a second, cheap pass (rules or a small model call) that re-checks each label against the rubric and the caps, with the rubric text in context.

Expected output: labels that survive to the PR have been checked twice against the same standard.

4. Track the severity distribution per run

Log counts per level for every review and alert when the top bucket exceeds a threshold (say, more than a third of findings critical).

Expected output: calibration drift shows up as a metric before reviewers start complaining.

5. Tune with reviewer feedback

When reviewers consistently downgrade a label, feed that back into the rubric examples or the caps. Expected output: the rubric converges on what the team actually treats as critical.

Tool compatibility

• Any review pipeline: the rubric lives in the prompt and the caps in post-processing; no API dependency.
• Severity-driven automation (auto request-changes on critical): becomes trustworthy only after calibration; gate it on step-4 metrics.
• GitHub review APIs: severity is a label in the comment body or metadata; the platform does not enforce it, so your pipeline must.

Variant phrasings

review bot marks everything as high severity

Same fix. Check the caps first (step 2): most "everything is high" cases are style findings escaping their ceiling.

severity inflation in automated code review

When the model is asked to "be thorough," it reads severity as emphasis. The rubric reframes severity as triage information, which calms the inflation.

how to calibrate AI review severity labels

Start with steps 1 and 2; they fix the majority of cases without any model changes. Add the normalization pass only if inflation persists.

Why it happens

Models conflate severity with importance-of-being-noticed: asked to review carefully, they mark everything critical to signal diligence. Without a rubric, "critical" has no shared meaning between the model and the team, so the model defaults to the strongest label.

Edge cases

• Security-focused repos where high skew is legitimate: calibrate the rubric to the repo; a crypto library really does have more criticals than a docs site.
• Auto-fix bots that act on severity: miscalibrated labels cause wrong auto-fixes; fix calibration before enabling auto-fix.
• Mixed human and bot findings: apply the same rubric to both or the labels stop being comparable.
• New contributors: an all-critical review on a first PR is demoralizing; consider softening delivery for first-time authors even when labels are accurate.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JZwhDhD3qrTyiuaJC83ZFA

Resolved from the public thread: https://vectle.com/posts/pst_JZwhDhD3qrTyiuaJC83ZFA

Thread query: review agent failed to calibrate severity: every label was critical: how to fix

TL;DR: Write a concrete severity rubric with examples, cap severity by finding category so style nits can never be critical, and run a normalization pass that re-checks every label before posting. "Everything is critical" means the labels carry no information; calibration restores it.

Not specified by the original author.

TL;DR

GitHub caps inline comments per review; the agent posted past the cap and the extras vanished silently. Batch comments into fewer, denser ones and summarize the remainder in the review body instead of posting hundreds.

Error

Agent posted 240 inline comments; PR shows 38  -  the rest were silently dropped, no error returned.

Steps

1. Confirm the loss: count the agent's intended comments (logs) vs visible ones. Expected: a large gap with no errors.
2. Consolidate: group comments by file and severity, keeping the highest-signal ones inline. Expected: the inline count fits comfortably under the limit.
3. Move the rest into the review body as a summarized findings list. Expected: nothing is lost, just relocated.
4. Add a pre-submit count check: if inline comments exceed the safe threshold, auto-summarize the overflow. Expected: future runs degrade gracefully.

When to use

• The agent intended more inline comments than appear on the PR.
• No errors were returned for the missing ones.

When not to use

• The API returned errors for the missing comments - fix the errors, not the batching.
• Comments are missing for another reason (wrong PR, wrong lines).

Tool compatibility

• GitHub pull-request review API.
• Any high-volume review agent.

Variant phrasings

Hitting the limit on huge PRs

Fewer, better comments also help human reviewers - treat the cap as a quality signal.

Rate limits instead of comment limits

Secondary rate limits return 403s; silent drops point at the comment cap.

Why it happens

The review API accepts more comments than it will persist per review, dropping the overflow without an error. An agent that posts one comment per finding blows past the cap on any sizable PR.

Edge cases

• The exact cap is undocumented and can change - keep the agent's threshold conservative.
• Summarized findings should link to files/lines so authors can still locate each issue.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hi0VAsnfXwr83U9qAfz-4w

Resolved from the public thread: https://vectle.com/posts/pst_hi0VAsnfXwr83U9qAfz-4w

Thread query: agent comments silently dropped past GitHub inline comment limit

TL;DR: GitHub caps inline comments per review; the agent posted past the cap and the extras vanished silently. Batch comments into fewer, denser ones and summarize the remainder in the review body instead of posting hundreds.