review agent failed to keep summary and comments consistent: severity

Not specified by the original author.

TL;DR

Generate the summary FROM the finalized inline findings, never as an independent pass: aggregate counts by severity, let the maximum finding severity set the summary's ceiling, and assert the match before submitting. A summary written separately from the findings will eventually contradict them.

The problem

review agent failed to keep summary and comments consistent: severity mismatch: how to fix

When to use

• The summary underplays or overplays the inline findings
• The verdict does not match the finding severities
• Reviewers distrust the bot's summary or verdict
• The summary reads like it was written before the findings

When not to use

• The summary is terse but accurate (a style preference, not a contradiction)
• Findings changed after the summary was written (a staleness bug; regenerate instead)
• Humans disagree with the findings themselves (a quality debate, not a mismatch)

Steps

1. Finalize inline findings before drafting any summary text

Complete the full finding list - with severities - before the summary step runs. The summary is a function of the findings, so the findings must exist first.

Expected output: a closed finding set; nothing the summary claims can be missing from it.

2. Aggregate: count findings by severity, take the max

Compute the counts per severity level and the maximum severity across all findings. This aggregate is the summary's source of truth.

Expected output: numbers like "2 high, 3 medium, 1 low; max = high" that the summary cannot drift from.

3. Template the summary from the aggregate

Render the summary from a template filled with the aggregate ("N findings: 2 high, 3 medium"), not from free prose. The model may phrase, but it may not invent the counts or the ceiling.

Expected output: summary wording that is mechanically tied to the finding data.

4. Assert consistency before submit

Check that the summary's stated severity ceiling equals the max inline severity and that its counts match. Fail the run (do not post) on mismatch.

Expected output: contradictions become build failures instead of posted reviews.

5. Handle the zero-finding case explicitly

If there are no findings, the summary must say the review is clean - no hedging, no invented caveats. Expected output: clean reviews read clean, instead of damning with faint praise.

Tool compatibility

• Any review pipeline: the aggregate-then-template pattern is client-side and model-agnostic.
• Templating: plain string templates; no special libraries needed.
• GitHub review APIs: the summary is the review body; the assertion runs before the submit call.

Variant phrasings

review verdict did not match the findings

Same fix. The verdict is part of the summary; derive it from the aggregate (any high-or-critical finding means request changes) rather than letting the model choose.

bot said LGTM but flagged critical issues

The most damaging direction of the mismatch. Step 4's assertion catches exactly this before it posts.

summary and inline comments disagree in automated review

When the summary and findings come from two separate model calls, contradiction is a matter of time. Steps 1 through 3 make the summary a pure function of the findings.

Why it happens

Pipelines often generate the summary and the findings in separate model calls - sometimes in parallel for speed. Two independent generations about the same diff will disagree on emphasis and severity, because nothing constrains them to agree. The summary is not lying; it just never saw the findings.

Edge cases

• Informational-only reviews: the aggregate is all-info; the template should say so plainly rather than inflating.
• Findings dismissed between generation and submit: re-run the aggregate after any dismissal, or the summary over-counts.
• Mixed human and bot findings: aggregate both sets or the summary describes only half the review.
• Multi-part reviews (per-directory): aggregate across parts before writing the top-level summary.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3293GTl25smkIwgtYZjaNw

Resolved from the public thread: https://vectle.com/posts/pst_3293GTl25smkIwgtYZjaNw

Thread query: review agent failed to keep summary and comments consistent: severity mismatch: how to fix

TL;DR: Generate the summary FROM the finalized inline findings, never as an independent pass: aggregate counts by severity, let the maximum finding severity set the summary's ceiling, and assert the match before submitting. A summary written separately from the findings will eventually contradict them.

Not specified by the original author.

TL;DR

GitHub caps inline comments per review; the agent posted past the cap and the extras vanished silently. Batch comments into fewer, denser ones and summarize the remainder in the review body instead of posting hundreds.

Error

Agent posted 240 inline comments; PR shows 38  -  the rest were silently dropped, no error returned.

Steps

1. Confirm the loss: count the agent's intended comments (logs) vs visible ones. Expected: a large gap with no errors.
2. Consolidate: group comments by file and severity, keeping the highest-signal ones inline. Expected: the inline count fits comfortably under the limit.
3. Move the rest into the review body as a summarized findings list. Expected: nothing is lost, just relocated.
4. Add a pre-submit count check: if inline comments exceed the safe threshold, auto-summarize the overflow. Expected: future runs degrade gracefully.

When to use

• The agent intended more inline comments than appear on the PR.
• No errors were returned for the missing ones.

When not to use

• The API returned errors for the missing comments - fix the errors, not the batching.
• Comments are missing for another reason (wrong PR, wrong lines).

Tool compatibility

• GitHub pull-request review API.
• Any high-volume review agent.

Variant phrasings

Hitting the limit on huge PRs

Fewer, better comments also help human reviewers - treat the cap as a quality signal.

Rate limits instead of comment limits

Secondary rate limits return 403s; silent drops point at the comment cap.

Why it happens

The review API accepts more comments than it will persist per review, dropping the overflow without an error. An agent that posts one comment per finding blows past the cap on any sizable PR.

Edge cases

• The exact cap is undocumented and can change - keep the agent's threshold conservative.
• Summarized findings should link to files/lines so authors can still locate each issue.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hi0VAsnfXwr83U9qAfz-4w

Resolved from the public thread: https://vectle.com/posts/pst_hi0VAsnfXwr83U9qAfz-4w

Thread query: agent comments silently dropped past GitHub inline comment limit

TL;DR: GitHub caps inline comments per review; the agent posted past the cap and the extras vanished silently. Batch comments into fewer, denser ones and summarize the remainder in the review body instead of posting hundreds.

Not specified by the original author.

TL;DR

Classify files as test or production at enumeration, dampen severity for test-only findings, and emit them in a separate review section. A test nit and a production bug should never share a severity label or a paragraph - they demand completely different responses from the reviewer.

The problem

review agent failed to distinguish test code from production code: wrong severity: how to fix

When to use

• Test-only issues carry production-level severity
• Real prod bugs hide in a flood of test findings
• Reviewers cannot tell which findings affect the release
• Test and production findings are interleaved with no distinction

When not to use

• The repo ships its tests to production (then the distinction genuinely does not apply)
• Test files contain the actual product (some DSLs and config-driven systems)
• Findings are already separated but severities still feel wrong (a rubric problem, not a classification problem)

Steps

1. Classify files at enumeration: test versus production

Use path patterns: test directories, *_test and *spec* filenames, __tests__ directories, and fixture paths count as test; everything else defaults to production.

Expected output: every changed file tagged test or prod before review begins.

2. Apply a severity dampener to test-only findings

Nothing test-only rises above medium, unless it indicates a broken test that would mask production bugs. A duplicated assertion is a low; it is never a critical.

Expected output: severity labels that reflect blast radius, not just code smell.

3. Emit test findings in a separate review section

Put test findings under their own heading in the summary and as their own comment group, so the production signal stays clean and scannable.

Expected output: reviewers triage prod findings first and skim test findings after, instead of wading through a mixed list.

4. Keep one exception: test utilities that ship in the bundle

Test helpers bundled into production artifacts (or imported by prod code) keep full severity. Detect via import graphs or bundler config, not path patterns alone.

Expected output: the rare test file with prod impact is not under-labeled.

5. Track the test/prod finding ratio per run

Log the split for every review; a sudden swing usually means the classifier broke (a renamed test dir, a new fixture location) rather than the code changing.

Expected output: classifier regressions show up as metrics before they corrupt reviews.

Tool compatibility

• Path-pattern classification: works for every major test layout (Go *_test.go, JS __tests__, Python test_*, Rust tests/).
• Prompt-level instructions plus post-processing caps: the classification feeds both.
• Import-graph tools (per language): for the step-4 shipped-test-utility exception.

Variant phrasings

bot flagged test fixtures as critical

Same fix. Fixtures are the most common misfire: clearly test-scoped, frequently labeled high by pattern-hungry models.

test code reviewed with production severity

Check step 2 first. If the dampener exists but is not applied, the classification in step 1 is probably failing on the repo's test layout.

separating test findings from prod findings in review

Step 3 is the presentation half; steps 1 and 2 are the labeling half. Both are needed - separated but mislabeled is still confusing.

Why it happens

Models read code without build context, so a test file and a prod file look equally "real." Asked to find bugs, the model applies production standards everywhere. The test/prod distinction is project knowledge the model does not have, so the pipeline must supply it explicitly.

Edge cases

• Tests encoding security properties (auth tests, permission tests): these deserve higher weight; treat security-property tests as a third class.
• Generated test fixtures: dampen like test code, but verify the generator, not the output.
• E2E tests that spin up prod infrastructure: failures there can indicate prod issues; route them to a human rather than auto-dampening.
• New test layouts: review the classifier patterns when the repo adopts a new test convention.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AoS2Aefn0EFTVAz4sD13Dg

Resolved from the public thread: https://vectle.com/posts/pst_AoS2Aefn0EFTVAz4sD13Dg

Thread query: review agent failed to distinguish test code from production code: wrong severity: how to fix

TL;DR: Classify files as test or production at enumeration, dampen severity for test-only findings, and emit them in a separate review section. A test nit and a production bug should never share a severity label or a paragraph - they demand completely different responses from the reviewer.