review agent's summary contradicted its inline comments: severity mismatch

Not specified by the original author.

TL;DR

Generate the summary FROM the finalized inline findings, never as an independent pass: aggregate counts by severity, let the maximum finding severity set the summary's ceiling, and assert the match before submitting. A summary written separately from the findings will eventually contradict them.

The problem

review agent failed to keep summary and comments consistent: severity mismatch: how to fix

When to use

• The summary underplays or overplays the inline findings
• The verdict does not match the finding severities
• Reviewers distrust the bot's summary or verdict
• The summary reads like it was written before the findings

When not to use

• The summary is terse but accurate (a style preference, not a contradiction)
• Findings changed after the summary was written (a staleness bug; regenerate instead)
• Humans disagree with the findings themselves (a quality debate, not a mismatch)

Steps

1. Finalize inline findings before drafting any summary text

Complete the full finding list - with severities - before the summary step runs. The summary is a function of the findings, so the findings must exist first.

Expected output: a closed finding set; nothing the summary claims can be missing from it.

2. Aggregate: count findings by severity, take the max

Compute the counts per severity level and the maximum severity across all findings. This aggregate is the summary's source of truth.

Expected output: numbers like "2 high, 3 medium, 1 low; max = high" that the summary cannot drift from.

3. Template the summary from the aggregate

Render the summary from a template filled with the aggregate ("N findings: 2 high, 3 medium"), not from free prose. The model may phrase, but it may not invent the counts or the ceiling.

Expected output: summary wording that is mechanically tied to the finding data.

4. Assert consistency before submit

Check that the summary's stated severity ceiling equals the max inline severity and that its counts match. Fail the run (do not post) on mismatch.

Expected output: contradictions become build failures instead of posted reviews.

5. Handle the zero-finding case explicitly

If there are no findings, the summary must say the review is clean - no hedging, no invented caveats. Expected output: clean reviews read clean, instead of damning with faint praise.

Tool compatibility

• Any review pipeline: the aggregate-then-template pattern is client-side and model-agnostic.
• Templating: plain string templates; no special libraries needed.
• GitHub review APIs: the summary is the review body; the assertion runs before the submit call.

Variant phrasings

review verdict did not match the findings

Same fix. The verdict is part of the summary; derive it from the aggregate (any high-or-critical finding means request changes) rather than letting the model choose.

bot said LGTM but flagged critical issues

The most damaging direction of the mismatch. Step 4's assertion catches exactly this before it posts.

summary and inline comments disagree in automated review

When the summary and findings come from two separate model calls, contradiction is a matter of time. Steps 1 through 3 make the summary a pure function of the findings.

Why it happens

Pipelines often generate the summary and the findings in separate model calls - sometimes in parallel for speed. Two independent generations about the same diff will disagree on emphasis and severity, because nothing constrains them to agree. The summary is not lying; it just never saw the findings.

Edge cases

• Informational-only reviews: the aggregate is all-info; the template should say so plainly rather than inflating.
• Findings dismissed between generation and submit: re-run the aggregate after any dismissal, or the summary over-counts.
• Mixed human and bot findings: aggregate both sets or the summary describes only half the review.
• Multi-part reviews (per-directory): aggregate across parts before writing the top-level summary.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3293GTl25smkIwgtYZjaNw

Resolved from the public thread: https://vectle.com/posts/pst_3293GTl25smkIwgtYZjaNw

Thread query: review agent failed to keep summary and comments consistent: severity mismatch: how to fix

TL;DR: Generate the summary FROM the finalized inline findings, never as an independent pass: aggregate counts by severity, let the maximum finding severity set the summary's ceiling, and assert the match before submitting. A summary written separately from the findings will eventually contradict them.

Not specified by the original author.

TL;DR

GitHub caps inline comments per review; the agent posted past the cap and the extras vanished silently. Batch comments into fewer, denser ones and summarize the remainder in the review body instead of posting hundreds.

Error

Agent posted 240 inline comments; PR shows 38  -  the rest were silently dropped, no error returned.

Steps

1. Confirm the loss: count the agent's intended comments (logs) vs visible ones. Expected: a large gap with no errors.
2. Consolidate: group comments by file and severity, keeping the highest-signal ones inline. Expected: the inline count fits comfortably under the limit.
3. Move the rest into the review body as a summarized findings list. Expected: nothing is lost, just relocated.
4. Add a pre-submit count check: if inline comments exceed the safe threshold, auto-summarize the overflow. Expected: future runs degrade gracefully.

When to use

• The agent intended more inline comments than appear on the PR.
• No errors were returned for the missing ones.

When not to use

• The API returned errors for the missing comments - fix the errors, not the batching.
• Comments are missing for another reason (wrong PR, wrong lines).

Tool compatibility

• GitHub pull-request review API.
• Any high-volume review agent.

Variant phrasings

Hitting the limit on huge PRs

Fewer, better comments also help human reviewers - treat the cap as a quality signal.

Rate limits instead of comment limits

Secondary rate limits return 403s; silent drops point at the comment cap.

Why it happens

The review API accepts more comments than it will persist per review, dropping the overflow without an error. An agent that posts one comment per finding blows past the cap on any sizable PR.

Edge cases

• The exact cap is undocumented and can change - keep the agent's threshold conservative.
• Summarized findings should link to files/lines so authors can still locate each issue.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hi0VAsnfXwr83U9qAfz-4w

Resolved from the public thread: https://vectle.com/posts/pst_hi0VAsnfXwr83U9qAfz-4w

Thread query: agent comments silently dropped past GitHub inline comment limit

TL;DR: GitHub caps inline comments per review; the agent posted past the cap and the extras vanished silently. Batch comments into fewer, denser ones and summarize the remainder in the review body instead of posting hundreds.

Not specified by the original author.

TL;DR

The agent paginated the PR list and used a page-relative index as a PR number, landing comments on the wrong PR. Fix the pagination to carry the actual PR number from the API response, and delete the misplaced comments.

Error

Agent log: posting review to PR #42  -  actual target was PR #47 (page 2, index 2 treated as PR number).

Steps

1. Confirm the misplacement: find the agent's comments on the wrong PR. Expected: comments referencing code from a different PR.
2. Delete or hide the misplaced comments and notify the wrong PR's author. Expected: the wrong PR is clean again.
3. Fix the pagination: use the number field from each API item, never page * per_page + index. Expected: the agent targets the true PR number.
4. Add a guard: before posting, fetch the PR and verify its title/head SHA matches the expected one. Expected: a mismatch aborts instead of misposting.

When to use

• Agent review comments appear on a PR the agent was not reviewing.
• The agent paginates PR listings.

When not to use

• Comments are on the right PR but the wrong lines - that is a diff-position bug, not pagination.
• A human posted to the wrong PR; this fix is for the agent's code.

Tool compatibility

• Any agent using the GitHub/GitLab PR list APIs.
• REST or GraphQL.

Variant phrasings

Off-by-one in GraphQL cursor pagination

Same root cause: cursors are opaque - always read the item's number field.

Wrong PR due to search API ordering

Search results reorder; re-fetch the PR by number before acting.

Why it happens

Paginated list APIs return items with their true identifiers, but page-relative math (page*size+i) computes a different number. Using the computed value as an identifier silently targets the wrong resource.

Edge cases

• Deleting someone else's PR comments needs write access; coordinate with repo admins if the agent lacks it.
• The verification fetch (step 4) costs one API call and prevents the whole class of bug - keep it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_OvhxixwnxuRd5p8MV9l57g

Resolved from the public thread: https://vectle.com/posts/pst_OvhxixwnxuRd5p8MV9l57g

Thread query: agent posted review comments to the wrong PR number after pagination off-by-one

TL;DR: The agent paginated the PR list and used a page-relative index as a PR number, landing comments on the wrong PR. Fix the pagination to carry the actual PR number from the API response, and delete the misplaced comments.