review agent failed to calibrate severity: every label was critical: how to fix
Fix for review agents that label every finding critical, which trains reviewers to ignore the label entirely. Use when severity distributions skew to the top, trivial nits ship as critical, or reviewers complain everything is on fire. Key triggers: all-critical runs, style nits marked critical, severity ignored by reviewers. Not for PRs that genuinely contain multiple critical issues.
TL;DR
Write a concrete severity rubric with examples, cap severity by finding category so style nits can never be critical, and run a normalization pass that re-checks every label before posting. "Everything is critical" means the labels carry no information; calibration restores it.
The problem
review agent failed to calibrate severity: every label was critical: how to fixWhen to use
- Most or all findings share the top severity label
- Trivial issues (formatting, import order) are labeled critical or high
- Reviewers ignore severity labels or complain about them
- Severity-driven automation (auto-request-changes) fires on every PR
When not to use
- The PR genuinely has multiple critical issues (a bad diff, not a bad labeler)
- Severity is absent rather than inflated (a different gap)
- Reviewers disagree with individual labels but the distribution looks sane
Steps
1. Write a severity rubric with concrete criteria and examples
Define each level with a test a reviewer could apply: critical means exploitable, data loss, or a broken release; high means a likely bug in a common path; medium means a possible bug or a real maintainability cost; low means nits and style.
Expected output: a rubric file the whole team can read, with one example finding per level.
2. Cap severity by finding category
Encode hard ceilings: style and formatting findings can never exceed low; documentation never exceeds medium; test-only issues never exceed medium. The model proposes; the caps dispose.
Expected output: entire classes of inflation become impossible regardless of what the model emits.
3. Add a normalization pass before posting
Run a second, cheap pass (rules or a small model call) that re-checks each label against the rubric and the caps, with the rubric text in context.
Expected output: labels that survive to the PR have been checked twice against the same standard.
4. Track the severity distribution per run
Log counts per level for every review and alert when the top bucket exceeds a threshold (say, more than a third of findings critical).
Expected output: calibration drift shows up as a metric before reviewers start complaining.
5. Tune with reviewer feedback
When reviewers consistently downgrade a label, feed that back into the rubric examples or the caps. Expected output: the rubric converges on what the team actually treats as critical.
Tool compatibility
- Any review pipeline: the rubric lives in the prompt and the caps in post-processing; no API dependency.
- Severity-driven automation (auto request-changes on critical): becomes trustworthy only after calibration; gate it on step-4 metrics.
- GitHub review APIs: severity is a label in the comment body or metadata; the platform does not enforce it, so your pipeline must.
Variant phrasings
review bot marks everything as high severity
Same fix. Check the caps first (step 2): most "everything is high" cases are style findings escaping their ceiling.
severity inflation in automated code review
When the model is asked to "be thorough," it reads severity as emphasis. The rubric reframes severity as triage information, which calms the inflation.
how to calibrate AI review severity labels
Start with steps 1 and 2; they fix the majority of cases without any model changes. Add the normalization pass only if inflation persists.
Why it happens
Models conflate severity with importance-of-being-noticed: asked to review carefully, they mark everything critical to signal diligence. Without a rubric, "critical" has no shared meaning between the model and the team, so the model defaults to the strongest label.
Edge cases
- Security-focused repos where high skew is legitimate: calibrate the rubric to the repo; a crypto library really does have more criticals than a docs site.
- Auto-fix bots that act on severity: miscalibrated labels cause wrong auto-fixes; fix calibration before enabling auto-fix.
- Mixed human and bot findings: apply the same rubric to both or the labels stop being comparable.
- New contributors: an all-critical review on a first PR is demoralizing; consider softening delivery for first-time authors even when labels are accurate.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_JZwhDhD3qrTyiuaJC83ZFA