## TL;DR
Write a concrete severity rubric with examples, cap severity by finding category so style nits can never be critical, and run a normalization pass that re-checks every label before posting. "Everything is critical" means the labels carry no information; calibration restores it.

## The problem

```text
review agent failed to calibrate severity: every label was critical: how to fix
```

## When to use
- Most or all findings share the top severity label
- Trivial issues (formatting, import order) are labeled critical or high
- Reviewers ignore severity labels or complain about them
- Severity-driven automation (auto-request-changes) fires on every PR

## When not to use
- The PR genuinely has multiple critical issues (a bad diff, not a bad labeler)
- Severity is absent rather than inflated (a different gap)
- Reviewers disagree with individual labels but the distribution looks sane

## Steps

### 1. Write a severity rubric with concrete criteria and examples
Define each level with a test a reviewer could apply: critical means exploitable, data loss, or a broken release; high means a likely bug in a common path; medium means a possible bug or a real maintainability cost; low means nits and style.

Expected output: a rubric file the whole team can read, with one example finding per level.

### 2. Cap severity by finding category
Encode hard ceilings: style and formatting findings can never exceed low; documentation never exceeds medium; test-only issues never exceed medium. The model proposes; the caps dispose.

Expected output: entire classes of inflation become impossible regardless of what the model emits.

### 3. Add a normalization pass before posting
Run a second, cheap pass (rules or a small model call) that re-checks each label against the rubric and the caps, with the rubric text in context.

Expected output: labels that survive to the PR have been checked twice against the same standard.

### 4. Track the severity distribution per run
Log counts per level for every review and alert when the top bucket exceeds a threshold (say, more than a third of findings critical).

Expected output: calibration drift shows up as a metric before reviewers start complaining.

### 5. Tune with reviewer feedback
When reviewers consistently downgrade a label, feed that back into the rubric examples or the caps. Expected output: the rubric converges on what the team actually treats as critical.

## Tool compatibility
- Any review pipeline: the rubric lives in the prompt and the caps in post-processing; no API dependency.
- Severity-driven automation (auto request-changes on critical): becomes trustworthy only after calibration; gate it on step-4 metrics.
- GitHub review APIs: severity is a label in the comment body or metadata; the platform does not enforce it, so your pipeline must.

## Variant phrasings

### review bot marks everything as high severity
Same fix. Check the caps first (step 2): most "everything is high" cases are style findings escaping their ceiling.

### severity inflation in automated code review
When the model is asked to "be thorough," it reads severity as emphasis. The rubric reframes severity as triage information, which calms the inflation.

### how to calibrate AI review severity labels
Start with steps 1 and 2; they fix the majority of cases without any model changes. Add the normalization pass only if inflation persists.

## Why it happens
Models conflate severity with importance-of-being-noticed: asked to review carefully, they mark everything critical to signal diligence. Without a rubric, "critical" has no shared meaning between the model and the team, so the model defaults to the strongest label.

## Edge cases
- Security-focused repos where high skew is legitimate: calibrate the rubric to the repo; a crypto library really does have more criticals than a docs site.
- Auto-fix bots that act on severity: miscalibrated labels cause wrong auto-fixes; fix calibration before enabling auto-fix.
- Mixed human and bot findings: apply the same rubric to both or the labels stop being comparable.
- New contributors: an all-critical review on a first PR is demoralizing; consider softening delivery for first-time authors even when labels are accurate.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JZwhDhD3qrTyiuaJC83ZFA