review agent failed to distinguish test code from production code: wrong severity: how to fix

TL;DR

Classify files as test or production at enumeration, dampen severity for test-only findings, and emit them in a separate review section. A test nit and a production bug should never share a severity label or a paragraph - they demand completely different responses from the reviewer.

The problem

review agent failed to distinguish test code from production code: wrong severity: how to fix

When to use

• Test-only issues carry production-level severity
• Real prod bugs hide in a flood of test findings
• Reviewers cannot tell which findings affect the release
• Test and production findings are interleaved with no distinction

When not to use

• The repo ships its tests to production (then the distinction genuinely does not apply)
• Test files contain the actual product (some DSLs and config-driven systems)
• Findings are already separated but severities still feel wrong (a rubric problem, not a classification problem)

Steps

1. Classify files at enumeration: test versus production

Use path patterns: test directories, *_test and *spec* filenames, __tests__ directories, and fixture paths count as test; everything else defaults to production.

Expected output: every changed file tagged test or prod before review begins.

2. Apply a severity dampener to test-only findings

Nothing test-only rises above medium, unless it indicates a broken test that would mask production bugs. A duplicated assertion is a low; it is never a critical.

Expected output: severity labels that reflect blast radius, not just code smell.

3. Emit test findings in a separate review section

Put test findings under their own heading in the summary and as their own comment group, so the production signal stays clean and scannable.

Expected output: reviewers triage prod findings first and skim test findings after, instead of wading through a mixed list.

4. Keep one exception: test utilities that ship in the bundle

Test helpers bundled into production artifacts (or imported by prod code) keep full severity. Detect via import graphs or bundler config, not path patterns alone.

Expected output: the rare test file with prod impact is not under-labeled.

5. Track the test/prod finding ratio per run

Log the split for every review; a sudden swing usually means the classifier broke (a renamed test dir, a new fixture location) rather than the code changing.

Expected output: classifier regressions show up as metrics before they corrupt reviews.

Tool compatibility

• Path-pattern classification: works for every major test layout (Go *_test.go, JS __tests__, Python test_*, Rust tests/).
• Prompt-level instructions plus post-processing caps: the classification feeds both.
• Import-graph tools (per language): for the step-4 shipped-test-utility exception.

Variant phrasings

bot flagged test fixtures as critical

Same fix. Fixtures are the most common misfire: clearly test-scoped, frequently labeled high by pattern-hungry models.

test code reviewed with production severity

Check step 2 first. If the dampener exists but is not applied, the classification in step 1 is probably failing on the repo's test layout.

separating test findings from prod findings in review

Step 3 is the presentation half; steps 1 and 2 are the labeling half. Both are needed - separated but mislabeled is still confusing.

Why it happens

Models read code without build context, so a test file and a prod file look equally "real." Asked to find bugs, the model applies production standards everywhere. The test/prod distinction is project knowledge the model does not have, so the pipeline must supply it explicitly.

Edge cases

• Tests encoding security properties (auth tests, permission tests): these deserve higher weight; treat security-property tests as a third class.
• Generated test fixtures: dampen like test code, but verify the generator, not the output.
• E2E tests that spin up prod infrastructure: failures there can indicate prod issues; route them to a human rather than auto-dampening.
• New test layouts: review the classifier patterns when the repo adopts a new test convention.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AoS2Aefn0EFTVAz4sD13Dg