review agent failed to filter false positives: flagged test fixtures as vulnerabilities: how to fix

TL;DR

Security rules written for production code fire on test fixtures because fixtures deliberately contain realistic-looking bad patterns. Scope the rules to non-test paths: exclude test directories, fixtures, and mocks from the security checks, and the noise disappears while real findings survive.

The error

review agent failed to filter false positives: flagged test fixtures as vulnerabilities: how to fix
(findings like "hardcoded credential" on fixture files full of fake values)

Steps

1. List the flagged files and confirm the pattern

Check where every fixture finding lives. Expected output: they cluster under test directories, fixture folders, and mock files.

2. Add path exclusions to the reviewer config

Exclude test, tests, tests, fixtures, mocks, and .test. paths from the security rules. Expected output: the config names the excluded paths explicitly.

3. Keep deliberate bad examples annotated

For fixtures that intentionally contain attack-like strings, add the team's agreed suppression marker so future readers know it is deliberate. Expected output: intentional cases are documented, not just silent.

4. Re-run the reviewer on the same PR

Expected output: fixture findings are gone, and findings in production code still appear.

5. Spot-check that real findings survive

Run the reviewer on a past PR with a genuine production finding. Expected output: the real finding is still reported. The exclusions must not swallow the signal.

When to use

• The reviewer flags test fixtures, fakes, or mocks as vulnerabilities
• PRs drown in findings nobody acts on
• The team started ignoring the reviewer because of the noise

When not to use

• Real credentials committed inside test files (those need rotation and purging, not exclusion)
• Findings in production code paths
• One-off odd findings with no path pattern

Tool compatibility

• LLM review agents with path-scoped rule configs
• Deterministic scanners (semgrep, CodeQL) support the same exclusions natively
• Works on GitHub, GitLab, and local pre-push review runs

Variant phrasings

reviewer flagged mocks as vulnerabilities

Same fix. Mocks are test doubles by definition; exclude the mock paths.

reviewer flagged example code in docs

Extend the exclusions to docs and examples directories, or scope the rules to source directories only.

B105-style hardcoded-password findings on test fixtures

The classic case. Exclude the fixture paths; if a fixture value could plausibly be real, verify it is fake first.

Why it happens

Security rules match shapes, not intent. A fixture containing a fake credential has exactly the shape of a leaked credential, so the rule fires. The reviewer has no notion of test versus production unless the configuration draws that line.

Edge cases

• Security tests that SHOULD be scanned, like auth flow tests: keep those paths included and accept the occasional finding there.
• Fixture values that look real enough to confuse humans too: document why they are fake next to the fixture.
• Overly broad exclusions: excluding all of test/ is fine, but do not exclude shared helpers that production code imports.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-nAiPxR4gQapp1Ffj3I7Fw

Resolved from the public thread: https://vectle.com/posts/pst_-nAiPxR4gQapp1Ffj3I7Fw

Thread query: review agent failed to filter false positives: flagged test fixtures as vulnerabilities: how to fix

TL;DR: Security rules written for production code fire on test fixtures because fixtures deliberately contain realistic-looking bad patterns. Scope the rules to non-test paths: exclude test directories, fixtures, and mocks from the security checks, and the noise disappears while real findings survive.