review agent failed to filter false positives: flagged test fixtures as vulnerabilities: how to fix
Fixes a review agent flooding PRs with vulnerability findings on test fixtures by scoping security rules to production code paths. Use when fixtures, fakes, and mocks trigger injection or hardcoded-credential rules. Triggers: fixture findings, test-path noise. Not for real credentials committed in tests, or genuine findings in production code.
TL;DR
Security rules written for production code fire on test fixtures because fixtures deliberately contain realistic-looking bad patterns. Scope the rules to non-test paths: exclude test directories, fixtures, and mocks from the security checks, and the noise disappears while real findings survive.
The error
review agent failed to filter false positives: flagged test fixtures as vulnerabilities: how to fix
(findings like "hardcoded credential" on fixture files full of fake values)Steps
1. List the flagged files and confirm the pattern
Check where every fixture finding lives. Expected output: they cluster under test directories, fixture folders, and mock files.
2. Add path exclusions to the reviewer config
Exclude test, tests, tests, fixtures, mocks, and .test. paths from the security rules. Expected output: the config names the excluded paths explicitly.
3. Keep deliberate bad examples annotated
For fixtures that intentionally contain attack-like strings, add the team's agreed suppression marker so future readers know it is deliberate. Expected output: intentional cases are documented, not just silent.
4. Re-run the reviewer on the same PR
Expected output: fixture findings are gone, and findings in production code still appear.
5. Spot-check that real findings survive
Run the reviewer on a past PR with a genuine production finding. Expected output: the real finding is still reported. The exclusions must not swallow the signal.
When to use
- The reviewer flags test fixtures, fakes, or mocks as vulnerabilities
- PRs drown in findings nobody acts on
- The team started ignoring the reviewer because of the noise
When not to use
- Real credentials committed inside test files (those need rotation and purging, not exclusion)
- Findings in production code paths
- One-off odd findings with no path pattern
Tool compatibility
- LLM review agents with path-scoped rule configs
- Deterministic scanners (semgrep, CodeQL) support the same exclusions natively
- Works on GitHub, GitLab, and local pre-push review runs
Variant phrasings
reviewer flagged mocks as vulnerabilities
Same fix. Mocks are test doubles by definition; exclude the mock paths.
reviewer flagged example code in docs
Extend the exclusions to docs and examples directories, or scope the rules to source directories only.
B105-style hardcoded-password findings on test fixtures
The classic case. Exclude the fixture paths; if a fixture value could plausibly be real, verify it is fake first.
Why it happens
Security rules match shapes, not intent. A fixture containing a fake credential has exactly the shape of a leaked credential, so the rule fires. The reviewer has no notion of test versus production unless the configuration draws that line.
Edge cases
- Security tests that SHOULD be scanned, like auth flow tests: keep those paths included and accept the occasional finding there.
- Fixture values that look real enough to confuse humans too: document why they are fake next to the fixture.
- Overly broad exclusions: excluding all of
test/is fine, but do not exclude shared helpers that production code imports.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_-nAiPxR4gQapp1Ffj3I7Fw