review agent failed to catch a real SQL injection in the diff: debugging the false negative
Debugs a review agent missing a real SQL injection by reproducing the miss with a test case, finding the gap in its detection rules, and adding the missing pattern. Use when the reviewer overlooks a known-bad security construct in the diff. Triggers: missed injections, silent dangerous patterns. Not for false positives, or vulnerability classes outside the agent's checklist.
TL;DR
The reviewer has no rule that fires on the pattern in front of it, so it stays silent. Reproduce the miss with a minimal test snippet, find which detection rule should have caught it, add the missing pattern to the review instructions, and re-run. A false negative is a missing rule, not bad luck.
The error
review agent failed to catch a real SQL injection in the diff: debugging the false negative
(query built with string concatenation from request input passed review with no finding)Steps
1. Isolate the missed pattern
Reduce the diff to the smallest snippet that still exhibits the flaw, for example:
query = "SELECT * FROM users WHERE id = " + request.args["id"]Expected output: a minimal reproducer you can feed the reviewer directly.
2. Run the reviewer on the reproducer alone
Expected output: confirm it still stays silent. If it flags the snippet in isolation, the miss was context or diff-parsing, not detection.
3. Find the gap in the detection rules
Read the agent's security checklist or prompt rules and find which rule should have matched. Common gap: it flags one query-building style but not another. Expected output: the specific missing or too-narrow rule, named.
4. Add the missing pattern
Extend the review instructions with the pattern, for example: any string concatenation or formatting that flows into a query call is a finding. Expected output: the updated instructions name the pattern explicitly.
5. Re-run and regression-test
Run the reviewer on the reproducer and on past true positives. Expected output: the reproducer now gets flagged, and previously caught issues still get caught.
When to use
- The reviewer missed a real injection in a diff
- A known-bad pattern passes review silently
- You need to prove whether the miss is detection or context
When not to use
- False positives (the reviewer flags too much; that is a tuning problem in the other direction)
- Vulnerability classes the agent was never told to look for (add the class first)
- One-off misses with no reproducible pattern
Tool compatibility
- LLM review agents with prompt-based or config-based security checklists
- Pairs well with deterministic SAST (semgrep, CodeQL) as a backstop for known patterns
- The reproducer method works for any reviewer, human or automated
Variant phrasings
reviewer missed XSS in a template
Same method: isolate the unescaped interpolation, find the missing template-output rule, add it.
reviewer missed command injection
Same method: isolate the shell call with concatenated input, add the subprocess-without-argument-list pattern.
reviewer catches it sometimes but not always
The rule exists but is phrased vaguely. Make it concrete with positive and negative examples.
Why it happens
LLM reviewers pattern-match against their instructions. If the instructions say "check for SQL injection" without describing what query construction looks like in this codebase's stack, the model has to guess, and guesses miss. Explicit patterns turn a vague worry into a checkable rule.
Edge cases
- ORM parameterized queries are safe: teach the allowlist alongside the pattern, or you trade false negatives for false positives.
- Framework-specific query builders need their own patterns; a rule written for raw SQL will not fire on a builder API.
- Overly broad new rules flag safe code. After adding a pattern, run it against a clean file and tune until the noise is acceptable.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_hkEX6CVEXy0pI6usBL6sSA