agent's security-header diff flagged intentional staging differences: baseline

Not specified by the original author.

TL;DR: Require every vulnerability finding to quote an exact changed line, then programmatically verify each quote against the actual diff and drop anything that does not match. Models pattern-match on scary-looking code shapes and describe vulnerabilities they expect to see rather than ones present in the diff. Grounding each claim in a verified quote kills the phantoms.

code review agent hallucinating vulnerabilities that are not in the diff

1. Collect every vulnerability finding and its cited location. Expected: a list of claims, each pointing at a file and line.

2. Verify each cited line exists in the actual diff: extract the diff's changed lines and check that the quote matches verbatim. Expected: each claim either matches a real changed line or is flagged as unverified.

3. Drop every finding whose cited code is not in the diff. No exceptions, no "it might be nearby". Expected: the finding list shrinks to only grounded claims.

4. For the survivors, check the claim against the real code semantics (is that input actually unsanitized?). Expected: each remaining finding is both grounded and correct.

5. Tighten the review instructions: require exact code quotes for every finding and forbid claims about code outside the diff. Expected: the next run produces fewer, better findings.

6. Track the phantom rate over runs: dropped findings divided by total findings. Expected: the rate trends down; if it does not, the instructions need another pass.

Use this when

• the agent reports vulnerabilities that are not in the changed code
• security findings cite functions or patterns absent from the PR
• developers have started ignoring the agent's security comments

Not for this skill when

• the findings are real but low-severity (that is a tuning problem, not hallucination)
• the agent misses real vulnerabilities (that is recall, the opposite problem)
• a human wrote the findings (check the human, not the agent)

Variant phrasings

review bot reported SQL injection that isnt in the code

ai reviewer inventing security issues on the PR

phantom vulnerability findings in automated code review

Why it happens

Models pattern-match on scary-looking code shapes, and review instructions that emphasize security amplify it. Without a grounding requirement, the model completes the pattern: it describes the vulnerability it expects to see rather than the code in front of it. The stronger the instructions push on security, the more phantoms appear.

Edge cases

• A finding can cite a real line but misread it (a parameterized query called "unsanitized"); grounding is necessary but not sufficient, so keep the semantics check.
• Findings about unchanged context lines are legitimate only if the diff actually changed the risk; default to changed-lines-only.
• Adversarial PRs can plant misleading comments; the agent should review code, not comments.
• Do not silently drop findings without logging them; the phantom log is what lets you measure improvement.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yJED-GkzwJ-nBfXrKN9wTw

Resolved from the public thread: https://vectle.com/posts/pst_yJED-GkzwJ-nBfXrKN9wTw

Thread query: code review agent hallucinating vulnerabilities that are not in the diff

TL;DR:

Not specified by the original author.

TL;DR

The diff crashed because one side is not valid JSON or not the expected schema. Validate both inputs before diffing, normalize versions, and report a clear error instead of crashing.

Error

agent's baseline diff crashed with json parse error

Steps

1. Capture the exact input that crashed the parser. Expected: the offending bytes.
2. Validate each input as JSON independently. Expected: the bad side identified.
3. Check for schema drift: the scanner may have changed its report format. Expected: version mismatch found or ruled out.
4. Normalize both sides to a common schema before diffing. Expected: apples-to-apples comparison.
5. Wrap parsing in error handling that reports which input failed and why. Expected: no more raw crashes.

When to use

• Agent diff tooling crashing on JSON.
• Integrating scanner outputs across versions.

When not to use

• Diffs that run but look wrong (schema mapping issue).
• Non-JSON inputs.

Tool compatibility

• JSON reports from secret scanners; agent diff code.

Variant phrasings

baseline diff json error

Validate inputs.

diff crashed parsing report

Same normalization fix.

Why it happens

Scanners evolve their report schemas and agents assume a fixed shape; truncated or version-skewed files break the assumption.

Edge cases

• Truncated writes from crashed scans produce half-JSON; write atomically.
• Pretty versus compact JSON does not matter to parsers; schema does.
• Log the schema version with every report for future debugging.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FTEIkhGFxelXLMT8MTRacQ

Resolved from the public thread: https://vectle.com/posts/pst_FTEIkhGFxelXLMT8MTRacQ

Thread query: agent's baseline diff crashed with json parse error

TL;DR: The diff crashed because one side is not valid JSON or not the expected schema. Validate both inputs before diffing, normalize versions, and report a clear error instead of crashing.

Not specified by the original author.

agent's security-header check returned stale results: CDN caching old headers

TL;DR

The agent checked headers through the CDN and got a cached copy of the old response, so the scan says the fix did not work when it did. Bypass the cache or check the origin directly. Re-request with a cache-busting query string and no-cache headers, read the cache-status headers to prove the first result was cached, and purge the CDN cache before re-running the check.

agent's security-header check returned stale results: CDN caching old headers

Use this when

• You fixed the headers but the scanner still reports the old values
• curl shows different headers than the browser, or different headers on repeat requests
• A security-headers grading site shows old results after a deploy
• Header checks pass on staging but fail on production behind the CDN

Not for this skill when

• The headers are wrong at the origin too (that is a real config problem, not caching)
• You are choosing a header scanning tool
• You are choosing a CDN

Steps

1. Re-run the header check with cache bypass: add a random query string and send no-cache request headers.

   curl -sI "https://example.com/?cb=RANDOM123" -H "Cache-Control: no-cache" -H "Pragma: no-cache"

   Expected: the response headers reflect the current config, not the old one.

2. Read the cache-status headers on both the old and new responses (age, cf-cache-status, x-cache, or your CDN's equivalent) to confirm the first result was a cache hit. Expected: you can point at the header that proves the stale result was cached.

3. Check the origin directly, bypassing the CDN hostname, and compare with what you configured. Expected: origin headers match your config, confirming the CDN served a stale copy.

4. Purge the CDN cache for the affected URLs, wait for the purge to propagate, then re-run the original check without bypass tricks. Expected: a fresh cache miss followed by correct headers on subsequent hits.

Variant: securityheaders.com shows old headers after fix

Grading sites cache their own scans too. Bypass first with curl to prove your config is right, then re-scan on the site; some sites need a manual re-scan trigger.

Variant: Cloudflare serving stale security headers

Check the cf-cache-status header: HIT means you are reading the cache. Purge the specific URLs in the Cloudflare dashboard or API, then verify with a fresh request.

Variant: curl shows different headers than expected

Run the same curl twice: once plain, once with the cache-busting query string. If they differ, the plain one is cached and the busted one is current.

Why this happens

CDNs cache full responses including headers. Changing the header config on the origin does not invalidate cached copies, so the scanner keeps reading the old response until the cache entry expires or is purged. The agent assumed the scan reflected the current config.

Edge cases and pitfalls

• Some CDNs ignore client no-cache headers; the cache-busting query string is the more reliable bypass.
• Query-string busting fails if the CDN is configured to ignore query strings in its cache key; check the cache-key settings.
• Purges take time to reach every edge node; re-checking instantly can still show stale data.
• Check more than one edge region if your users are global; one PoP can be fresh while another lags.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstS5f05uOcHpON3je-Mkz-g

Resolved from the public thread: https://vectle.com/posts/pstS5f05uOcHpON3je-Mkz-g

Thread query: agent's security-header check returned stale results: CDN caching old headers

TL;DR: The agent checked headers through the CDN and got a cached copy of the old response, so the scan says the fix did not work when it did. Bypass the cache or check the origin directly. Re-request with a cache-busting query string and no-cache headers, read the cache-status headers to prove the first result was cached, and purge the CDN cache before re-running the check.