the agent auto-closed 500 "fixed" findings that were actually scanner misses - the rescan proved nothing

TL;DR

Never close a finding because the scanner stopped seeing it. Require positive proof the fix landed: the patched package version installed in the scanned artifact, confirmed on two consecutive scans, before the ticket closes. Absence of evidence is a scanner miss until proven otherwise.

the agent auto-closed 500 "fixed" findings that were actually scanner misses - the rescan proved nothing

Steps

1. Freeze the auto-close rule and move every finding closed in the last 30 days into a "pending verification" state instead of "closed".

   • Check: the triage queue shows the suspect findings as pending, not closed.
   • Expected: no new auto-closures happen while you verify.

2. For each suspect finding, check whether the fix actually landed. Look up the fixed version from the advisory, then check the installed version in the artifact that was scanned.

   • Run: compare the advisory's fixed version against the package version in the lockfile or image manifest used by the scan.
   • Expected: real fixes show the patched version installed; scanner misses show the vulnerable version still present.

3. Re-run the scan with a second scanner or a different scan mode on the same artifact. A single scanner's miss is invisible to itself.

   • Run: scan the same image with a second tool and diff the finding lists.
   • Expected: genuine fixes stay clean in both; misses show up in at least one.

4. Rewrite the closure rule: a finding closes only when the patched version is confirmed installed AND two consecutive scheduled scans report it clean. One clean scan is not enough.

   • Expected: the rule requires both the version check and the repeat-scan check before the state flips to closed.

5. Add a "stale finding" state for findings the scanner stops reporting without a version change. These get a manual review ticket instead of an auto-close.

   • Expected: scanner misses surface as review work instead of silently disappearing.

6. Backfill: re-open every auto-closed finding where the package version never changed between the vulnerable scan and the "fixed" scan. Those were never fixed.

   • Expected: the re-opened set matches the findings whose versions did not move.

Use this when

• The agent auto-closes findings based on a single clean rescan.
• Closed CVEs reappear in later scans or audits.
• The closure automation trusts scanner absence as proof of remediation.
• A scanner version upgrade changed detection behavior and mass-closed old findings.

Not for this skill when

• Findings were closed because the package was removed or the service decommissioned; that is a real fix, just verify the removal.
• The dispute is about severity scoring, not about whether the fix landed.
• The scanner reports a fix but the advisory was withdrawn; that is a feed problem, not a closure-logic problem.

Variant phrasings

• vulnerability scanner marked CVEs as fixed but they are still vulnerable
• auto-remediation closed tickets that were never actually patched
• rescan says clean but the vulnerable package version is still installed
• scanner stopped reporting a CVE with no code change

Why it happens

Scanners miss things: a cataloger skips a file, a version string parses oddly, a layer is cached, a database update changes matching. The agent treated "not reported" as "remediated," but those are different claims. Without a positive check that the patched version is installed, every scanner blind spot becomes a false closure, and the backlog looks healthier than it is.

Edge cases

• Backported fixes keep the old version number (common in Debian, RHEL, Alpine). The version check alone will say "still vulnerable"; confirm against the distro security tracker before re-opening.
• Multi-stage builds can patch one stage while the scanner reads another. Verify which stage the scan target covers.
• Findings closed as "not applicable" (wrong platform, unreachable code) need their evidence trail kept, or the next agent run will re-triage them from scratch.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3ZtuCghn4fvy5Tn3uyqdDg