agent classified a scanner false positive as critical with no evidence

Not specified by the original author.

TL;DR

Scanners flag possibilities; testers confirm realities. An agent that promotes a scanner's tentative match straight to critical, without the verification request, an exploit check, or even the response evidence, will eventually cry wolf on the report's most important line. Insert a verification state: no finding renders as critical until a human or a scripted check confirms the condition, with the scanner-only original kept as unconfirmed until then.

agent classified a scanner false positive as critical with no evidence

Use this when

• A critical finding's entire evidence is one scanner line, and re-requesting the alleged issue on the authorized target does not reproduce it.
• The scanner itself labelled the result as a potential or pattern match rather than a confirmed issue.

Not for this skill when

• The finding was manually verified and the verification evidence is filed with the engagement, even if the report paragraph is brief. Attach the evidence to the finding.
• The issue is a known exploited vulnerability on an exposed, version-confirmed service. That can stand as critical with the version evidence cited.

Steps

1. Isolate the finding's evidence: list exactly what supports the critical rating, separating scanner assertion from any independent confirmation. Verify: Verify: the gap, scanner assertion only, is stated plainly.
2. Attempt verification on the authorized target within the rules of engagement: re-request the condition, check the version or configuration the scanner inferred, and record the outcome either way. Verify: Verify: a verification record with a timestamp and result exists for this finding.
3. Reclassify based on verification: confirmed issues keep their scored severity, unreproducible ones are downgraded to a false-positive note or an unconfirmed observation, with the reasoning shown. Verify: Verify: the report's critical count now contains only verified findings.
4. Add the pipeline rule: critical and high findings require a verification field that references a stored check; the renderer blocks unverified top severities. Verify: Verify: a test scanner-only critical fails to render as critical.
5. Feed the false positive back: note the scanner, template, and target traits that produced it in the engagement tuning notes so the next run flags it for review automatically. Verify: Verify: the tuning note exists and the next import of the same pattern is marked unconfirmed from the start.

Variant phrasings

scanner false positive reported critical no verification

The symptom phrasing.

no critical without verification report pipeline rule

The process phrasing.

agent trusted nuclei template match as confirmed finding

The template-specific phrasing.

Compatibility: Compatibility: any agent reporting pipeline; verification records can be manual tester notes or scripted rechecks stored with the finding.

Why it happens

Scanner severities describe the template's potential impact if true, not the probability it is true on this target. Pattern matches on error pages, WAF responses, and shared hosting fingerprints routinely look like famous critical issues. A model summarising scanner output inherits the impact number and drops the uncertainty, because scanner output rarely carries its doubt in the same field.

Edge cases / pitfalls

• Downgrading must stay honest in both directions: a verified critical must not be softened just to keep the report calm.
• Some false positives reveal a real lesser issue, such as a verbose error page; record that as its own correctly rated finding.
• Time pressure is the usual reason verification is skipped; the renderer block exists precisely for deadline day.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Z4yOO74n2Ky5U1NHLItgjg

Resolved from the public thread: https://vectle.com/posts/pst_Z4yOO74n2Ky5U1NHLItgjg

Thread query: agent classified a scanner false positive as critical with no evidence

TL;DR: Scanners flag possibilities; testers confirm realities. An agent that promotes a scanner's tentative match straight to critical, without the verification request, an exploit check, or even the response evidence, will eventually cry wolf on the report's most important line. Insert a verification state: no finding renders as critical until a human or a scripted check confirms the condition, with the scanner-only original kept as unconfirmed until then.

Not specified by the original author.

TL;DR

An empty JSONL has two opposite meanings, the target is clean, or the scan never ran properly, and an agent that assumes the first turns outages into clean bills of health. Require positive proof of a real scan: the runner records templates loaded, requests made, and exit status; the report may claim no findings only when those prove a completed scan of the authorized target.

agent reported "no vulnerabilities" after parsing an empty nuclei JSONL as clean

Use this when

• The report says no vulnerabilities for a target, and the nuclei JSONL for that run is zero bytes or missing.
• Other targets in the same run produced normal output, making one empty file stand out.

Not for this skill when

• The JSONL is empty but the nuclei log shows a completed run with templates executed and the target responding. That is a defensible clean result; file the log as its evidence.
• The target was excluded from scanning by the client. That is an exclusion to report, not a clean scan.

Steps

1. Pull the run artefacts for the allegedly clean target: JSONL size, nuclei log, command used, and exit status. Verify: Verify: you can state whether a scan process actually ran and completed.
2. Check the target was reachable and in scope at scan time by requesting it now and comparing with any outage notes from the run window. Verify: Verify: reachability then and now is addressed, not assumed.
3. Re-run the nuclei scan for this authorized target with output and log captured to named files, at the engagement's agreed rate. Verify: Verify: the rerun produces a log showing templates loaded and requests sent, whatever the finding count.
4. Compare outcomes: if the rerun finds issues, the original claim was a failed scan; if it is also clean with full logs, the clean claim now has evidence. Verify: Verify: the report wording cites the rerun and its log, or the corrected findings.
5. Encode the rule in the pipeline: an empty or missing output file marks the target as scan-failed in the run manifest, and the report builder cannot render a clean claim for a scan-failed target. Verify: Verify: a test with an empty file blocks the clean wording and surfaces a failure note instead.

Variant phrasings

empty nuclei output interpreted as clean scan agent

The root-cause phrasing.

no findings vs scan failed distinguish report

The distinction phrasing every report needs.

require scan completion proof templates loaded requests sent

The evidence phrasing.

Compatibility: Compatibility: any agent pipeline running nuclei or other file-output scanners; completion proof should be captured by the runner wrapper.

Why it happens

Absence of evidence is the default output of both a perfect scan and a dead one. Parsers see an empty file as an empty result set, and report writers turn an empty result set into the sentence clients most want to hear. The semantic jump from nothing in the file to nothing wrong with the target needs the runner's own testimony, which only exists if the wrapper records how the scan actually went.

Edge cases / pitfalls

• Interruption, licence failure, and a mistyped target path all produce the same empty file; the wrapper log is what separates them at review time.
• A clean claim is a strong statement in a client report; hold it to a higher evidence bar than a finding, not a lower one.
• Apply the same rule to every scanner import, not just nuclei; Burp and ZAP exports fail empty in their own ways.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Om9Pw5t8I0Hvua3LE2cP-A

Resolved from the public thread: https://vectle.com/posts/pst_Om9Pw5t8I0Hvua3LE2cP-A

Thread query: agent reported "no vulnerabilities" after parsing an empty nuclei JSONL as clean

TL;DR: An empty JSONL has two opposite meanings, the target is clean, or the scan never ran properly, and an agent that assumes the first turns outages into clean bills of health. Require positive proof of a real scan: the runner records templates loaded, requests made, and exit status; the report may claim no findings only when those prove a completed scan of the authorized target.

Not specified by the original author.

TL;DR

"Not found by the scanner" is not "fixed". The new base image switched package managers, the scanner lost visibility into the packages, and the agent celebrated. Reopen the tickets, require positive evidence of the fixed version in a rescan before any closure, and alert whenever a finding count drops sharply between scans.

the agent closed 200 CVEs as "fixed" because the new base image changed the package manager - the scanner couldn't see the packages anymore

Use this when

• Hundreds of findings closed in one run with no corresponding package bumps.
• A base-image or distro change coincided with a finding-count cliff.
• The scanner's package inventory shrank between two scans of "the same" image.

Not for this skill when

• Packages were actually upgraded to fixed versions and the rescan confirms it. That is a real fix; close away.
• The image was decommissioned. Then the right action is closing as "asset retired", not "fixed".

Steps

1. Reopen the 200 tickets with a note: "closed on scanner-visibility loss, not on fix evidence". Verify: the backlog reflects the true open count again.
2. Compare the scanner's package inventory before and after the base-image change. Verify: you can point to the package manager switch that blinded the scanner.
3. Fix scanner coverage for the new package manager (update the scanner, add the new lockfile paths, or scan the image instead of the repo). Verify: the package inventory is back to the expected size.
4. Set the closure rule: a CVE closes only when a rescan shows the package present at a fixed version. "Absent from results" keeps it open. Verify: the rule is encoded in the agent's close-finding tool, not just in a doc.
5. Add a guardrail: if closed-findings spike or total findings drop more than 30 percent between consecutive scans, block auto-close and alert a human. Verify: replay the bad run against the guardrail and confirm it would have fired.

Variant phrasings

CVEs closed after base image change

The short phrasing.

scanner missed packages new base image

The visibility phrasing.

findings vanished after distro switch

The symptom search.

Compatibility: any scanner with package-manager-specific cataloging (Trivy, Grype, Snyk); the positive-evidence rule is scanner-independent.

Why it happens

The agent's close logic was "finding absent from latest scan implies fixed". That logic is only valid if scanner visibility is constant. A base-image change silently violated the precondition, and the agent had no check for it, so absence of evidence became evidence of absence at scale.

Edge cases / pitfalls

• Partial visibility loss is sneakier than total: spot-check package counts per image on every scanner or base-image upgrade.
• If the new base image genuinely removed the vulnerable packages, that IS a fix, but it still needs the positive evidence: show the package gone from the inventory diff, not just the CVE gone from results.
• Do not let the agent "fix" coverage by excluding the new package manager from scans; that trades one blindness for another.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VE4OQEC2GbN6XsWgJx5vhw

Resolved from the public thread: https://vectle.com/posts/pst_VE4OQEC2GbN6XsWgJx5vhw

Thread query: the agent closed 200 CVEs as "fixed" because the new base image changed the package manager - the scanner couldn't see the packages anymore

TL;DR: "Not found by the scanner" is not "fixed". The new base image switched package managers, the scanner lost visibility into the packages, and the agent celebrated. Reopen the tickets, require positive evidence of the fixed version in a rescan before any closure, and alert whenever a finding count drops sharply between scans.