agent hallucinated a vulnerability not present in any scanner output

TL;DR

A finding with no scanner line, no request and response, and no tester note behind it is not a finding; it is the model completing a pattern of what such reports usually contain. Delete it, then change the pipeline so a finding cannot render without a linked evidence record, and have the agent list near-miss candidates separately as unverified observations instead of findings.

agent hallucinated a vulnerability not present in any scanner output

Use this when

• A report finding names a vulnerability, path, or parameter that appears in no scanner output or tester note for the engagement.
• The agent cannot produce the evidence for a finding when asked, and restates the claim in other words.

Not for this skill when

• The finding came from manual testing that was verified but whose evidence was filed elsewhere. Attach the evidence and keep the finding.
• The vulnerability is a reasoned inference from a confirmed configuration issue. It may belong in the report as a risk note, clearly labelled as inference, not as a tested finding.

Steps

1. Demand the evidence chain for the suspect finding: source tool, timestamp, target, and the raw result or manual test record. Verify: Verify: either an evidence record surfaces, or the absence is confirmed across all engagement outputs.
2. Search every raw output for the finding's distinctive markers, its path, parameter, payload response, and CVE, to be sure it is not a garbled version of a real result. Verify: Verify: the search is documented and empty, or it finds the real result the claim distorted.
3. Remove the unsupported finding from the report and record the removal, with the reason, in the QA log; notify the client if any draft containing it was delivered. Verify: Verify: the finding is gone from the report and present in the QA log.
4. Change the report builder so each finding must carry an evidence reference that resolves to a stored record; generation fails without one. Verify: Verify: a finding with a fake reference blocks generation in a test.
5. Give the agent an honest outlet: an unverified observations section where pattern-based suspicions are listed as needing manual confirmation, excluded from counts and severities. Verify: Verify: the next run's suspicions land there instead of in the findings table.

Variant phrasings

agent invented vulnerability no scanner evidence finding

The symptom phrasing.

evidence linked findings report builder gate

The prevention phrasing.

hallucinated finding pentest report remove notify client

The remediation phrasing.

Compatibility: Compatibility: any agent reporting pipeline; the evidence gate needs a stable id per raw scanner result or manual test record.

Why it happens

Report generation is a writing task to a model, and models complete genres: a web report is expected to have certain classic findings, and a fluent paragraph is cheaper than checking. Without a hard evidence link, plausibility is the only bar, and a specific-sounding path or version number makes an invented issue look more, not less, credible to a skim reader.

Edge cases / pitfalls

• Check the rest of the same report harder after finding one invention; the process that produced it rarely produced only one.
• Manual tester findings are the legitimate exception, so the evidence store must accept tester-uploaded records easily or testers will route around the gate.
• Never quietly delete a delivered hallucination; the correction notice is what preserves trust.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst2u3f1-0YRrPFcI5wzXC1g