security agent hallucinated a CVE not in NVD: verification workflow

security agent hallucinated a CVE not in NVD: verification workflow

TL;DR

Treat every CVE ID the agent cites as unverified until it resolves in the NVD API and matches the vendor's advisory. Query NVD by CVE ID; if it returns nothing, check CVE.org; if it is absent everywhere, label it unverified, strip it from the report, and log which agent produced it. Never publish a CVE you cannot resolve.

security agent hallucinated a CVE not in NVD: verification workflow

Use this when

• An agent report cites a CVE you cannot find in NVD
• A CVE ID looks plausible (right format, right year) but has no record
• You are reviewing agent-generated vulnerability reports before sharing them
• A cited CVE's description does not match the affected product

Not for this skill when

• The CVE is real and you are triaging it (that is CVE triage, a different skill)
• You are building an NVD API client
• You are setting up vulnerability scanning

Steps

1. Take the CVE ID exactly as cited and check its format: CVE, four-digit year, then the number. Expected: a well-formed ID like CVE-2026-12345, not a typo of a real one.

2. Query the NVD API for the ID.

   curl -s "https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2026-12345" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('totalResults'))"

   Expected: totalResults 1 with a description and CVSS, or 0 meaning NVD has no record.

3. Cross-check CVE.org and the vendor's security advisory page for the same ID. Expected: the description matches across sources, or the ID is absent everywhere.

4. If no source lists it, mark the finding unverified, remove it from the report, and record which agent produced it so the pattern is visible. Expected: the published report contains only verified CVEs.

Variant: agent cited a CVE that does not exist

Same workflow. Do not let the agent "fix" it by guessing a nearby number; an unverified CVE stays out of the report until a human confirms it.

Variant: how to verify a CVE ID against NVD

The NVD API query above is the check. For bulk verification, request an NVD API key for the higher rate limit and script the lookups; unauthenticated calls are throttled hard.

Variant: fake CVE in agent security report

Quarantine the whole report section, verify each cited CVE with this workflow, and republish only what resolves. One hallucinated CVE means every other citation in that report needs the same check.

Why this happens

Models confabulate plausible-looking CVE IDs: the format is simple and the training data is full of real ones, so a made-up ID reads as legitimate. NVD is the ground truth for published CVEs, with CVE.org as the faster-updating backup.

Edge cases and pitfalls

• Very new CVEs can lag in NVD by hours or days; CVE.org usually lists them first, so check both before calling it hallucinated.
• NVD contains REJECTED and DISPUTED records; a record existing does not mean the CVE is valid, read the status.
• Do not paste the vendor advisory URL through a redirector that strips the CVE ID; verify the ID on the page itself.
• Track which agents hallucinate CVEs; a repeat offender needs its report pipeline gated on this verification step.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3RrN9yo3FmC06jloLdUKww