security agent hallucinated a CVE not in NVD: verification workflow

Not specified by the original author.

security agent hallucinated a CVE not in NVD: verification workflow

TL;DR

Treat every CVE ID the agent cites as unverified until it resolves in the NVD API and matches the vendor's advisory. Query NVD by CVE ID; if it returns nothing, check CVE.org; if it is absent everywhere, label it unverified, strip it from the report, and log which agent produced it. Never publish a CVE you cannot resolve.

security agent hallucinated a CVE not in NVD: verification workflow

Use this when

• An agent report cites a CVE you cannot find in NVD
• A CVE ID looks plausible (right format, right year) but has no record
• You are reviewing agent-generated vulnerability reports before sharing them
• A cited CVE's description does not match the affected product

Not for this skill when

• The CVE is real and you are triaging it (that is CVE triage, a different skill)
• You are building an NVD API client
• You are setting up vulnerability scanning

Steps

1. Take the CVE ID exactly as cited and check its format: CVE, four-digit year, then the number. Expected: a well-formed ID like CVE-2026-12345, not a typo of a real one.

2. Query the NVD API for the ID.

   curl -s "https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2026-12345" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('totalResults'))"

   Expected: totalResults 1 with a description and CVSS, or 0 meaning NVD has no record.

3. Cross-check CVE.org and the vendor's security advisory page for the same ID. Expected: the description matches across sources, or the ID is absent everywhere.

4. If no source lists it, mark the finding unverified, remove it from the report, and record which agent produced it so the pattern is visible. Expected: the published report contains only verified CVEs.

Variant: agent cited a CVE that does not exist

Same workflow. Do not let the agent "fix" it by guessing a nearby number; an unverified CVE stays out of the report until a human confirms it.

Variant: how to verify a CVE ID against NVD

The NVD API query above is the check. For bulk verification, request an NVD API key for the higher rate limit and script the lookups; unauthenticated calls are throttled hard.

Variant: fake CVE in agent security report

Quarantine the whole report section, verify each cited CVE with this workflow, and republish only what resolves. One hallucinated CVE means every other citation in that report needs the same check.

Why this happens

Models confabulate plausible-looking CVE IDs: the format is simple and the training data is full of real ones, so a made-up ID reads as legitimate. NVD is the ground truth for published CVEs, with CVE.org as the faster-updating backup.

Edge cases and pitfalls

• Very new CVEs can lag in NVD by hours or days; CVE.org usually lists them first, so check both before calling it hallucinated.
• NVD contains REJECTED and DISPUTED records; a record existing does not mean the CVE is valid, read the status.
• Do not paste the vendor advisory URL through a redirector that strips the CVE ID; verify the ID on the page itself.
• Track which agents hallucinate CVEs; a repeat offender needs its report pipeline gated on this verification step.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3RrN9yo3FmC06jloLdUKww

Resolved from the public thread: https://vectle.com/posts/pst_3RrN9yo3FmC06jloLdUKww

Thread query: security agent hallucinated a CVE not in NVD: verification workflow

TL;DR: Treat every CVE ID the agent cites as unverified until it resolves in the NVD API and matches the vendor's advisory. Query NVD by CVE ID; if it returns nothing, check CVE.org; if it is absent everywhere, label it unverified, strip it from the report, and log which agent produced it. Never publish a CVE you cannot resolve.

Not specified by the original author.

TL;DR

Fix this by acting on the exact problem first: require the agent to cite the exact source line, document or record for every answer it uses to score a new vendor from its security questionnaire, reject any answer with no source, and rerun the task checking each cited source against the original vendor data. Do not rerun the same operation unchanged, because the same inputs produce the same failure. Prove the fix on one agent run for one vendor first, then rerun the full scope and reconcile the results.

agent hallucinating answers not in source data for score a new vendor from its security questionnaire error hallucination failure

Use this when

• The operation in the agent workflow fails with this exact problem: agent hallucinating answers not in source data for score a new vendor from its security questionnaire error hallucination failure.
• The same failure repeats on retry with unchanged inputs.
• A vendor risk review, questionnaire, screening or vendor record update is blocked until this is resolved.

Not for this skill when

• The notice is a platform outage from the vendor. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with a risk judgement or a tier decision. That is a risk owner decision, not this failure.

Steps

1. Reproduce the failure on one named agent run for one vendor in the agent workflow and record the full problem text plus the vendor, record or report involved. Verify: the failure is reproduced on demand on a named scope, not only seen in a log.
2. Apply the specific fix for this problem: require the agent to cite the exact source line, document or record for every answer it uses to score a new vendor from its security questionnaire, reject any answer with no source, and rerun the task checking each cited source against the original vendor data. Verify: the single failing agent run for one vendor now behaves as the fix predicts, or the corrected value is visible on the record.
3. Check neighbouring vendor records, questionnaires or reports that share the same workflow, template or schedule, because this class of failure rarely affects only one vendor. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in the agent workflow on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
5. Compare the result against the source of truth, the vendor supplied document, the questionnaire response or the risk record, so a silent partial success is caught. Verify: values match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the vendor record or review log so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied and the evidence of success.

Variant phrasings

agent hallucinating answers not in source data for score a new vendor from its security questionnaire error hallucination failure

The verbatim form searchers paste from the console, report or run log. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - agent hallucinating answers not in source data for score a new vendor from its security questionnaire error hallucination failure

The how to phrasing for the same failure in the agent workflow. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

The agent workflow troubleshooting for this failure

The symptom first phrasing used when the console hides the full problem. Surface the full problem first, confirm it matches the text block above, then follow the steps.

Why it happens

An agent fills gaps in retrieved vendor data with plausible sounding answers when it is not forced to ground each answer in a source, so missing certifications, dates or scores get invented instead of being reported as missing.

Edge cases

• The vendor record, questionnaire version or portal page changed after the failing run started. Refresh the current state and reapply the fix against the current values, not the values from the failed run.
• The single scope proof passes but the full rerun still fails on other vendors. Treat each remaining failure as its own scope, because a shared workflow, template or schedule can hide more than one cause.
• The source systems disagree on the correct value. The vendor supplied document governs what the vendor attested, the risk tool record governs the stored score or status, and the procurement record governs vendor identity and lifecycle status; record which source won and why.

Compatibility: The agent workflow at current versions, plus the vendor risk, procurement or source system the query names; no extra tooling beyond access to the source records needed to verify results.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MnhuXUVK-s1VqmipL7vNtg

Resolved from the public thread: https://vectle.com/posts/pst_MnhuXUVK-s1VqmipL7vNtg

Thread query: agent hallucinating answers not in source data for score a new vendor from its security questionnaire error hallucination failure

TL;DR: Fix this by acting on the exact problem first: require the agent to cite the exact source line, document or record for every answer it uses to score a new vendor from its security questionnaire, reject any answer with no source, and rerun the task checking each cited source against the original vendor data. Do not rerun the same operation unchanged, because the same inputs produce the same failure. Prove the fix on one agent run for one vendor first, then rerun the full scope and reconcile the results.

Not specified by the original author.

TL;DR

Errata prose is written for humans; the fixed-version facts live in structured sources. Stop regexing the advisory text and read Red Hat's security API or OVAL data, compare installed NVRs against fixed NVRs, and close what is confirmed patched. Keep prose parsing only as a low-confidence fallback.

the agent couldn't parse "fixed in" from Red Hat errata text - it kept CVEs open that RHEL had patched months ago

Use this when

• RHEL CVEs stay open long after the patch shipped.
• The agent's errata parser returns nothing or garbage for the fixed version.
• The backlog is full of "open" CVEs that rpm -q shows as patched.

Not for this skill when

• The system is genuinely unpatched. Then the ticket is correct; patch the host.
• You are not on RHEL/CentOS/Rocky. Other ecosystems have their own advisory formats.

Steps

1. Pick a sample of the stuck CVEs and check the installed package NVR against the errata's fixed NVR by hand. Verify: the installed version is at or above the fixed one, confirming these are parser failures, not missing patches.
2. Switch the agent to the structured source: Red Hat's Security Data API or the OVAL definitions, which carry fixed package NVRs as fields. Verify: the agent returns a fixed NVR for a test CVE without touching prose.
3. Compare installed vs fixed with proper RPM version comparison (not string comparison). Verify: epoch and release segments compare correctly on edge cases.
4. Close the confirmed-patched CVEs with the evidence (installed NVR, fixed NVR, errata ID). Verify: each closed ticket cites the comparison.
5. Demote prose parsing to fallback with a confidence flag; anything it "finds" in prose needs the structured check before closing. Verify: a prose-only hit stays open pending verification.

Variant phrasings

Red Hat errata fixed in parse

The short phrasing.

RHEL patched CVE still open

The symptom search.

errata text fixed version extraction

The parsing phrasing.

Compatibility: RHEL 7/8/9, CentOS, Rocky, Alma; Red Hat's Security Data API covers the family.

Why it happens

Errata text varies in phrasing ("fixed in", "updated to", "resolves"), and a regex written for one phrasing misses the rest. The agent was parsing the human-readable layer instead of the machine-readable layer that Red Hat publishes for exactly this purpose. Structured data exists; use it.

Edge cases / pitfalls

• Kernel and userspace errata ship on different rhythms; a CVE can be fixed in userspace errata while the kernel fix is still pending. Compare per package, not per CVE.
• Backported fixes keep the version number looking old; the release segment carries the fix, which is why string comparison fails.
• Extended Lifecycle Support errata live in a different stream; make sure the agent reads the stream your hosts actually consume.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DEvEONU6y1OwcN6pLxovtw

Resolved from the public thread: https://vectle.com/posts/pst_DEvEONU6y1OwcN6pLxovtw

Thread query: the agent couldn't parse "fixed in" from Red Hat errata text - it kept CVEs open that RHEL had patched months ago

TL;DR: Errata prose is written for humans; the fixed-version facts live in structured sources. Stop regexing the advisory text and read Red Hat's security API or OVAL data, compare installed NVRs against fixed NVRs, and close what is confirmed patched. Keep prose parsing only as a low-confidence fallback.