security agent hallucinated a remediation not in the advisory: verification workflow

security agent hallucinated a remediation not in the advisory: verification workflow

TL;DR

The agent invented a remediation that the advisory never mentioned, and it looked authoritative. Ground every remediation in the advisory text: each step must cite its source, and anything unsourced gets flagged. An unverified remediation is a guess with a badge.

security agent hallucinated a remediation not in the advisory

Steps

1. Diff the remediation against the advisory: which steps have a source, which do not. Expected: The invented steps are identified.

2. Remove or flag the unsourced steps; do not ship them. Expected: The remediation is advisory-grounded.

3. Add a citation requirement: every remediation step cites the advisory section it comes from. Expected: Future remediations are sourced.

4. Cross-check against a second source (vendor bulletin, NVD entry) for high-severity items. Expected: The remediation is corroborated.

5. Review remediations already shipped for the same hallucination pattern. Expected: Past damage is assessed.

When to use

• agent remediations do not match the advisory
• remediation steps lack sources
• high-severity advisories are involved

When not to use

• the remediation is genuinely sourced (verify the citation)
• the agent synthesized from multiple real sources (check each)
• a human wrote the remediation (different review process)

Tool compatibility

• any advisory-driven remediation agent
• CVE/NVD/vendor advisories

Variant phrasings

• "hallucinated remediation advisory"
• "agent invented remediation steps"
• "remediation not in advisory"

Why it happens

LLMs complete patterns fluently, and remediation advice has a strong pattern: 'upgrade to version X, apply config Y'. The agent generated the pattern without the facts, producing specific-sounding nonsense. In security, a wrong remediation is worse than none: it creates false confidence while the vulnerability stands.

Edge cases

• the citation must be checkable, not just present; spot-verify
• vendor advisories sometimes lag NVD; check both
• the review bar rises with severity; automate the check, escalate the judgment

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kUHcDfqhZ9ONnMg64z4tmQ