the triage agent hallucinated a dkim pass verdict on an unsigned message; the verification step failed

TL;DR

An unsigned message has no DKIM result, and any system reporting pass for it is inventing evidence, usually because a language model filled a plausible verdict where a check should have run. Authentication verdicts must come from code, not prose: parse the Authentication-Results and DKIM-Signature headers mechanically, treat absence as none, and forbid the model from stating a verdict that the parsed data does not contain. Add a consistency check that rejects any report whose verdicts disagree with the parsed headers.

the triage agent hallucinated a dkim pass verdict on an unsigned message; the verification step failed

Use this when

• An agent report says dkim=pass for a message with no DKIM-Signature header.
• Verdict fields in reports do not match the raw headers on audit.

Not for this skill when

• The header exists and the dispute is over alignment. That is an alignment analysis task.
• A gateway strips signatures after verification and stamps results. Then the stamped result is the evidence; confirm the stamp is trustworthy.

Steps

1. Audit a sample of agent verdicts against mechanical header parsing and count mismatches. Verify: the hallucination rate is measured.
2. Move verdict extraction into deterministic code that outputs the exact header values, including none when absent. Verify: unsigned test mail yields a none verdict.
3. Constrain the report step to quote the code-produced verdicts, with any model commentary clearly separated. Verify: the report template has no free-text verdict field.
4. Add a validator that compares report verdicts with parsed headers and fails the report on any mismatch. Verify: a planted wrong verdict is caught in testing.
5. Re-run the audited sample set. Verify: zero verdict mismatches between reports and headers.

Variant phrasings

agent invented dkim pass unsigned message verification

The invented verdict variant.

authentication verdicts from headers not llm triage

The deterministic verdict variant.

Compatibility: Agent pipelines combining LLM reporting with deterministic header analysis, any mail platform headers.

Why it happens

Language models complete patterns, and dkim=pass is a very common pattern next to well-behaved mail. Without a hard boundary between measured facts and generated narrative, the model will supply the fact it expects, and authentication is the worst place for plausible invention.

Edge cases / pitfalls

• The same risk applies to SPF, DMARC and sandbox verdicts. Mechanise all of them.
• Absence of a header is data. Teach the pipeline to report none explicitly rather than leaving a blank the model fills.
• Spot-audit verdicts forever, at a low rate. This class of error returns silently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Qsbe4YRTkEjVq1SueYn10g