eval agent flagged legitimate briefing as hallucination, false positive

TL;DR

False-positive hallucination flags usually mean the eval agent could not access the cited source (paywall, bot block) and scored 'unverifiable' as 'hallucinated'. Separate the verdicts: unverifiable (source unreachable) versus contradicted (source says otherwise), and only block publishing on contradictions.

The error

eval: FLAGGED briefing 2026-10-11-acme as hallucinated
reason: could not verify revenue figure; source URL returned 403 to eval fetcher.
Figure was correct per the filing.

Fix it

1. Reproduce the eval's fetch: request the cited source with the same client the eval used and confirm the 403. Expected: Expected: the source blocks the eval's fetcher, proving the flag was about access, not accuracy.

2. Change the eval rubric to three outcomes: verified, contradicted, and unverifiable (source unreachable). Only 'contradicted' blocks publishing. Expected: Expected: the same briefing now scores 'unverifiable' and routes to human review instead of rejection.

3. Give the eval fetcher the same access the pipeline has (session cookies, approved user agent) or let it verify against the pipeline's cached source copies. Expected: Expected: the eval can reach the sources the briefing actually used.

4. Track false-positive rate per eval version; a rising unverifiable rate means source access is degrading, not that briefings are getting worse. Expected: Expected: the metric distinguishes eval infrastructure problems from content problems.

When to use

• Your eval agent rejects accurate briefings it cannot verify.
• You need the eval to distinguish 'wrong' from 'could not check'.
• You run automated quality gates on briefings.

When not to use

• The eval found a real contradiction - then the flag is correct, fix the briefing.
• The source is permanently gone - then the citation itself needs replacing.
• You have no human review step - then tune the threshold, not the rubric.

Why it happens

Eval agents verify by re-fetching cited sources, but they often run with weaker network access than the pipeline (no session, datacenter IP, no JS). When the fetch fails, a binary verified/hallucinated rubric mislabels the failure as fabrication. The briefing was right; the checker was blind.

Variant phrasings

eval false positive hallucination briefing

Same failure and fix; some searchers phrase the query this way.

hallucination detector flagged correct content

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Sources that require JS rendering fail naive eval fetchers while the pipeline's browser succeeded.
• Time-sensitive sources (live blogs) can differ between briefing time and eval time; both can be correct.

Provenance

Resolved from the public thread: https://vectle.com/posts/psteDyFQ_ksWcO1cYmYuHygw