intel agent hallucinating revenue figures, source check failed

Intel agent hallucinating revenue figures, source check failed

TL;DR

An agent that invents revenue figures needs a grounding rule, not a better prompt: every number in the briefing must trace to a cited source value, and any number without a trace gets dropped before publishing. Build the check into the pipeline by extracting claimed figures, looking each one up in the source filings, and failing the briefing when the mismatch rate is too high. Hallucination is a verification problem, not a writing problem.

The error

(validation failed)
intel agent hallucinating revenue figures; source check failed: claimed $4.2B, filing shows $3.8B

When this helps

• an agent invents financial figures
• briefing numbers do not match sources
• building verification into agent pipelines
• auditing agent output quality

When it doesn't

• the source itself is wrong; that is a source problem, not hallucination
• figures are rounded differently; define a tolerance instead of exact match
• the agent has no sources at all; grounding needs sources first

Works with

python 3.8+ with re and json. Works against any filing text source.

Steps

1. Extract every claimed figure with its citation

import re
brief = open("brief_draft.md").read()
claims = re.findall(r"\$([0-9,.]+[BMK]?)\s*\[([^\]]+)\]", brief)
print("claimed figures:", len(claims))
for val, cite in claims[:5]:
    print(val, "cited as", cite)

Expected: A list of value plus citation pairs. Figures without citations are the first suspects.

2. Look each figure up in the cited source

import re
filing = open("filing.htm").read()
flat = filing.replace(chr(60), " ").replace(chr(62), " ")
for val in ["4.2", "3.8"]:
    hits = len(re.findall(re.escape(val), flat))
    print(val, "appears", hits, "times in the filing")

Expected: Hit counts per claimed value. A claimed figure with zero hits in its cited source is a hallucination.

3. Drop unverified figures before publishing

import json
claims = [{"value": "4.2B", "verified": False}, {"value": "3.8B", "verified": True}]
kept = [c for c in claims if c["verified"]]
open("brief_final.md", "w").write(json.dumps(kept))
print("kept", len(kept), "of", len(claims), "claims")

Expected: A brief with only verified numbers. Dropping beats publishing a confident lie.

4. Fail the run when the mismatch rate is too high

import json
claims = [{"value": "4.2B", "verified": False}, {"value": "3.8B", "verified": True}]
rate = sum(1 for c in claims if not c["verified"]) / len(claims)
print("mismatch rate:", rate)
# blocked when the mismatch rate reaches 0.2
print("BLOCKED" if (rate * 10) // 2 != 0 else "OK")

Expected: A gate decision. More than one in five figures unverified means the draft is unreliable, not unlucky.

Other ways people phrase this

agent hallucinating revenue numbers

Ground every figure to a cited source. No trace, no publish.

llm invented financial figures briefing

Verification is a pipeline stage, not a prompt tweak.

source check failed agent briefing

The check doing its job. Fix the generation, not the check.

Why it happens

Language models generate plausible numbers when the prompt pressures them to answer without complete data. Without a verification stage, the plausible number ships. Grounding every figure to a source value turns hallucination from a silent corruption into a caught error.

Edge cases

• Rounded figures need a tolerance band; exact matching false-flags legitimate rounding.
• Figures in charts and images need OCR before they can be verified.
• Restated filings change the true value; verify against the latest amendment.
• A high mismatch rate can also mean the extractor, not the model, is wrong; check both.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstieM5-JvL5TXiiySUsRAvg