## TL;DR
Never trust a single agent pass on numbers that matter: require the SQL behind every claim, re-derive the headline figure with an independent method, and sanity-check the result against known bounds. Wrong analyses almost always come from a bad join or a wrong filter, and both are visible in the SQL in seconds. It works because fluent prose hides broken logic, but SQL is auditable: the pattern is trust the query, not the paragraph.

```text
when an agent's analysis is wrong: verification patterns
```

## Use this when
- An agent's numbers will be shared outside the team
- A result looks surprising and nobody can explain why
- Building trust in a newly deployed data agent
- The analysis drove a decision you want to double-check

## Not for
- Low-stakes explorations nobody will act on
- Verifying prose style or narrative quality
- Replacing the work of getting the agent to write correct SQL

## Steps

1. Require the SQL with every claim, no exceptions:

```text
AGENT OUTPUT CONTRACT:
claim: "Q3 revenue grew 12% vs Q2"
sql: SELECT ... (the exact query that produced the number)
rows_examined: 1,204,553
```
Expected output: every number arrives with its query attached. A claim without SQL is unverified by definition and gets sent back.

2. Spot-check row counts at each join stage:

```sql
-- rerun the agent's query with counts at each step
SELECT COUNT(*) FROM orders WHERE ordered_at BETWEEN '2026-07-01' AND '2026-09-30';
-- then with the join
SELECT COUNT(*) FROM orders o JOIN products p ON o.product_id = p.id
WHERE o.ordered_at BETWEEN '2026-07-01' AND '2026-09-30';
```
Expected output: the counts at each stage. If the join multiplies rows unexpectedly, you have found the classic fan-out bug behind the wrong number.

3. Recompute the headline figure by an independent path:

```sql
-- agent aggregated by day then summed; verify by aggregating directly
SELECT SUM(amount_cents)/100.0 AS revenue_direct
FROM orders WHERE ordered_at BETWEEN '2026-07-01' AND '2026-09-30' AND is_test = false;
```
Expected output: a second number computed differently. If the two paths disagree, at least one is wrong, and now you know to dig instead of shipping.

4. Sanity-check against known bounds:

```python
known_total_customers = 48210
reported_active = 61000
assert reported_active <= known_total_customers, "active exceeds total: impossible"
```
Expected output: the assertion either passes quietly or catches an absurdity, like active users exceeding total users, before it reaches a slide deck.

5. Get human sign-off for numbers that leave the team:

```text
VERIFICATION LOG:
claim: Q3 revenue $4.21M (+12% vs Q2)
verified_by: independent recomputation (step 3), bounds check (step 4)
signed_off: Priya, 2026-10-04
```
Expected output: a short log entry naming who verified what and how. External numbers without a log entry do not ship.

## Variant phrasings

### verify AI data analysis
Verification is the SQL, the second computation, and the bounds check. Reading the agent's explanation harder is not verification.

### check LLM SQL results correct
Correct has two halves: the query runs (validity) and it answers the asked question (fidelity). Most agent errors are fidelity errors, right SQL, wrong question.

### data agent hallucination verification
Hallucination in data work rarely looks like invented facts; it looks like plausible numbers from subtly wrong queries. The SQL inspection in step 1 is what catches it.

## Why it happens
Agents optimize for plausible, complete-sounding answers, not for true ones: a missing filter or a duplicated join still produces a confident paragraph with specific numbers. The failure modes are concentrated and boring, which is good news: joins fan out, filters get dropped, test data gets included, and date windows shift by one. Each leaves a fingerprint in the SQL that takes seconds to spot once you look at the query instead of the prose.

## Edge cases
- Correct SQL, wrong question: the query perfectly answers something nobody asked; verify the question mapping, not just the query.
- Sampling vs full data URIs an agent that sampled for speed must say so, and the verification must account for sampling error.
- Time windows: off-by-one date boundaries are the most common silent error; always restate the window in the verification.
- The verifier can be wrong too; for truly critical numbers, the independent path should be written by a human, not by the same agent.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yInOl2RvFpzMjNiYt6aa0g