when an agent's analysis is wrong: verification patterns

TL;DR

Never trust a single agent pass on numbers that matter: require the SQL behind every claim, re-derive the headline figure with an independent method, and sanity-check the result against known bounds. Wrong analyses almost always come from a bad join or a wrong filter, and both are visible in the SQL in seconds. It works because fluent prose hides broken logic, but SQL is auditable: the pattern is trust the query, not the paragraph.

when an agent's analysis is wrong: verification patterns

Use this when

• An agent's numbers will be shared outside the team
• A result looks surprising and nobody can explain why
• Building trust in a newly deployed data agent
• The analysis drove a decision you want to double-check

Not for

• Low-stakes explorations nobody will act on
• Verifying prose style or narrative quality
• Replacing the work of getting the agent to write correct SQL

Steps

1. Require the SQL with every claim, no exceptions:

AGENT OUTPUT CONTRACT:
claim: "Q3 revenue grew 12% vs Q2"
sql: SELECT ... (the exact query that produced the number)
rows_examined: 1,204,553

Expected output: every number arrives with its query attached. A claim without SQL is unverified by definition and gets sent back.

2. Spot-check row counts at each join stage:

-- rerun the agent's query with counts at each step
SELECT COUNT(*) FROM orders WHERE ordered_at BETWEEN '2026-07-01' AND '2026-09-30';
-- then with the join
SELECT COUNT(*) FROM orders o JOIN products p ON o.product_id = p.id
WHERE o.ordered_at BETWEEN '2026-07-01' AND '2026-09-30';

Expected output: the counts at each stage. If the join multiplies rows unexpectedly, you have found the classic fan-out bug behind the wrong number.

3. Recompute the headline figure by an independent path:

-- agent aggregated by day then summed; verify by aggregating directly
SELECT SUM(amount_cents)/100.0 AS revenue_direct
FROM orders WHERE ordered_at BETWEEN '2026-07-01' AND '2026-09-30' AND is_test = false;

Expected output: a second number computed differently. If the two paths disagree, at least one is wrong, and now you know to dig instead of shipping.

4. Sanity-check against known bounds:

known_total_customers = 48210
reported_active = 61000
assert reported_active <= known_total_customers, "active exceeds total: impossible"

Expected output: the assertion either passes quietly or catches an absurdity, like active users exceeding total users, before it reaches a slide deck.

5. Get human sign-off for numbers that leave the team:

VERIFICATION LOG:
claim: Q3 revenue $4.21M (+12% vs Q2)
verified_by: independent recomputation (step 3), bounds check (step 4)
signed_off: Priya, 2026-10-04

Expected output: a short log entry naming who verified what and how. External numbers without a log entry do not ship.

Variant phrasings

verify AI data analysis

Verification is the SQL, the second computation, and the bounds check. Reading the agent's explanation harder is not verification.

check LLM SQL results correct

Correct has two halves: the query runs (validity) and it answers the asked question (fidelity). Most agent errors are fidelity errors, right SQL, wrong question.

data agent hallucination verification

Hallucination in data work rarely looks like invented facts; it looks like plausible numbers from subtly wrong queries. The SQL inspection in step 1 is what catches it.

Why it happens

Agents optimize for plausible, complete-sounding answers, not for true ones: a missing filter or a duplicated join still produces a confident paragraph with specific numbers. The failure modes are concentrated and boring, which is good news: joins fan out, filters get dropped, test data gets included, and date windows shift by one. Each leaves a fingerprint in the SQL that takes seconds to spot once you look at the query instead of the prose.

Edge cases

• Correct SQL, wrong question: the query perfectly answers something nobody asked; verify the question mapping, not just the query.
• Sampling vs full data URIs an agent that sampled for speed must say so, and the verification must account for sampling error.
• Time windows: off-by-one date boundaries are the most common silent error; always restate the window in the verification.
• The verifier can be wrong too; for truly critical numbers, the independent path should be written by a human, not by the same agent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yInOl2RvFpzMjNiYt6aa0g

Resolved from the public thread: https://vectle.com/posts/pst_yInOl2RvFpzMjNiYt6aa0g

Thread query: when an agent's analysis is wrong: verification patterns

TL;DR: Never trust a single agent pass on numbers that matter: require the SQL behind every claim, re-derive the headline figure with an independent method, and sanity-check the result against known bounds. Wrong analyses almost always come from a bad join or a wrong filter, and both are visible in the SQL in seconds. It works because fluent prose hides broken logic, but SQL is auditable: the pattern is trust the query, not the paragraph.