how to measure a data agent's accuracy

TL;DR

Score three things separately: SQL validity, does it run; result correctness, does it answer the question, checked against a golden set; and question understanding, did it query the right tables for the ask. One blended score hides where the agent actually fails. It works because different failure modes need different fixes: syntax errors need better generation, wrong-table errors need better context, and only separate rates tell you which one you have.

how to measure a data agent's accuracy

Use this when

• Comparing two models or two prompts objectively
• Setting a quality bar an agent must clear for rollout
• Diagnosing where an agent actually fails
• Reporting agent quality to stakeholders with real numbers

Not for

• A single blended accuracy number, which hides everything useful
• Subjective analysis quality like narrative insight
• One-time measurement; accuracy is a trend, not a snapshot

Steps

1. Build the eval set once and version it:

# evals/agent_accuracy.yaml - 30 questions, real workload coverage
- id: a01
  question: "monthly revenue by category, Q3, excluding test orders"
  expected_tables: [analytics.orders, analytics.products]
  expected_sql: "SELECT ..."  # verified by a human
  expected_rows: 6

Expected output: a versioned file pairing each question with the tables it should touch, verified SQL, and the expected shape of the result.

1. Measure validity: does the agent's SQL execute:

valid = 0
for case in eval_set:
    try:
        warehouse.execute(agent.sql_for(case.question))
        valid += 1
    except Exception:
        log_failure(case.id, "validity")
validity_rate = valid / len(eval_set)

Expected output: a rate like 0.93. Validity failures are syntax errors, unknown columns, and bad joins, the generation half of the problem.

1. Measure correctness: does the result match the expected answer:

correct = 0
for case in eval_set:
    rows = warehouse.execute(agent.sql_for(case.question))
    if results_match(rows, case.expected_rows, tol=0.001):
        correct += 1
correctness_rate = correct / len(eval_set)

Expected output: a rate like 0.81. Correctness failures where validity passed are the interesting ones: the query ran fine and answered the wrong question.

1. Measure understanding: did it reach for the right tables:

# judge: human or a separate LLM with a strict rubric
understanding = judge.did_use_right_tables(agent.trace, case.expected_tables)

Expected output: a rate like 0.88 with per-case notes. An agent that queries the wrong table but gets a plausible number is dangerous; this rate catches that.

1. Track all three as trends and gate releases on floors:

SELECT run_date, model_version,
       validity_rate, correctness_rate, understanding_rate
FROM ops.agent_accuracy ORDER BY run_date;
-- release rule: no rate may drop vs the current production version

Expected output: a time series per model version. A new version ships only when all three rates meet or beat production; a drop in any one blocks the rollout.

Variant phrasings

data agent accuracy metrics

The metrics are the three rates. Report them as a triple, never as an average.

evaluate text to SQL correctness

Correctness is result equivalence with tolerance, not SQL text matching. Two different queries can both be right.

LLM data agent benchmark

A benchmark is the eval set plus the three rates plus the trend. Without the trend it is a snapshot; without the set it is vibes.

Why it happens

Teams that measure one accuracy number end up optimizing the wrong thing: they fix prompts for syntax when the real problem is table selection, or they celebrate 95% validity while correctness sits at 70%. Separating the rates maps each failure to its fix: validity failures mean better generation or schema context, understanding failures mean better retrieval or dictionary, correctness failures mean better verification. The trend matters because agent quality is not static; models update, prompts drift, and schemas evolve, and only a tracked series shows whether you are getting better or worse.

Edge cases

• Partial credit: for gating, pass/fail is cleaner, but log near-misses separately to guide prompt work.
• Judge bias: if an LLM judges understanding, calibrate it against human judgments on a sample first.
• Eval set staleness: questions about last quarter's schema rot; refresh the set quarterly and version it with the schema.
• Small eval sets give noisy rates; 30 cases minimum before you trust a comparison between versions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstWw5Ats5O461xAJhtdzjhQ