## TL;DR
Score three things separately: SQL validity, does it run; result correctness, does it answer the question, checked against a golden set; and question understanding, did it query the right tables for the ask. One blended score hides where the agent actually fails. It works because different failure modes need different fixes: syntax errors need better generation, wrong-table errors need better context, and only separate rates tell you which one you have.

```text
how to measure a data agent's accuracy
```

## Use this when
- Comparing two models or two prompts objectively
- Setting a quality bar an agent must clear for rollout
- Diagnosing where an agent actually fails
- Reporting agent quality to stakeholders with real numbers

## Not for
- A single blended accuracy number, which hides everything useful
- Subjective analysis quality like narrative insight
- One-time measurement; accuracy is a trend, not a snapshot

## Steps

1. Build the eval set once and version it:

```yaml
# evals/agent_accuracy.yaml - 30 questions, real workload coverage
- id: a01
  question: "monthly revenue by category, Q3, excluding test orders"
  expected_tables: [analytics.orders, analytics.products]
  expected_sql: "SELECT ..."  # verified by a human
  expected_rows: 6
```
Expected output: a versioned file pairing each question with the tables it should touch, verified SQL, and the expected shape of the result.

2. Measure validity: does the agent's SQL execute:

```python
valid = 0
for case in eval_set:
    try:
        warehouse.execute(agent.sql_for(case.question))
        valid += 1
    except Exception:
        log_failure(case.id, "validity")
validity_rate = valid / len(eval_set)
```
Expected output: a rate like 0.93. Validity failures are syntax errors, unknown columns, and bad joins, the generation half of the problem.

3. Measure correctness: does the result match the expected answer:

```python
correct = 0
for case in eval_set:
    rows = warehouse.execute(agent.sql_for(case.question))
    if results_match(rows, case.expected_rows, tol=0.001):
        correct += 1
correctness_rate = correct / len(eval_set)
```
Expected output: a rate like 0.81. Correctness failures where validity passed are the interesting ones: the query ran fine and answered the wrong question.

4. Measure understanding: did it reach for the right tables:

```python
# judge: human or a separate LLM with a strict rubric
understanding = judge.did_use_right_tables(agent.trace, case.expected_tables)
```
Expected output: a rate like 0.88 with per-case notes. An agent that queries the wrong table but gets a plausible number is dangerous; this rate catches that.

5. Track all three as trends and gate releases on floors:

```sql
SELECT run_date, model_version,
       validity_rate, correctness_rate, understanding_rate
FROM ops.agent_accuracy ORDER BY run_date;
-- release rule: no rate may drop vs the current production version
```
Expected output: a time series per model version. A new version ships only when all three rates meet or beat production; a drop in any one blocks the rollout.

## Variant phrasings

### data agent accuracy metrics
The metrics are the three rates. Report them as a triple, never as an average.

### evaluate text to SQL correctness
Correctness is result equivalence with tolerance, not SQL text matching. Two different queries can both be right.

### LLM data agent benchmark
A benchmark is the eval set plus the three rates plus the trend. Without the trend it is a snapshot; without the set it is vibes.

## Why it happens
Teams that measure one accuracy number end up optimizing the wrong thing: they fix prompts for syntax when the real problem is table selection, or they celebrate 95% validity while correctness sits at 70%. Separating the rates maps each failure to its fix: validity failures mean better generation or schema context, understanding failures mean better retrieval or dictionary, correctness failures mean better verification. The trend matters because agent quality is not static; models update, prompts drift, and schemas evolve, and only a tracked series shows whether you are getting better or worse.

## Edge cases
- Partial credit: for gating, pass/fail is cleaner, but log near-misses separately to guide prompt work.
- Judge bias: if an LLM judges understanding, calibrate it against human judgments on a sample first.
- Eval set staleness: questions about last quarter's schema rot; refresh the set quarterly and version it with the schema.
- Small eval sets give noisy rates; 30 cases minimum before you trust a comparison between versions.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ww5Ats5O461xAJhtd_zjhQ