## TL;DR

Document-level accuracy hides the truth: measure per field. Score precision and recall separately for header fields and line items, weight amounts and tax IDs highest, and track over time and per vendor. Report the numbers that drive decisions: line-item F1 determines matching success, not header accuracy.

## Steps

1. Define the field set and weights (amounts highest).
   Expected: A scoring rubric.
2. Score predictions against labels per field.
   Expected: Precision/recall per field.
3. Aggregate with weights; report line items separately.
   Expected: Decision-useful numbers.
4. Slice by vendor, layout, and scan quality.
   Expected: Where to improve.
5. Track trends release over release.
   Expected: Quality over time.

## When to use

- Extractor evaluation
- Quality monitoring
- Vendor comparisons

## When not to use

- Business KPIs (cost per invoice)
- Production alerting
- Single-invoice debugging

## Compatibility

Framework-agnostic.

## Variant phrasings

### invoice extraction accuracy metrics

### field-level F1 invoices

### measure OCR quality AP

## Root cause

A 98% header accuracy with 80% line-item accuracy still breaks matching. Field-level measurement shows where the value leaks.

## Edge cases

- Partial credit for close amounts needs defined rules
- Line-item alignment (which predicted row matches which label) needs care
- Human labelers disagree; measure inter-annotator agreement

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2BfCo3xLWZWBVkSwEO0VnA