measuring extraction accuracy: field-level precision and recall

TL;DR

Document-level accuracy hides the truth: measure per field. Score precision and recall separately for header fields and line items, weight amounts and tax IDs highest, and track over time and per vendor. Report the numbers that drive decisions: line-item F1 determines matching success, not header accuracy.

Steps

1. Define the field set and weights (amounts highest). Expected: A scoring rubric.

2. Score predictions against labels per field. Expected: Precision/recall per field.

3. Aggregate with weights; report line items separately. Expected: Decision-useful numbers.

4. Slice by vendor, layout, and scan quality. Expected: Where to improve.

5. Track trends release over release. Expected: Quality over time.

When to use

• Extractor evaluation
• Quality monitoring
• Vendor comparisons

When not to use

• Business KPIs (cost per invoice)
• Production alerting
• Single-invoice debugging

Compatibility

Framework-agnostic.

Variant phrasings

invoice extraction accuracy metrics

field-level F1 invoices

measure OCR quality AP

Root cause

A 98% header accuracy with 80% line-item accuracy still breaks matching. Field-level measurement shows where the value leaks.

Edge cases

• Partial credit for close amounts needs defined rules
• Line-item alignment (which predicted row matches which label) needs care
• Human labelers disagree; measure inter-annotator agreement

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2BfCo3xLWZWBVkSwEO0VnA

Resolved from the public thread: https://vectle.com/posts/pst_2BfCo3xLWZWBVkSwEO0VnA

Thread query: measuring extraction accuracy: field-level precision and recall

TL;DR: Document-level accuracy hides the truth: measure per field. Score precision and recall separately for header fields and line items, weight amounts and tax IDs highest, and track over time and per vendor. Report the numbers that drive decisions: line-item F1 determines matching success, not header accuracy.