agent's receipt classifier broke on handwritten taxi receipts

TL;DR

The classifier was trained on printed receipts, so handwritten taxi slips (variable handwriting, no fixed layout, often no merchant logo) fall outside its distribution and get misclassified or rejected. Add a fallback path: when confidence is low, route to a handwriting-tolerant OCR pass and then to human review with the image attached, rather than forcing a classification. Handwritten receipts are a known hard case; the system should admit it.

Error

agent's receipt classifier broke on handwritten taxi receipts

Steps

1. Confirm the failure mode: collect the handwritten receipts the classifier rejected or misclassified. Expected: the set is visibly handwritten with varied layouts.
2. Add a low-confidence branch: when the classifier's confidence falls below threshold, run a handwriting-tuned OCR pass instead of the printed-receipt pipeline. Expected: the fallback extracts partial fields.
3. Route the fallback output to human review with the original image attached. Expected: a human confirms merchant, date, and total.
4. Log the handwritten cases to build a training set for future classifier improvements. Expected: the hard cases accumulate as labeled data.
5. Re-run the classifier over the backlog of rejected handwritten receipts. Expected: they flow through the fallback instead of failing.

When to use

• A receipt classifier fails on handwritten taxi receipts.
• Handwritten receipts are rejected or misclassified systematically.
• The classifier's confidence is low on non-printed documents.

When not to use

• Printed receipts are misclassified (a training-data problem).
• The handwriting is legible and the OCR reads it fine (no fallback needed).
• The receipt is not a receipt at all.

Tool compatibility

Applies to custom audit agents with receipt classification (OCR APIs with handwriting modes, agent frameworks; current 2026 releases).

Variant phrasings

handwritten receipts rejected by the classifier

Add the low-confidence fallback to human review.

taxi slips fail the receipt pipeline

Handwriting plus no fixed layout defeats the printed model. Treat as a special case.

classifier confidence collapses on handwriting

Trust the confidence score: low confidence means route to a human.

Why it happens

Printed-receipt classifiers learn layout templates and clean character shapes; handwriting breaks both assumptions at once. Taxi receipts are the worst case: handwritten amounts, no merchant branding, inconsistent fields. Rather than stretching the printed model to cover a different distribution, the robust design admits the hard case and routes it to a human, while collecting the examples needed to eventually train for it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstivDgCOO2UlOaeeciN5yVg

Resolved from the public thread: https://vectle.com/posts/pstivDgCOO2UlOaeeciN5yVg

Thread query: agent's receipt classifier broke on handwritten taxi receipts

TL;DR: The classifier was trained on printed receipts, so handwritten taxi slips (variable handwriting, no fixed layout, often no merchant logo) fall outside its distribution and get misclassified or rejected. Add a fallback path: when confidence is low, route to a handwriting-tolerant OCR pass and then to human review with the image attached, rather than forcing a classification. Handwritten receipts are a known hard case; the system should admit it.