handwritten annotations on invoices breaking extraction

TL;DR

Handwritten annotations (approved initials, corrected amounts, notes) sit on top of printed invoice text and corrupt the fields they touch. Treat handwriting as noise for extraction but signal for workflow: mask or segment handwritten regions, extract the printed text cleanly, and separately flag that a human annotated the invoice, since annotations often carry approval or correction intent.

Steps

1. Run handwriting detection/segmentation to mask handwritten regions.

Expected: A mask separating handwriting from print.

1. Extract printed fields from the unmasked regions.

Expected: Clean printed text.

1. Run handwriting recognition on the masked regions separately.

Expected: Transcribed annotations, kept apart from fields.

1. Flag the invoice as human-annotated and surface the annotation text to reviewers.

Expected: Approval or correction intent is not lost.

1. If an annotation overlaps a critical field like the total, route to human review.

Expected: No guessing where a human overwrote a value.

When to use

• Invoices with handwritten approvals, notes, or corrections
• Signatures overlapping printed text
• Printed totals with handwritten changes nearby

When not to use

• Fully handwritten invoices (different pipeline)
• Digital approval workflows
• Stamp/seal overlaps (printed, not handwritten)

Compatibility

AWS Textract (handwriting detection), Azure Document Intelligence, or dedicated HTR models. Printed-text OCR as usual.

Variant phrasings

handwriting on invoice breaks OCR

approved stamp handwritten extraction

signature overlapping invoice total

Root cause

Handwriting has irregular strokes that OCR reads as broken printed characters, corrupting adjacent fields. Meanwhile the annotation itself often carries workflow meaning (an approval initial), so deleting it loses information.

Edge cases

• A handwritten corrected total should override the printed one only after human confirmation
• Initials-only annotations carry no extractable text; the flag is the value
• Blue ink on white scans better than pencil; note quality in the flag

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Zva3MKJ33JpkKuPFxRUCcA