OCR ran with the English pack on a German-language amendment - every umlaut became two characters and the agent's...

TL;DR

The amendment is in German but OCR ran with the English pack, so every umlaut became two garbage characters and term matching found nothing. Detect the language per page first, then re-run OCR with the right pack (tesseract -l deu, or deu+eng for mixed pages). Validate by checking that umlauts come back as single characters.

The symptom

OCR ran with the English pack on a German-language amendment  -  every umlaut became two characters and the agent's term matcher found nothing

Steps

1. Detect the page language before OCR

Run language detection on any existing text layer, or OCR a sample with osd/langdetect. For this amendment the answer is German. Do this per page: exhibits can mix languages. Expected: Each page labeled with its dominant language (de, en, or mixed).

2. Re-run OCR with the matching language pack

Run tesseract page.png stdout -l deu for German pages, or -l deu+eng for mixed pages. Install the pack first if it is missing (tesseract-ocr-deu). Expected: Umlauts extract as single correct characters: a with diaeresis, o with diaeresis, u with diaeresis, and the sharp s.

3. Normalize the output encoding

Make sure the pipeline treats the OCR output as UTF-8 end to end. Double-encoded output (two characters where one umlaut should be) usually means a latin-1/UTF-8 mismatch downstream of correct OCR. Expected: No multi-character mojibake sequences where umlauts belong.

4. Re-run the term matcher

Run the term matching step again against the corrected text and confirm the expected German legal terms now hit. Expected: Term matcher finds the clauses it missed before.

When to use

• Umlauts, accents, or non-Latin characters come out as mojibake
• Term matching finds nothing on a document you know contains the terms
• The document language differs from the OCR default

When not to use

• The document is English and the mojibake comes from a bad scan (different fix)
• Characters are correct but the term list itself is wrong (fix the terms, not the OCR)
• The text layer already exists and is correct (skip OCR entirely)

Why it happens

OCR language packs carry the character models. The English pack has no umlaut models, so it approximates each umlaut as the closest ASCII-looking sequence, often two characters. Downstream term matching then compares clean German terms against mangled text and matches nothing. The OCR was confident and wrong.

Edge cases

• Truly mixed pages (German body, English exhibit labels): use deu+eng so neither language degrades.
• Swiss German sharp-s vs double-s: normalize variants before term matching.
• Fraktur or old typefaces: need the specialized historical packs, not standard deu.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ofuHvz4TVzUyXKSCbBOHMA