pypdf extract_text returns "mojibake" on a contract with embedded CID fonts - how to fix

Fix pypdf extract_text returning mojibake on contracts with embedded CID fonts

TL;DR

The embedded CID font is missing a working ToUnicode map, so glyph IDs map to wrong characters and pypdf returns garbage; the page renders fine because rendering uses glyph shapes, not the character map. Try PyMuPDF, which handles more font edge cases; if it also returns garbage, the map is genuinely missing and you should rasterize the pages and OCR them.

extract_text() returned garbled text (mojibake) for embedded CID fonts

Steps

1. Extract the same page with PyMuPDF and compare the output to pypdf's. Expected: either clean text (PyMuPDF handled the font) or the same garbage (the ToUnicode map is genuinely missing).

2. Inspect the font's character map: dump the font info with qpdf --json and look for a ToUnicode entry. Expected: a missing or degenerate ToUnicode entry confirms the diagnosis.

3. If the map is missing, render each page to an image and run OCR over the images, then use the OCR text for the contract fields you need. Expected: accurate text at OCR quality; slower, but correct.

4. For one-off contracts, ask the counterparty for a text-based export (Word or clean PDF) instead of repairing the broken file. Expected: a clean source file with no repair work needed.

Use this when

• pypdf extract_text returns garbled characters on a contract that looks fine on screen
• The PDF uses embedded CID fonts (common in CJK text and some legal publishing tools)
• Text extraction works on some pages or fonts but not others

Not for this skill when

• extract_text returns empty strings (that's a scanned-image PDF; go straight to OCR)
• The PDF won't open at all (that's a corruption problem)
• Text is correct but layout is lost (that's a layout problem, not a font-map problem)

Variant phrasings

pypdf extract_text garbled characters

Usually a ToUnicode problem; the PyMuPDF comparison in step 1 settles it fast.

pdf text extraction mojibake python

Mojibake specifically (wrong but plausible characters) points at the font map, not encoding settings.

pdf cid font no tounicode extract text

Without ToUnicode, no parser can recover the text; OCR is the honest fallback.

Why it happens

Text extraction depends on the font's ToUnicode CMap, which maps internal glyph IDs to Unicode characters. Some PDF producers embed CID fonts with a broken or absent map; the glyphs still render because the viewer draws shapes directly. Every parser hits the same wall because the mapping information simply isn't in the file.

Edge cases

• Subset fonts may map only the glyphs actually used; rare characters can still garble.
• A document can mix good and bad fonts; check per-font, not per-document.
• OCR quality depends on render resolution; render at 300 DPI or higher for small legal print.
• Never silently ship garbled extraction; assert on expected clause keywords and fail loud.

Provenance

Resolved from the public thread: https://vectle.com/posts/psteAuigAjDE3MiJup9G6vKw