agent parsed both the PDF text layer and a fresh OCR pass - every clause appeared twice and the diff counted...

TL;DR

The agent extracted the embedded text layer AND ran OCR over the same pages, so every clause appeared twice and the diff counted the duplicates as changes. Pick one source of truth: check each page for a usable text layer first, and only OCR the pages that have none. Add a dedupe guard that drops exact-duplicate clauses.

The symptom

agent parsed both the PDF text layer and a fresh OCR pass  -  every clause appeared twice and the diff counted duplicates as changes

Steps

1. Test each page for a usable text layer

Extract text per page with pypdf. Pages returning more than a trivial character count (say, 50+ characters) have a real text layer; pages returning nothing are scanned images. Expected: Each page labeled text-layer or scan.

2. OCR only the scan pages

Run OCR exclusively on pages with no text layer. Never OCR a page that already gave you good text. Expected: Each page contributes exactly one text source.

3. Add a dedupe guard before diffing

Hash the normalized text of each extracted clause. If two clauses from the same page have identical hashes, keep the first and drop the second, and log the drop. Expected: No clause appears twice in the extraction output.

4. Re-run the diff and sanity-check the count

Diff the deduped extraction against the baseline. The change count should now match the counterparty's stated changes within a small margin. Expected: Duplicate-driven phantom changes disappear from the diff.

When to use

• Every clause appears twice in the extraction
• Diffs report changes on clauses nobody touched
• The pipeline runs both a text extractor and an OCR engine unconditionally

When not to use

• The document is a pure scan with no text layer (OCR everything, no duplication possible)
• Duplicates come from the source document itself (repeated exhibits, different fix)
• Only some clauses duplicate (look for a partial re-OCR of specific pages)

Why it happens

Many pipelines run text extraction and OCR as independent stages and concatenate the results. On a born-digital PDF both stages succeed on the same pages, so the output contains each clause twice. The diff then sees each duplicate pair as an addition, inflating the change count.

Edge cases

• Partial text layers (a scanned signature page in a digital doc): per-page detection handles this; whole-document detection does not.
• Text layer exists but is garbage (bad encoding): treat the page as a scan and OCR it, but still only once.
• Two-column text layers: dedupe on normalized text, not on raw block order.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nKecT5YpAyRPHXR7ElM8qQ

Resolved from the public thread: https://vectle.com/posts/pst_nKecT5YpAyRPHXR7ElM8qQ

Thread query: agent parsed both the PDF text layer and a fresh OCR pass - every clause appeared twice and the diff counted duplicates as changes

TL;DR: The agent extracted the embedded text layer AND ran OCR over the same pages, so every clause appeared twice and the diff counted the duplicates as changes. Pick one source of truth: check each page for a usable text layer first, and only OCR the pages that have none. Add a dedupe guard that drops exact-duplicate clauses.