footnote text got merged into the operative clause — OCR dropped the separator line and the agent quoted the footnote…

Not specified by the original author.

TL;DR

OCR dropped the footnote separator and fused the footnote into the clause. Re-extract with layout analysis that keeps footer regions separate, then run a post-pass that splits footnote markers (like bracketed numbers) into their own field. Never let text from the footnote zone be quoted as an operative term.

The symptom

footnote text got merged into the operative clause  -  OCR dropped the separator line and the agent quoted the footnote as a term

Steps

1. Confirm the footnote merge

Search the extracted clause text for footnote markers (bracketed numbers, asterisks) or sentences that read like asides. Compare against the rendered page: footnotes sit below a separator line in smaller type. Expected: Footnote sentences embedded inside the clause text where the separator used to be.

2. Re-extract with block-level layout analysis

Use tesseract with hOCR output or a layout-aware parser, then split the page into blocks by vertical position. Treat blocks in the bottom margin zone (below the separator line) as footnotes, not body. Expected: Footnotes land in a separate list, keyed by their marker, instead of inside clause text.

3. Add a footnote-marker post-pass

Scan extracted clause text for patterns like [1], (1), or * that reference footnotes, and cut the trailing footnote sentence out of the clause into the footnote field. Keep the marker reference so the link survives. Expected: Clause text ends where the footnote begins; the footnote is stored separately with its marker.

4. Guard the quoting step

Add a rule to the extraction agent: quoted terms must come from body-text blocks only. Any candidate quote containing a footnote marker pattern gets rejected and re-sourced. Expected: No quoted term contains footnote text; footnotes are cited as footnotes, not terms.

When to use

• Extracted clauses contain footnote sentences or footnote markers
• The page has visible footnotes below a separator line
• The agent quoted an aside or citation as if it were an operative term

When not to use

• Two columns are merging (different fix: column segmentation)
• Stamps or headers are injecting text (different fix: region masking)
• The document has no footnotes at all (look elsewhere)

Why it happens

Separator lines are thin and often vanish at scan resolution, and footnote type is only slightly smaller than body type. The OCR engine sees one continuous text flow, so the footnote becomes the clause's last sentence. The agent then treats it as operative language because nothing marks it as a footnote.

Edge cases

• Endnotes at the document end instead of page footnotes: same fix, different zone.
• Multiple footnote markers in one clause: split each and keep the marker links.
• Legal documents where a footnote genuinely modifies a term: keep the link visible so reviewers can check it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst5JWQOqjhpidJll1lfMyLw

Resolved from the public thread: https://vectle.com/posts/pst5JWQOqjhpidJll1lfMyLw

Thread query: footnote text got merged into the operative clause - OCR dropped the separator line and the agent quoted the footnote as a term

TL;DR: OCR dropped the footnote separator and fused the footnote into the clause. Re-extract with layout analysis that keeps footer regions separate, then run a post-pass that splits footnote markers (like bracketed numbers) into their own field. Never let text from the footnote zone be quoted as an operative term.

Not specified by the original author.

TL;DR

Footers and margin text get fused into clause text by OCR, and then the page-segmentation step discards the whole region as non-content. The fix is to segment the page image into header, body, and footer zones before OCR, extract each zone separately, and run clause extraction on the body while pulling notice addresses from the footer zone explicitly.

The query

termination notice address missed - OCR merged the footer text into the clause and the agent's page segmentation dropped it

Use this when

• Addresses, dates, or notice details vanish from extraction on scanned contracts.
• The missing content sits in a footer, header, or margin in the page image.
• Re-running OCR with layout preservation shows the text the pipeline dropped.

Not for

• Clauses missing from the source document itself.
• Digital PDFs with a clean text layer (no OCR or segmentation involved).
• Redline-diff false positives between versions.

Steps

Step 1: Confirm the address is in a dropped zone

tesseract page-14.png stdout --psm 6 | grep -i -A 2 "notice"

Expected output: the notice address appears in the raw OCR output when layout is preserved. If it appears here but not in the pipeline's extraction, the segmentation step dropped it, and the fix belongs in zoning, not in OCR quality.

Step 2: Add a zoning pass before text extraction

# split each page image into zones by vertical position
zones = {
    "header": page.crop(top=0.00, bottom=0.08),
    "body":   page.crop(top=0.08, bottom=0.92),
    "footer": page.crop(top=0.92, bottom=1.00),
}

Expected output: three separate images per page. The footer band that carried the notice address is now its own extraction input instead of noise fused into the clause above it.

Step 3: Extract zones separately and keep them labeled

for zone in header body footer; do
  tesseract page-14-$zone.png page-14-$zone.txt
 done

Expected output: three text files with zone labels. Footer text never flows into clause paragraphs again, because the zones never mix before extraction.

Step 4: Run clause extraction on the body, address extraction on the footer

python extract_clauses.py --input page-14-body.txt
python extract_notice_address.py --input page-14-footer.txt

Expected output: clauses come from the body zone, the notice address comes from the footer zone. Each extractor reads the zone where its content actually lives.

Step 5: Add a completeness check for notice details

Add an extraction check: a termination clause without a notice address is incomplete and must be flagged for review.

Expected output: the pipeline now treats a missing notice address as a failed extraction, not a successful one with less data. The next dropped footer surfaces as a flag, not a silent gap.

Step 6: Apply zoning per page across the whole document

python zone_all_pages.py contract.pdf --out zones/

Expected output: every page zoned the same way. Footers repeat on every page, so the zoning must run per page. Deduplicate footer content after extraction, not before, or the one page whose footer differs gets lost.

Variant phrasings

agent skipped the auto-renewal date, coffee stain on the scanned page

Same dropped-content class, different cause (image damage instead of zoning). Flag low-confidence regions for human review.

scanned schedule table read as one long column

Same layout-loss family: the table structure died in OCR. Extract tables with a table-aware tool before the text pipeline.

OCR merged the header letterhead into the first clause

Same zoning fix, top of the page instead of the bottom. The header zone exists for exactly this.

Why it happens

OCR is line-oriented, and page-segmentation models trained on articles and books treat footers as noise to discard. Legal footers are not noise: they carry notice addresses, execution dates, and page-level operative text. The pipeline fused two zones into one text stream, then a segmentation step trained on the wrong document genre threw away the part that mattered. The content was never missing. The pipeline's model of the page was wrong.

Edge cases

• Some contracts put the notice address in the body on the signature page and in the footer everywhere else. Extract from both zones and reconcile, preferring the signature-page instance.
• Zone boundaries by fixed percentage fail on pages with unusual layouts (full-page exhibits, landscape pages). Detect zones by content (repeated text across pages signals a footer) as a fallback.
• Do not deduplicate footer text before extraction. The one footer that differs (an amended notice address on the last page) is usually the operative one.
• Multi-column body layouts need column-aware zoning too, or the body extraction interleaves columns the same way the footer fused into the clause.
• Keep the zone-labeled OCR output alongside the final extraction. When an address is disputed, the auditor needs to see which zone it came from.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0mnRGPPKr1Hr4gBe7-HkA

Resolved from the public thread: https://vectle.com/posts/pst0mnRGPPKr1Hr4gBe7-HkA

Thread query: termination notice address missed - OCR merged the footer text into the clause and the agent's page segmentation dropped it

TL;DR: Footers and margin text get fused into clause text by OCR, and then the page-segmentation step discards the whole region as non-content. The fix is to segment the page image into header, body, and footer zones before OCR, extract each zone separately, and run clause extraction on the body while pulling notice addresses from the footer zone explicitly.

Not specified by the original author.

TL;DR A handwritten margin strike-through got OCR'd as characters, injected into the operative clause, and the agent quoted the junk verbatim. Separate the handwriting layer from the print layer before OCR, mask or inpaint the margin marks, and never quote clause text that contains tokens no dictionary or classifier recognizes. Junk in means flagged for review, not quoted as fact.

What the OCR output looked like:

the Licensee shall pay the Fee ... [unreadable stroke characters] ... within thirty days

Steps

1. Find the damage. Search your OCR output for the quoted clause and look for character runs that do not form words. Command: grep -rn "Licensee shall" ocr_text/ | head Expected: the clause line contains a run of nonsense characters where the margin strike-through bled into the text column.

2. Add a handwriting separation pass before OCR. Run the page through a print-versus-handwriting segmentation step and OCR the print layer alone, keeping the handwriting layer as a separate annotation. Expected: the operative clause OCRs cleanly with no injected characters, and the margin mark is stored as an annotation, not body text.

3. If you do not have a segmentation model, mask the margin regions. Detect dark stroke regions in the margins and inpaint them before OCR, or simply exclude a margin band from the text column. Expected: re-OCR of the page produces the clause without the junk characters.

4. Add a junk-token guard on quoted clauses. Before the agent quotes any clause, check that every token is a real word or a known legal term; if junk tokens remain, flag the quote for review instead of emitting it. Expected: any remaining OCR junk triggers a review flag, never a verbatim quote presented as operative language.

Use this when

• Scanned contracts have handwritten strike-throughs, margin notes, or initials
• OCR output contains runs of nonsense characters inside otherwise clean clauses
• Your agent quotes clause text that includes obvious garbage
• Signature pages or initialed pages bleed marks into nearby text

Not for this skill when

• The handwriting is the operative content, like a handwritten amendment, which needs its own transcription pass, not masking
• The junk comes from a stain or fax artifact rather than handwriting, use image cleanup instead
• The document is born-digital, there is no handwriting layer to separate
• The quote is clean but the agent misattributes which clause it came from, that is a citation problem

Variant phrasings

• handwritten strike-through read as characters by OCR
• OCR injected junk into the contract clause text
• agent quoted garbage characters as operative language
• how to separate handwriting from print before OCR
• margin annotations polluting contract OCR output

Why it happens

OCR engines do not know the difference between printed contract text and a handwritten mark crossing it; they transcribe every dark pixel cluster they find. A strike-through drawn across a clause becomes a string of stroke fragments that the engine renders as characters, and those characters land inline in the clause text. The agent then has no reason to doubt them, so it quotes the corrupted clause verbatim. The root problem is treating the page as one text layer when it has two: the printed agreement and the human marks on top of it.

Edge cases

• A handwritten strike-through can be legally operative, like a negotiated deletion, so keep the annotation layer and surface it to the reviewer, do not just delete it silently.
• Initials in the margin are common and harmless; your segmentation should classify mark types, not nuke all handwriting.
• Inpainting can smear nearby print if the mask is too aggressive, keep the mask tight to the detected strokes and re-check the surrounding words.
• Cursive handwriting near printed text confuses some segmentation models, fall back to the margin-band exclusion in step 3 for those pages.
• If the clause is quoted from a page you already cleaned, keep the before and after images so a reviewer can verify the strike-through did not remove real text.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9RMQVlRKAw3R-LI3k69OrQ

Resolved from the public thread: https://vectle.com/posts/pst_9RMQVlRKAw3R-LI3k69OrQ

Thread query: handwritten margin strike-through read as characters - OCR injected junk into the operative clause and the agent quoted it verbatim

TL;DR: