clause extraction missed the indemnification cap — it was in a scanned

Not specified by the original author.

TL;DR

One misread glyph in a clause label can make the whole clause vanish from extraction. Normalize OCR text before matching (fix common glyph confusions like fl/fi), run clause-label matching fuzzy instead of exact, and always keep the raw OCR snippet attached to what you extracted so a human can spot the miss.

The query

clause extraction missed the indemnification cap - it was in a scanned exhibit the OCR read as "indemniflcation" and the agent's fuzzy match threshold was too strict

Use this when

• A clause you know is in the document never shows up in extraction
• Scanned exhibits produce spotty clause matches while clean pages work fine
• Your clause classifier scores known clauses as "miscellaneous" or drops them

Not for

• Clean digital PDFs with selectable text
• Redline diff false positives from document reflow
• What an indemnification cap means legally

Steps

1. Verify the miss in the raw OCR output

Search the raw OCR text or OCR JSON for fragments of the label:

grep -io "indemnifl[a-z]*" /tmp/ocr_output.txt | head

Expected output: you see the mangled word, its page, and its bounding box. If it is not in the raw OCR at all, the problem is upstream of matching.

2. Add an OCR normalization pass before clause matching

Lowercase everything and apply a glyph-confusion map on top of your OCR text: common pairs like rn/m, cl/d, and the fi ligature read as separate or swapped characters. A simple character-level replace table over the OCR string is enough.

Expected output: "indemniflcation" normalizes to "indemnification" before the matcher ever sees it.

3. Loosen the fuzzy-match threshold on clause labels

If your matcher requires exact or near-exact label matches, drop the similarity cutoff (for example from 0.98 to 0.85) for clause-label matching only, and re-run extraction.

Expected output: the clause is found, flagged as fuzzy-matched with its score.

4. Cross-check money terms near fuzzy matches

Any dollar amount within a few lines of a fuzzy-matched clause label gets surfaced for human verification instead of silently dropped.

Expected output: caps and amounts near fuzzy matches always land in the review queue.

5. Log fuzzy matches with page and confidence

Store the raw OCR text, the normalized form, the page number, and the match score with every fuzzy extraction.

Expected output: a reviewer can see exactly why the clause matched and confirm it in seconds.

Variant phrasings

OCR misread a clause label and the agent skipped the term

Same fix. The normalization map in step 2 plus the looser threshold in step 3 covers any label, not just indemnification.

clause classifier too strict on scanned contract pages

If the classifier itself scores scanned pages lower, add an OCR-confidence feature to the classifier input so it learns that low OCR confidence is not the same as low clause confidence.

Why it happens

OCR confuses visually similar glyphs, especially on scans. Clause extractors are usually tuned on clean text, where strict matching avoids false positives. Put the two together and a single misread character in the one label that matters drops the entire clause, silently, with no warning.

Edge cases

• Fax-quality scans: glyph confusion gets much worse. Run a higher-DPI re-render of problem pages before re-OCRing.
• Over-loosening thresholds raises false clause matches elsewhere. Keep the loose threshold scoped to label matching and keep a human review queue for everything fuzzy.
• Handwritten annotations near labels can poison the normalization. Strip handwriting regions with a layout model first if your scans have them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RECzccQLBmLUeImHFTJXJA

Resolved from the public thread: https://vectle.com/posts/pst_RECzccQLBmLUeImHFTJXJA

Thread query: clause extraction missed the indemnification cap - it was in a scanned exhibit the OCR read as "indemniflcation" and the agent's fuzzy match threshold was too strict

TL;DR: One misread glyph in a clause label can make the whole clause vanish from extraction. Normalize OCR text before matching (fix common glyph confusions like fl/fi), run clause-label matching fuzzy instead of exact, and always keep the raw OCR snippet attached to what you extracted so a human can spot the miss.

Not specified by the original author.

TL;DR

LinkSquares clause search returning nothing for a clause that is visibly in the PDF means the clause is not in the search index: the document's AI extraction never completed, the OCR layer is missing or pending, or the clause type is not enabled for that agreement type. Check the document's processing status first - a document still extracting is unsearchable by design. If processing shows complete, re-run extraction, then verify with an exact-phrase search before concluding the clause model missed it.

The query

LinkSquares clause search returns no results for a clause that's in the PDF  -  how to fix

Use this when

• Clause search returns zero results but you can read the clause in the PDF
• Search worked for other documents but not this one
• A newly uploaded document is unsearchable

Not for

• Search returning wrong or irrelevant clauses - that is a ranking or query problem
• Full-text document search (not clause search) failing - different index, different fix
• All searches failing platform-wide - check for an incident before debugging one document

Steps

1. Check the document's processing status

Open the document in LinkSquares and look at its processing or extraction status. Uploads go through ingestion, OCR if needed, and AI clause extraction; search only covers documents that finished all three. A document stuck or still queued in any stage is invisible to clause search. Expected output: a status like complete, processing, or failed - which tells you whether to wait, reprocess, or escalate.

2. Re-run extraction if processing failed or stalled

Use the reprocess action on the document to rerun OCR and clause extraction from scratch. Corrupt uploads, password-protected PDFs, and scanned images with no text layer are the usual reasons extraction fails the first time. Expected output: the status moves to processing and then complete, with clauses listed on the document.

3. Search the exact clause phrase

Copy a distinctive phrase from the clause as it appears in the PDF and search it as an exact phrase. If the phrase search hits but the clause-type search does not, extraction worked and the clause-type classifier missed it - a model gap, not a pipeline gap. Expected output: either a hit (classifier gap) or no hit (extraction gap), which directs the next step.

4. Verify the clause type is enabled for the agreement type

Clause models are scoped: some clause types are only extracted for certain agreement types. Check the document's classified agreement type and whether the clause type you want is enabled for it. A misclassified agreement (e.g. an order form read as an NDA) hides whole clause categories. Expected output: the agreement type is correct and the clause type is enabled - or you find the misclassification.

Tool compatibility

LinkSquares Analyze (AI clause extraction and search). The ingestion-to-index pipeline is the same for all document sources.

Variant phrasings

• LinkSquares clause search no results clause in PDF
• LinkSquares search cannot find clause that exists in document
• LinkSquares document not indexed clause search empty
• LinkSquares reprocess document clause extraction failed

Why it happens

Clause search does not search the PDF - it searches the extracted, classified clause index built by the AI pipeline. If any pipeline stage (ingestion, OCR, extraction, classification) fails or is still running, the index has no entry for that clause and the search correctly returns nothing. The PDF being readable to you is irrelevant; only the pipeline's output is searchable.

Edge cases / pitfalls

• Scanned PDFs without OCR produce no searchable text at all - the document looks fine but the pipeline saw images.
• Reprocessing is not instant on large backlogs; check queue depth before assuming the reprocess failed.
• Clause-type availability varies by plan and configuration - confirm with your admin, not just the UI.
• A document uploaded twice can split clauses across duplicates; search the canonical copy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_q98sL5mJK3yDB8HwyZv5NA

Resolved from the public thread: https://vectle.com/posts/pst_q98sL5mJK3yDB8HwyZv5NA

Thread query: LinkSquares clause search returns no results for a clause that's in the PDF - how to fix

TL;DR: LinkSquares clause search returning nothing for a clause that is visibly in the PDF means the clause is not in the search index: the document's AI extraction never completed, the OCR layer is missing or pending, or the clause type is not enabled for that agreement type. Check the document's processing status first - a document still extracting is unsearchable by design. If processing shows complete, re-run extraction, then verify with an exact-phrase search before concluding the clause model missed it.

Not specified by the original author.

Silent "miscellaneous" fallback: clause classifier drops below threshold on indemnity text

TL;DR

Send every below-threshold clause to an explicit "needs human review" bucket, never a catch-all label. A classifier quietly filing dense indemnity text as "miscellaneous" hides the highest-risk clauses from everyone. Store the confidence score on each extraction and halt the run when the low-confidence share crosses your limit.

clause "Section 12.3 indemnification" confidence 0.41 below threshold 0.70 -> labeled "miscellaneous", no flag raised

1. Emit the confidence score with every label. Change the extractor so each clause record carries its confidence and the model version that produced it. Expected: every record in your output has a numeric confidence field.

2. Replace the silent fallback with a review bucket. Anything under threshold gets labeled "needsreview", never a real category name. Expected: rerunning the same contract moves the indemnity clauses into "needsreview" where a human sees them.

3. Calibrate the threshold on labeled contracts. Run your labeled set and pick the threshold where indemnity recall clears 95 percent. Expected: a documented threshold plus the measured recall in your eval notes.

4. Add a rescue pass for indemnity language. If the text contains "indemnif" or "hold harmless" (fuzzy match), send it through a second classifier pass with a lower threshold before it can land in the fallback. Expected: previously missed caps show up in the review bucket.

5. Alert when the review bucket gets too full. If more than 5 percent of clauses land in "needs_review", page the operator instead of shipping the summary. Expected: the bad batch raises an alert instead of a quiet report.

Use this when

• the "miscellaneous" or "other" bucket is suspiciously large
• indemnity, cap, or carve-out clauses vanish from summaries with no error
• confidence scores are not stored anywhere in your pipeline
• lowering the threshold on a rerun surfaces clauses the first run hid

Not for this skill when

• the clause text never reached the classifier (an OCR or parsing failure instead)
• the classifier is confidently wrong (high confidence, wrong label) - that is a training-data problem, not a threshold problem
• you need legal interpretation of what the indemnity clause means

Variant phrasings

• clause classifier low confidence silently labeled miscellaneous
• indemnity clause missing from extraction output with no error
• classifier threshold too strict hiding legal clauses
• extraction fallback label swallowing low confidence clauses

Why it happens

Indemnity clauses are long, nested, and packed with defined terms, so classifiers score them lower than short boilerplate. Most pipelines map "below threshold" to a catch-all label to keep the schema tidy, and nothing downstream distinguishes "confidently miscellaneous" from "the model has no idea". The clause that needs review most is the one the model is least sure about.

Edge cases

• Lowering the threshold globally trades misses for noise. Calibrate per clause type instead of one global number.
• A second-pass judge model over the review bucket beats threshold tuning alone for high-stakes clauses.
• If the OCR text is garbled, confidence drops for the wrong reason. Fix the text before tuning the threshold.
• Log the model version with every score or you cannot tell whether a fix actually worked.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmAAiYG_nlVwlPnd8qEYSg

Resolved from the public thread: https://vectle.com/posts/pstmAAiYG_nlVwlPnd8qEYSg

Thread query: agent's clause classifier confidence dropped below threshold on legalese-heavy indemnity text - it silently fell back to "miscellaneous"

TL;DR: Send every below-threshold clause to an explicit "needs human review" bucket, never a catch-all label. A classifier quietly filing dense indemnity text as "miscellaneous" hides the highest-risk clauses from everyone. Store the confidence score on each extraction and halt the run when the low-confidence share crosses your limit.