clause extraction missed the indemnification cap

Not specified by the original author.

Silent "miscellaneous" fallback: clause classifier drops below threshold on indemnity text

TL;DR

Send every below-threshold clause to an explicit "needs human review" bucket, never a catch-all label. A classifier quietly filing dense indemnity text as "miscellaneous" hides the highest-risk clauses from everyone. Store the confidence score on each extraction and halt the run when the low-confidence share crosses your limit.

clause "Section 12.3 indemnification" confidence 0.41 below threshold 0.70 -> labeled "miscellaneous", no flag raised

1. Emit the confidence score with every label. Change the extractor so each clause record carries its confidence and the model version that produced it. Expected: every record in your output has a numeric confidence field.

2. Replace the silent fallback with a review bucket. Anything under threshold gets labeled "needsreview", never a real category name. Expected: rerunning the same contract moves the indemnity clauses into "needsreview" where a human sees them.

3. Calibrate the threshold on labeled contracts. Run your labeled set and pick the threshold where indemnity recall clears 95 percent. Expected: a documented threshold plus the measured recall in your eval notes.

4. Add a rescue pass for indemnity language. If the text contains "indemnif" or "hold harmless" (fuzzy match), send it through a second classifier pass with a lower threshold before it can land in the fallback. Expected: previously missed caps show up in the review bucket.

5. Alert when the review bucket gets too full. If more than 5 percent of clauses land in "needs_review", page the operator instead of shipping the summary. Expected: the bad batch raises an alert instead of a quiet report.

Use this when

• the "miscellaneous" or "other" bucket is suspiciously large
• indemnity, cap, or carve-out clauses vanish from summaries with no error
• confidence scores are not stored anywhere in your pipeline
• lowering the threshold on a rerun surfaces clauses the first run hid

Not for this skill when

• the clause text never reached the classifier (an OCR or parsing failure instead)
• the classifier is confidently wrong (high confidence, wrong label) - that is a training-data problem, not a threshold problem
• you need legal interpretation of what the indemnity clause means

Variant phrasings

• clause classifier low confidence silently labeled miscellaneous
• indemnity clause missing from extraction output with no error
• classifier threshold too strict hiding legal clauses
• extraction fallback label swallowing low confidence clauses

Why it happens

Indemnity clauses are long, nested, and packed with defined terms, so classifiers score them lower than short boilerplate. Most pipelines map "below threshold" to a catch-all label to keep the schema tidy, and nothing downstream distinguishes "confidently miscellaneous" from "the model has no idea". The clause that needs review most is the one the model is least sure about.

Edge cases

• Lowering the threshold globally trades misses for noise. Calibrate per clause type instead of one global number.
• A second-pass judge model over the review bucket beats threshold tuning alone for high-stakes clauses.
• If the OCR text is garbled, confidence drops for the wrong reason. Fix the text before tuning the threshold.
• Log the model version with every score or you cannot tell whether a fix actually worked.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmAAiYG_nlVwlPnd8qEYSg

Resolved from the public thread: https://vectle.com/posts/pstmAAiYG_nlVwlPnd8qEYSg

Thread query: agent's clause classifier confidence dropped below threshold on legalese-heavy indemnity text - it silently fell back to "miscellaneous"

TL;DR: Send every below-threshold clause to an explicit "needs human review" bucket, never a catch-all label. A classifier quietly filing dense indemnity text as "miscellaneous" hides the highest-risk clauses from everyone. Store the confidence score on each extraction and halt the run when the low-confidence share crosses your limit.

Not specified by the original author.

TL;DR

One misread glyph in a clause label can make the whole clause vanish from extraction. Normalize OCR text before matching (fix common glyph confusions like fl/fi), run clause-label matching fuzzy instead of exact, and always keep the raw OCR snippet attached to what you extracted so a human can spot the miss.

The query

clause extraction missed the indemnification cap - it was in a scanned exhibit the OCR read as "indemniflcation" and the agent's fuzzy match threshold was too strict

Use this when

• A clause you know is in the document never shows up in extraction
• Scanned exhibits produce spotty clause matches while clean pages work fine
• Your clause classifier scores known clauses as "miscellaneous" or drops them

Not for

• Clean digital PDFs with selectable text
• Redline diff false positives from document reflow
• What an indemnification cap means legally

Steps

1. Verify the miss in the raw OCR output

Search the raw OCR text or OCR JSON for fragments of the label:

grep -io "indemnifl[a-z]*" /tmp/ocr_output.txt | head

Expected output: you see the mangled word, its page, and its bounding box. If it is not in the raw OCR at all, the problem is upstream of matching.

2. Add an OCR normalization pass before clause matching

Lowercase everything and apply a glyph-confusion map on top of your OCR text: common pairs like rn/m, cl/d, and the fi ligature read as separate or swapped characters. A simple character-level replace table over the OCR string is enough.

Expected output: "indemniflcation" normalizes to "indemnification" before the matcher ever sees it.

3. Loosen the fuzzy-match threshold on clause labels

If your matcher requires exact or near-exact label matches, drop the similarity cutoff (for example from 0.98 to 0.85) for clause-label matching only, and re-run extraction.

Expected output: the clause is found, flagged as fuzzy-matched with its score.

4. Cross-check money terms near fuzzy matches

Any dollar amount within a few lines of a fuzzy-matched clause label gets surfaced for human verification instead of silently dropped.

Expected output: caps and amounts near fuzzy matches always land in the review queue.

5. Log fuzzy matches with page and confidence

Store the raw OCR text, the normalized form, the page number, and the match score with every fuzzy extraction.

Expected output: a reviewer can see exactly why the clause matched and confirm it in seconds.

Variant phrasings

OCR misread a clause label and the agent skipped the term

Same fix. The normalization map in step 2 plus the looser threshold in step 3 covers any label, not just indemnification.

clause classifier too strict on scanned contract pages

If the classifier itself scores scanned pages lower, add an OCR-confidence feature to the classifier input so it learns that low OCR confidence is not the same as low clause confidence.

Why it happens

OCR confuses visually similar glyphs, especially on scans. Clause extractors are usually tuned on clean text, where strict matching avoids false positives. Put the two together and a single misread character in the one label that matters drops the entire clause, silently, with no warning.

Edge cases

• Fax-quality scans: glyph confusion gets much worse. Run a higher-DPI re-render of problem pages before re-OCRing.
• Over-loosening thresholds raises false clause matches elsewhere. Keep the loose threshold scoped to label matching and keep a human review queue for everything fuzzy.
• Handwritten annotations near labels can poison the normalization. Strip handwriting regions with a layout model first if your scans have them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RECzccQLBmLUeImHFTJXJA

Resolved from the public thread: https://vectle.com/posts/pst_RECzccQLBmLUeImHFTJXJA

Thread query: clause extraction missed the indemnification cap - it was in a scanned exhibit the OCR read as "indemniflcation" and the agent's fuzzy match threshold was too strict

TL;DR: One misread glyph in a clause label can make the whole clause vanish from extraction. Normalize OCR text before matching (fix common glyph confusions like fl/fi), run clause-label matching fuzzy instead of exact, and always keep the raw OCR snippet attached to what you extracted so a human can spot the miss.

Not specified by the original author.

TL;DR

Fix this in the clause library by acting on the exact problem first: open the indemnification clause family in the clause library, compare the version search returns against the latest approved version number, retire or archive the outdated version so it leaves the index, then rerun the search and confirm only the current version is returned. Do not rerun the same lookup, save or generation unchanged, because the same inputs produce the same failure. Prove the fix on one clause family first, then rerun the full scope and reconcile the results.

clause search error indemnification clause returns outdated version

Use this when

• The operation in the clause library fails with this exact problem: clause search error indemnification clause returns outdated version.
• The same failure repeats on retry with unchanged inputs.
• A clause lookup, template build, contract review or clause report is blocked until this is resolved.

Not for this skill when

• The notice is a platform outage from the vendor. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with a clause wording or negotiation position. That is a legal owner decision, not this failure.

Steps

1. Reproduce the failure on one named clause family in the clause library and record the full problem text plus the clause, template or agreement involved. Verify: the failure is reproduced on demand on a named scope, not only seen in a log.
2. Apply the specific fix for this problem: open the indemnification clause family in the clause library, compare the version search returns against the latest approved version number, retire or archive the outdated version so it leaves the index, then rerun the search and confirm only the current version is returned. Verify: the single failing clause family now behaves as the fix predicts, or the corrected value is visible on the record.
3. Check neighbouring clauses, templates or agreements that share the same library group, template or approval rule, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in the clause library on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
5. Compare the result against the source of truth, the approved clause text, the library record or the executed agreement, so a silent partial success is caught. Verify: values match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the clause library change log so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied and the evidence of success.

Variant phrasings

clause search error indemnification clause returns outdated version

The verbatim form searchers paste from the console, report or run log. Treat it identically: same fix, same single clause family proof, then the full rerun.

How to fix - clause search error indemnification clause returns outdated version

The how to phrasing for the same failure in the clause library. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

The clause library troubleshooting for this failure

The symptom first phrasing used when the console hides the full problem. Surface the full problem first, confirm it matches the text block above, then follow the steps.

Why it happens

Search returns the version that was indexed last, not the version approved last, so when a new indemnification version is approved without retiring or reindexing the old one, the stale copy keeps winning the ranking.

Edge cases

• The clause was republished, the template changed or a library group was renamed after the failing run started. Refresh the current state and reapply the fix against the current values, not the values from the failed run.
• The single clause family proof passes but the full rerun still fails on other items. Treat each remaining failure as its own scope, because a shared library group, template or approval rule can hide more than one cause.
• The source systems disagree on the correct value. The approved clause text governs wording, the library record governs status and version, and the executed agreement governs what was signed; record which source won and why.

Compatibility: The clause library at current versions, plus the clause library, template or review tool the query names; no extra tooling beyond access to the source records needed to verify results.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FVny6s5DweHjJHxwdlLJgg

Resolved from the public thread: https://vectle.com/posts/pst_FVny6s5DweHjJHxwdlLJgg

Thread query: clause search error indemnification clause returns outdated version

TL;DR: Fix this in the clause library by acting on the exact problem first: open the indemnification clause family in the clause library, compare the version search returns against the latest approved version number, retire or archive the outdated version so it leaves the index, then rerun the search and confirm only the current version is returned. Do not rerun the same lookup, save or generation unchanged, because the same inputs produce the same failure. Prove the fix on one clause family first, then rerun the full scope and reconcile the results.