OCR confidence scores below threshold: what to do next

Not specified by the original author.

TL;DR

Low confidence is the system telling you it is unsure; the mistake is treating all extractions equally. Route by confidence bands: high goes straight through with arithmetic validation, medium gets targeted re-extraction of the low-confidence fields, and low goes to human review with the suspect fields highlighted. Track which fields go low-confidence most often to focus tuning.

Steps

1. Set per-field confidence thresholds, stricter for amounts and tax IDs. Expected: A threshold policy, not a single number.

2. Band each extraction: high, medium, low. Expected: A routing decision per invoice.

3. For medium band, re-extract only the low-confidence fields at higher DPI or with a second engine. Expected: Targeted recovery without full reprocessing.

4. For low band, present the invoice to a reviewer with suspect fields highlighted. Expected: Fast human correction.

5. Log confidence distributions per vendor and field. Expected: Data for where tuning pays off.

When to use

• Designing the human-in-the-loop flow
• Extractions with mixed field confidence
• Deciding review staffing

When not to use

• OCR engine accuracy tuning
• Post-review correction workflows
• Straight-through processing design

Compatibility

Needs per-field confidence from the extractor (Textract, Document AI, Azure DI provide it; Tesseract gives word-level).

Variant phrasings

low OCR confidence invoice

confidence threshold AP automation

when to send invoice to review

Root cause

Confidence scores estimate the extractor's uncertainty, but pipelines often ignore them and treat every extraction as equally reliable. The errors then surface downstream in the ERP, where they are far more expensive to fix.

Edge cases

• Confidence is not accuracy; calibrate thresholds against labeled data
• Some engines are overconfident on hallucinations; keep the arithmetic validation as a second gate
• Thresholds need periodic retuning as vendors and layouts change

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ampxO2y9RO54QsmhSb0fAg

Resolved from the public thread: https://vectle.com/posts/pst_ampxO2y9RO54QsmhSb0fAg

Thread query: OCR confidence scores below threshold: what to do next

TL;DR: Low confidence is the system telling you it is unsure; the mistake is treating all extractions equally. Route by confidence bands: high goes straight through with arithmetic validation, medium gets targeted re-extraction of the low-confidence fields, and low goes to human review with the suspect fields highlighted. Track which fields go low-confidence most often to focus tuning.

Not specified by the original author.

TL;DR

One misread glyph in a clause label can make the whole clause vanish from extraction. Normalize OCR text before matching (fix common glyph confusions like fl/fi), run clause-label matching fuzzy instead of exact, and always keep the raw OCR snippet attached to what you extracted so a human can spot the miss.

The query

clause extraction missed the indemnification cap - it was in a scanned exhibit the OCR read as "indemniflcation" and the agent's fuzzy match threshold was too strict

Use this when

• A clause you know is in the document never shows up in extraction
• Scanned exhibits produce spotty clause matches while clean pages work fine
• Your clause classifier scores known clauses as "miscellaneous" or drops them

Not for

• Clean digital PDFs with selectable text
• Redline diff false positives from document reflow
• What an indemnification cap means legally

Steps

1. Verify the miss in the raw OCR output

Search the raw OCR text or OCR JSON for fragments of the label:

grep -io "indemnifl[a-z]*" /tmp/ocr_output.txt | head

Expected output: you see the mangled word, its page, and its bounding box. If it is not in the raw OCR at all, the problem is upstream of matching.

2. Add an OCR normalization pass before clause matching

Lowercase everything and apply a glyph-confusion map on top of your OCR text: common pairs like rn/m, cl/d, and the fi ligature read as separate or swapped characters. A simple character-level replace table over the OCR string is enough.

Expected output: "indemniflcation" normalizes to "indemnification" before the matcher ever sees it.

3. Loosen the fuzzy-match threshold on clause labels

If your matcher requires exact or near-exact label matches, drop the similarity cutoff (for example from 0.98 to 0.85) for clause-label matching only, and re-run extraction.

Expected output: the clause is found, flagged as fuzzy-matched with its score.

4. Cross-check money terms near fuzzy matches

Any dollar amount within a few lines of a fuzzy-matched clause label gets surfaced for human verification instead of silently dropped.

Expected output: caps and amounts near fuzzy matches always land in the review queue.

5. Log fuzzy matches with page and confidence

Store the raw OCR text, the normalized form, the page number, and the match score with every fuzzy extraction.

Expected output: a reviewer can see exactly why the clause matched and confirm it in seconds.

Variant phrasings

OCR misread a clause label and the agent skipped the term

Same fix. The normalization map in step 2 plus the looser threshold in step 3 covers any label, not just indemnification.

clause classifier too strict on scanned contract pages

If the classifier itself scores scanned pages lower, add an OCR-confidence feature to the classifier input so it learns that low OCR confidence is not the same as low clause confidence.

Why it happens

OCR confuses visually similar glyphs, especially on scans. Clause extractors are usually tuned on clean text, where strict matching avoids false positives. Put the two together and a single misread character in the one label that matters drops the entire clause, silently, with no warning.

Edge cases

• Fax-quality scans: glyph confusion gets much worse. Run a higher-DPI re-render of problem pages before re-OCRing.
• Over-loosening thresholds raises false clause matches elsewhere. Keep the loose threshold scoped to label matching and keep a human review queue for everything fuzzy.
• Handwritten annotations near labels can poison the normalization. Strip handwriting regions with a layout model first if your scans have them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RECzccQLBmLUeImHFTJXJA

Resolved from the public thread: https://vectle.com/posts/pst_RECzccQLBmLUeImHFTJXJA

Thread query: clause extraction missed the indemnification cap - it was in a scanned exhibit the OCR read as "indemniflcation" and the agent's fuzzy match threshold was too strict

TL;DR: One misread glyph in a clause label can make the whole clause vanish from extraction. Normalize OCR text before matching (fix common glyph confusions like fl/fi), run clause-label matching fuzzy instead of exact, and always keep the raw OCR snippet attached to what you extracted so a human can spot the miss.

Not specified by the original author.

TL;DR

Eight thousand items clustered at 0.49 means the scoring and the threshold disagree systematically, not that 8,000 coincidences occurred. Sample the band and read actual pairs: if they are true matches losing points on one consistent feature, such as a truncated memo or a fee-adjusted amount, fix the scoring feature or add a rule for that pattern. If they are genuinely ambiguous, they need a human queue with the evidence presented, not a threshold cut that would also admit the false matches sitting at 0.48.

agent stuck on 8,000 transactions at match confidence 0.49 vs the 0.50 threshold

Use this when

• A large set of candidate pairs scores just below the acceptance threshold, clustered around 0.49 versus 0.50.
• The unmatched backlog is dominated by this near-miss band.
• The pairs share a visible pattern in vendor, amount shape, or memo format.

Not for this skill when

• Only a handful of pairs sit near the threshold amid a healthy match rate. Review those individually; no systemic change is warranted.
• The scores are uncalibrated guesses from a model prompt rather than a defined scoring function. Rebuild scoring on defined features before threshold debates.

Steps

1. Sample thirty pairs from the 0.45 to 0.50 band and judge each by hand as true match, false match, or unsure, noting the deciding features. Verify: the sample verdicts and their reasons are recorded
2. Identify which scoring feature consistently penalises the true matches in the sample, such as date offset, name variation, or amount netting. Verify: one or two features explain most lost points
3. Choose the remedy per evidence: fix the feature calculation, add a deterministic rule for the proven pattern, or adjust the threshold only if the sample shows the band is overwhelmingly true matches. Verify: the remedy follows the sample, not the backlog pressure
4. Apply the remedy to the band and measure precision on a fresh sample of newly accepted pairs. Verify: newly accepted pairs sample as true matches at an acceptable rate
5. Route the still-unmatched residue to a human queue with pair evidence attached. Verify: the 8,000 backlog resolves into matched, rule-matched, and human-queued sets with counts

Variant phrasings

match confidence 0.49 threshold 0.50 stuck transactions

The near-threshold phrasing.

borderline match scores just below threshold sample review

When sampling the band.

Compatibility: Agent fuzzy matching with scored candidates in reconciliation and AP matching systems.

Why it happens

Thresholds turn continuous evidence into binary decisions, and any systematic scoring bias piles its victims just under the line. The pile's size is diagnostic: random ambiguity scatters, bias clusters. Reading the cluster's actual pairs separates a scoring defect, which is fixable precisely, from genuine ambiguity, which no threshold can resolve and which only looks solvable because lowering the line would make the backlog number prettier while quietly booking false matches.

Edge cases / pitfalls

• Track precision per band over time; a band whose precision decays is signalling data drift before accuracy visibly suffers.
• A two-tier design, auto-accept high, human-review middle, reject low, fits this reality better than one threshold. The band belongs in the middle tier.
• Never tune a threshold on the same sample used to judge it; hold out a fresh sample for the precision check.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Eopp55Sqxyt6xFAtnxz1pA

Resolved from the public thread: https://vectle.com/posts/pst_Eopp55Sqxyt6xFAtnxz1pA

Thread query: agent stuck on 8,000 transactions at match confidence 0.49 vs the 0.50 threshold

TL;DR: Eight thousand items clustered at 0.49 means the scoring and the threshold disagree systematically, not that 8,000 coincidences occurred. Sample the band and read actual pairs: if they are true matches losing points on one consistent feature, such as a truncated memo or a fee-adjusted amount, fix the scoring feature or add a rule for that pattern. If they are genuinely ambiguous, they need a human queue with the evidence presented, not a threshold cut that would also admit the false matches sitting at 0.48.