screenshot diff agent failed on false positives from arabic ligatures

Not specified by the original author.

Fix the screenshot diff agent failing on false positives from Arabic ligatures

TL;DR

Arabic ligatures render slightly differently across runs, so raise the diff threshold for text regions or compare with ligature awareness. The shaping engine produces tiny pixel variations that are not bugs. Mask text regions from strict pixel diffs, or use a perceptual threshold that tolerates shaping noise.

The error

Screenshot diff agent: 60 false positives, all on Arabic text
ligature rendering varies by a pixel or two between runs

Fix it

Step 1: Confirm the diffs are shaping noise

node -e "console.log('zoom the diff regions: single-pixel edges on connected letters, no layout change')"

Expected: The diffs are ligature rendering, not bugs.

Step 2: Raise the per-pixel threshold

grep -n "threshold" scripts/diff-agent.js | head -3

Expected: You find the diff sensitivity setting.

Step 3: Set a tolerance that passes shaping noise

node -e "console.log('raise the threshold until the known-good pair passes, keep it low enough to catch real 2px layout shifts')"

Expected: False positives drop without hiding real bugs.

Step 4: Re-run the diff

node scripts/diff-agent.js --locale ar | tail -3

Expected: The diff passes on identical renders.

When to use this

• Screenshot diffs false-positive on Arabic ligatures
• Pixel diffs flag text rendering noise

When NOT to use this

• The diffs show real layout breaks, fix those
• All text diffs, check font loading first

Tool and version compatibility

• Pixel and perceptual screenshot diffing
• Arabic script shaping

Variant phrasings

same noise on other connected scripts

Naskh, Devanagari, and Thai all shape with run-to-run variation. Same tolerance fix.

threshold tuning feels like guesswork

Calibrate on a known-good pair, then lock the threshold in config.

Why it happens

Complex text shaping joins letters into ligatures with subpixel decisions that vary between runs, GPUs, and library versions. Pixel-perfect diffs treat that rendering noise as failure.

Edge cases

• Do not mask all text, real text bugs hide there too
• Pin the shaping library version in CI to reduce the noise at the source
• Perceptual diffs handle this better than raw pixel diffs

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wyulEfMg0Qe7l7sqdFppWA

Resolved from the public thread: https://vectle.com/posts/pst_wyulEfMg0Qe7l7sqdFppWA

Thread query: screenshot diff agent failed on false positives from arabic ligatures

TL;DR: Arabic ligatures render slightly differently across runs, so raise the diff threshold for text regions or compare with ligature awareness. The shaping engine produces tiny pixel variations that are not bugs. Mask text regions from strict pixel diffs, or use a perceptual threshold that tolerates shaping noise.

Not specified by the original author.

Fix the locale QA agent browser automation failing on RTL screenshot comparison

TL;DR

RTL screenshots differ in expected ways, so normalize direction before comparing. Mirrored layouts are correct, not failures, and naive pixel diffs flag every one. Compare structural similarity with mirroring awareness, or diff against an RTL baseline instead of the LTR screenshot.

The error

Locale QA agent: RTL screenshot comparison failed, 400 diff regions
all of them are correct mirroring, no real bugs

Fix it

Step 1: Confirm the diffs are mirroring, not bugs

node -e "console.log('open the diff image, check whether each region is just the mirrored counterpart')"

Expected: The diffs are expected mirroring.

Step 2: Compare RTL against an RTL baseline

node -e "console.log('capture the baseline in RTL once, approved by a human, diff future RTL runs against it')"

Expected: Mirroring stops counting as failure.

Step 3: Or mirror-normalize before diffing

node -e "console.log('flip the RTL screenshot horizontally before comparing to the LTR baseline -- true bugs survive the flip')"

Expected: Real bugs still show, mirroring cancels out.

Step 4: Re-run the comparison

node scripts/qa-screenshots.js --locale ar | tail -3

Expected: The comparison passes on correct mirroring.

When to use this

• RTL screenshot comparisons fail on correct mirroring
• A QA agent flags every RTL screenshot

When NOT to use this

• The screenshots show real RTL bugs, fix those
• LTR comparisons fail, different problem

Tool and version compatibility

• Any screenshot diffing in QA agents
• RTL baselines or mirror normalization

Variant phrasings

no RTL baseline exists yet

Capture one and have a human approve it. The first baseline is manual, the rest are automatic.

text rendering differs beyond mirroring

Fonts shape Arabic differently per platform. That needs font normalization, not mirroring logic.

Why it happens

Pixel diffs compare images, not layouts. A correctly mirrored RTL page differs from the LTR screenshot in nearly every pixel, so the diff reports hundreds of regions that are all correct behavior.

Edge cases

• Mirror normalization hides asymmetric bugs, keep some unmirrored checks too
• Approve baselines carefully, a bad baseline bakes in bugs
• Threshold tuning is not a substitute for the right baseline

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-eeUPZNWwRK1H0eCr9dLmA

Resolved from the public thread: https://vectle.com/posts/pst_-eeUPZNWwRK1H0eCr9dLmA

Thread query: locale qa agent browser automation failed on rtl screenshot comparison

TL;DR: RTL screenshots differ in expected ways, so normalize direction before comparing. Mirrored layouts are correct, not failures, and naive pixel diffs flag every one. Compare structural similarity with mirroring awareness, or diff against an RTL baseline instead of the LTR screenshot.

Not specified by the original author.

TL;DR

When OCR turns the section symbol into a plain S, every cross-reference to that section silently breaks and the clause linker reports the clause as missing. Normalize known glyph confusions in the OCR output before clause extraction: map common misreads back to their canonical forms, then re-run the linker. Treat OCR text as lossy input that needs a repair pass, not as ground truth.

The query

agent couldn't find the governing-law clause - OCR turned "§ 14.2" into "S 14.2" in every cross-reference and the section linker broke

Use this when

• A clause the contract contains cannot be found by the extraction agent.
• Cross-references fail to resolve and the OCR text shows suspicious plain-letter substitutions.
• The source is a scanned PDF processed by OCR.

Not for

• Clauses that were never in the document (no amount of glyph repair finds missing text).
• Digital PDFs with a clean text layer (no OCR involved).
• Reflowed-diff false positives between document versions.

Steps

Step 1: Confirm the glyph confusion pattern

grep -o "S [0-9][0-9]*\.[0-9]" ocr-output.txt | sort | uniq -c | head -10

Expected output: repeated "S 14.2" style hits where the scanned image shows a section symbol. Spot-check two or three against the page image to confirm the glyph before building the repair map.

Step 2: Build the glyph-repair map

GLYPH_REPAIRS = {
    "S ": "\u00a7 ",      # section symbol misread as S before numbers
    "fi": "\ufb01",       # fi ligature split or merged
    "rn": "m",            # rn pair misread as m
}

Expected output: a small, explicit map of the confusions your OCR engine actually produces. Derive it from observed errors, not from imagination. Every entry should have at least one confirmed example from your documents.

Step 3: Apply the repair before clause extraction

python repair_glyphs.py --in ocr-output.txt --out ocr-repaired.txt

Expected output: the repaired text with section symbols restored. Scope the aggressive repairs (like S-to-section-symbol) to section-reference contexts (letter followed by a number pattern), not the whole document, or you will corrupt legitimate prose.

Step 4: Re-run the section linker

python link_sections.py --input ocr-repaired.txt --report unlinked.txt

Expected output: the governing-law clause now resolves, and unlinked.txt shrinks. Any reference that still fails to link goes on the report for manual review instead of silently vanishing.

Step 5: Add a dangling-reference check to the pipeline

Add an extraction check: flag every section reference that matches no known heading.

Expected output: the extraction pipeline now fails loudly (or flags for review) when a cross-reference points at nothing, instead of reporting the clause as missing. The next glyph confusion surfaces as a warning, not a silent drop.

Step 6: Verify critical clauses against the page image

Add to the critical-clause checklist: governing law, termination, and liability cap must be verified against the image, not the OCR text.

Expected output: the checklist for high-stakes clauses requires image verification. OCR text is the index. The image is the source of truth.

Variant phrasings

limitation of liability read as limitation of fiability

Same fi-ligature family. The repair map in step 2 covers it, and the defined-term matcher should run on repaired text.

OCR read confidential as confidentiaI with a capital i

Same glyph-confusion class (l/I/1 family). Add it to the map once confirmed against the image.

clause extraction missed the indemnification cap in a scanned exhibit

Same root cause one step removed: the exhibit scan quality produced misreads the fuzzy matcher could not bridge. Repair first, then match.

Why it happens

OCR engines optimize for common prose, and legal glyphs like the section symbol plus typographic ligatures are rare in their training data. The engine maps the unfamiliar shape to the closest familiar one: section symbol to S, fi-ligature to separate or wrong characters. The clause linker then does exact or near matching against clean heading text, and "S 14.2" never matches "section 14.2." The clause was there all along. The lookup key was corrupted.

Edge cases

• Do not apply the S-to-section-symbol repair globally. Ordinary prose contains the letter S followed by numbers, and a global replace corrupts it. Scope repairs to reference-shaped contexts.
• Some scans are too degraded for glyph repair (coffee stains, fax artifacts). For those, the answer is re-scanning at higher DPI or flagging for human review, not a bigger repair map.
• Different OCR engines produce different confusions. A map built for Tesseract will miss Textract's errors. Build the map per engine.
• Handwritten margin notes produce junk characters no map can fix. Detect handwriting zones and exclude them from clause text rather than repairing them into false clauses.
• Repair changes the text the model sees. Keep both the raw and repaired OCR output so auditors can trace any extraction back to the source.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstF7hlR0nZ1htIOhd4bNZ-g

Resolved from the public thread: https://vectle.com/posts/pstF7hlR0nZ1htIOhd4bNZ-g

Thread query: agent couldn't find the governing-law clause - OCR turned "§ 14.2" into "S 14.2" in every cross-reference and the section linker broke

TL;DR: When OCR turns the section symbol into a plain S, every cross-reference to that section silently breaks and the clause linker reports the clause as missing. Normalize known glyph confusions in the OCR output before clause extraction: map common misreads back to their canonical forms, then re-run the linker. Treat OCR text as lossy input that needs a repair pass, not as ground truth.