screenshot diff agent failed on false positives from arabic ligatures

Fix the screenshot diff agent failing on false positives from Arabic ligatures

TL;DR

Arabic ligatures render slightly differently across runs, so raise the diff threshold for text regions or compare with ligature awareness. The shaping engine produces tiny pixel variations that are not bugs. Mask text regions from strict pixel diffs, or use a perceptual threshold that tolerates shaping noise.

The error

Screenshot diff agent: 60 false positives, all on Arabic text
ligature rendering varies by a pixel or two between runs

Fix it

Step 1: Confirm the diffs are shaping noise

node -e "console.log('zoom the diff regions: single-pixel edges on connected letters, no layout change')"

Expected: The diffs are ligature rendering, not bugs.

Step 2: Raise the per-pixel threshold

grep -n "threshold" scripts/diff-agent.js | head -3

Expected: You find the diff sensitivity setting.

Step 3: Set a tolerance that passes shaping noise

node -e "console.log('raise the threshold until the known-good pair passes, keep it low enough to catch real 2px layout shifts')"

Expected: False positives drop without hiding real bugs.

Step 4: Re-run the diff

node scripts/diff-agent.js --locale ar | tail -3

Expected: The diff passes on identical renders.

When to use this

• Screenshot diffs false-positive on Arabic ligatures
• Pixel diffs flag text rendering noise

When NOT to use this

• The diffs show real layout breaks, fix those
• All text diffs, check font loading first

Tool and version compatibility

• Pixel and perceptual screenshot diffing
• Arabic script shaping

Variant phrasings

same noise on other connected scripts

Naskh, Devanagari, and Thai all shape with run-to-run variation. Same tolerance fix.

threshold tuning feels like guesswork

Calibrate on a known-good pair, then lock the threshold in config.

Why it happens

Complex text shaping joins letters into ligatures with subpixel decisions that vary between runs, GPUs, and library versions. Pixel-perfect diffs treat that rendering noise as failure.

Edge cases

• Do not mask all text, real text bugs hide there too
• Pin the shaping library version in CI to reduce the noise at the source
• Perceptual diffs handle this better than raw pixel diffs

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wyulEfMg0Qe7l7sqdFppWA