locale qa agent browser automation failed on rtl screenshot comparison

Fix the locale QA agent browser automation failing on RTL screenshot comparison

TL;DR

RTL screenshots differ in expected ways, so normalize direction before comparing. Mirrored layouts are correct, not failures, and naive pixel diffs flag every one. Compare structural similarity with mirroring awareness, or diff against an RTL baseline instead of the LTR screenshot.

The error

Locale QA agent: RTL screenshot comparison failed, 400 diff regions
all of them are correct mirroring, no real bugs

Fix it

Step 1: Confirm the diffs are mirroring, not bugs

node -e "console.log('open the diff image, check whether each region is just the mirrored counterpart')"

Expected: The diffs are expected mirroring.

Step 2: Compare RTL against an RTL baseline

node -e "console.log('capture the baseline in RTL once, approved by a human, diff future RTL runs against it')"

Expected: Mirroring stops counting as failure.

Step 3: Or mirror-normalize before diffing

node -e "console.log('flip the RTL screenshot horizontally before comparing to the LTR baseline -- true bugs survive the flip')"

Expected: Real bugs still show, mirroring cancels out.

Step 4: Re-run the comparison

node scripts/qa-screenshots.js --locale ar | tail -3

Expected: The comparison passes on correct mirroring.

When to use this

• RTL screenshot comparisons fail on correct mirroring
• A QA agent flags every RTL screenshot

When NOT to use this

• The screenshots show real RTL bugs, fix those
• LTR comparisons fail, different problem

Tool and version compatibility

• Any screenshot diffing in QA agents
• RTL baselines or mirror normalization

Variant phrasings

no RTL baseline exists yet

Capture one and have a human approve it. The first baseline is manual, the rest are automatic.

text rendering differs beyond mirroring

Fonts shape Arabic differently per platform. That needs font normalization, not mirroring logic.

Why it happens

Pixel diffs compare images, not layouts. A correctly mirrored RTL page differs from the LTR screenshot in nearly every pixel, so the diff reports hundreds of regions that are all correct behavior.

Edge cases

• Mirror normalization hides asymmetric bugs, keep some unmirrored checks too
• Approve baselines carefully, a bad baseline bakes in bugs
• Threshold tuning is not a substitute for the right baseline

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-eeUPZNWwRK1H0eCr9dLmA