locale qa agent loop detected re-running failed screenshot suite

Fix the locale QA agent loop re-running a failed screenshot suite

TL;DR

The agent retries the whole screenshot suite on any failure, so quarantine the failing specs and continue. A single flaky screenshot retriggers everything, which looks like a loop and burns hours. Mark failed specs as quarantined after one retry, finish the suite, and report the quarantine list.

The error

Locale QA agent: screenshot suite run 1 failed, run 2 failed, run 3 failed...
same 2 specs fail each time, the suite never completes

Fix it

Step 1: Identify the consistently failing specs

grep -n "failed" logs/qa-run-*.log | sort | uniq -c | sort -rn | head -5

Expected: You see the 2 specs that always fail.

Step 2: Quarantine after one retry

node -e "console.log('pattern: spec fails, retry once, on second failure mark quarantined and continue the suite')"

Expected: Flaky specs stop restarting the suite.

Step 3: Re-run the full suite

node scripts/qa-screenshots.js | tail -3

Expected: The suite completes with a quarantine report.

Step 4: Fix or accept the quarantined specs

node -e "console.log('triage each quarantined spec: real bug, flaky baseline, or obsolete -- then unquarantine')"

Expected: The quarantine list shrinks deliberately.

When to use this

• A locale QA agent loops re-running a failed screenshot suite
• The same specs fail every run

When NOT to use this

• Different specs fail each run, that is systemic flakiness
• The suite passes on retry, raise the flake tolerance instead

Tool and version compatibility

• Screenshot QA suites with retry logic
• Quarantine patterns for flaky specs

Variant phrasings

quarantine list grows forever

Triage it weekly. Quarantine is a pause button, not a trash can.

failures are real bugs not flakes

Then the loop is correct behavior begging for a fix. Fix the bugs, the loop stops.

Why it happens

Suite-level retry treats any spec failure as a reason to redo everything. Two deterministically failing specs then guarantee infinite reruns, because the rerun fails the same way every time.

Edge cases

• One retry distinguishes flakes from deterministic failures, keep it
• Report quarantined specs loudly, silent quarantine hides regressions
• Time-box the whole suite too, quarantine handles specs not hangs

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_quueZXPbIPbEqIwNuAzgWg