how to retry flaky tests without hiding real failures

TL;DR

Retries are a diagnostic tool, not a fix. Retry failures a small fixed number of times, track what needed retries, and treat a test that always needs its retries as broken.

Error

(Not an error; a policy. The failure it prevents: retries hiding a growing flake problem.)

Steps

1. Set retries low: 1-2 in CI, 0 locally. Expected: flakes get a second chance; developers feel the pain.
2. Record which tests passed only on retry. Expected: the flaky set is visible.
3. Alert when a test needs retries consistently (for example 3 days running). Expected: chronic flakes get fixed.
4. Never raise retries to make a red suite green. Expected: the retry count is a policy, not a dial.
5. Pair retries with quarantine: chronic retry-users get quarantined. Expected: the suite stays honest.

When to use

• Configuring retries for the first time.
• Retries are masking a growing problem.

When not to use

• Deciding whether a specific test is flaky (detect first).
• Load or soak testing (different kind of repetition).

Tool compatibility

• Playwright retries, Cypress retries, pytest-rerunfailures, Jest --retry.

Variant phrasings

Test retry best practices

The policy question; low retries plus tracking.

Should I retry flaky tests

Yes, but as a diagnostic with tracking, not a fix.

Why it happens

Retries convert some flakes into passes, which is useful, but unmonitored retries convert signal into silence.

Edge cases

• Retries multiply CI time; budget them.
• Some frameworks retry the whole file, not the test; know which.
• Retry data is the input to quarantine decisions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstxy0Ee1sOk6usstfieO05Q