how to detect flaky tests automatically

TL;DR

Detect flakes by running tests repeatedly and watching for inconsistent results: rerun failures automatically, track pass/fail history per test, and alert when a test's flakiness crosses a threshold.

Error

(Not an error; a detection system. The symptom it addresses: "I think that test is flaky" without evidence.)

Steps

1. Enable automatic reruns of failures in CI (Playwright retries, Cypress retries, pytest-rerunfailures). Expected: transient failures get a second chance.
2. Record per-test results over time in your CI analytics or a simple database. Expected: history per test.
3. Define flaky: fails then passes on retry, or fails intermittently across runs. Expected: a rule, not a feeling.
4. Alert the owning team when a test crosses the threshold. Expected: flakes get owners.
5. Review the flaky list weekly and quarantine or fix. Expected: the list is worked, not just watched.

When to use

• Setting up flake detection for the first time.
• You need data to justify fixing time.

When not to use

• You already know which tests flake (quarantine them).
• One-off investigation (run it 50 times locally instead).

Tool compatibility

• Playwright retries, Cypress retries, pytest-rerunfailures; any CI analytics.

Variant phrasings

Flaky test detection tools

The tooling question; reruns plus history is the core.

Identify flaky tests in CI

The goal; the method is rerun-and-track.

Why it happens

Flakiness is statistical. One run proves nothing; only history distinguishes flaky from broken.

Edge cases

• Reruns hide real flakes if overused; track rerun rates, not just final status.
• Detection needs enough runs; small repos need longer windows.
• Distinguish infra flakes (runner died) from test flakes (test raced).

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VS1pQ-8xcHDpLypa6-cCJQ