how to quarantine flaky tests in CI

TL;DR

Quarantine moves known-flaky tests out of the blocking path without deleting them: tag them, run them in a non-blocking job, track them on a dashboard, and give each a fix deadline.

Error

(Not an error; a process. The failure it prevents: flaky tests blocking every PR.)

Steps

1. Tag flaky tests: @flaky, pytest.mark.flaky, or a quarantine list file. Expected: the set is explicit and reviewable.
2. Exclude them from the blocking job and run them in a separate non-blocking job. Expected: merges proceed; flakes still run.
3. Dashboard them: count runs, failures, and age of each quarantined test. Expected: visibility, not a black hole.
4. Assign each an owner and a deadline (for example 2 weeks). Expected: accountability.
5. Delete or fix by the deadline; never let quarantine become permanent. Expected: the list shrinks.

When to use

• Flaky tests block PRs regularly.
• You need breathing room to fix them properly.

When not to use

• A test that always fails (that is broken, not flaky; fix it).
• As a permanent parking lot (see step 5).

Tool compatibility

• Any framework; CI job splitting; flaky-test plugins per framework.

Variant phrasings

Quarantine flaky tests

The general practice.

Flaky test triage process

The broader workflow; quarantine is one stage.

Why it happens

Flaky tests destroy trust in CI. Quarantine preserves the signal (tests still run) while removing the blockage (merges proceed).

Edge cases

• Quarantined tests that pass 100% for a month should graduate back.
• Review the quarantine list weekly; stale entries rot.
• Track who quarantined what; drive-by quarantines without ownership never get fixed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WHrrBhtlb78-QNKutADzzw