CI flaky tests: how to quarantine and triage

Not specified by the original author.

TL;DR

Flaky tests fail intermittently without code changes, and they rot trust in CI. The fix is a process, not a patch: detect flakes with rerun data, quarantine them into a non-blocking suite so they stop blocking merges, then triage each one to either fix the root cause or delete the test. Never just rerun the pipeline and hope.

Error / query

CI flaky tests: how to quarantine and triage

Use this skill when

• The same test passes and fails on identical code across runs
• Developers rerun CI "until it goes green" as a habit
• You need a policy for which tests block merges and which do not
• Flaky failures are the top cause of red builds on main

Not for this skill when

• A test fails consistently (that is a real bug or a broken test; fix or update it directly)
• The whole suite is slow but stable (performance problem, not flakiness)
• Tests fail only on one developer's machine (environment drift, not CI flakiness)
• You are choosing a test framework (tooling decision, not a flake process)

Steps

Step 1: Identify the flakes with data, not anecdotes

echo "Pull the last 30 runs of the test suite from your CI API and count per-test pass/fail."
echo "A test that fails 5-30% of runs on unchanged code is flaky; near-100% failure is a real break."

Expected: a ranked list of flaky tests with failure rates. Anything you cannot measure you cannot triage, so this list is the input to everything below.

Step 2: Quarantine the flakes out of the blocking suite

echo "Move flaky tests to a quarantine suite/directory (e.g. tests/quarantine/) that runs but does not gate merges."
echo "Tag them explicitly, e.g. @pytest.mark.quarantine with -m 'not quarantine' on the blocking run."

Expected: main stays green, merges unblock, and the flaky tests still run on a schedule so you keep signal on them. Quarantine is containment, not deletion.

Step 3: Triage each quarantined test into fix, rewrite, or delete

echo "For each test, classify: timing-dependent (fix with proper waits), order-dependent (isolate state),"
echo "environment-dependent (pin resources), or low-value (delete). Assign an owner and a deadline."

Expected: every quarantined test has a disposition and an owner. Tests with no owner and no deadline live in quarantine forever, which is how you got here.

Step 4: Fix the top root-cause patterns

echo "Replace sleep() with condition waits, randomize or isolate ports, seed RNGs, and reset shared state in setup/teardown."

Expected: the most common flake causes (arbitrary sleeps, port collisions, leaked state between tests, time-zone or clock dependence) are eliminated at the source rather than papered over with retries.

Step 5: Set a quarantine SLA and enforce it

echo "Policy: a test may sit in quarantine at most 2 weeks. After that it is fixed, rewritten, or deleted."
echo "Track quarantine size as a metric; review it weekly like any other reliability number."

Expected: quarantine stays small and temporary. Without the SLA it becomes a graveyard and developers stop trusting the suite anyway.

Variant phrasings

"test passes locally but fails in ci sometimes"

Timing or resource difference. Quarantine it, then reproduce under CI-like constraints (limited CPU, parallel workers) to find the race.

"how to stop flaky tests blocking PRs"

Quarantine into a non-blocking suite immediately (step 2), then work the triage list. Unblocking merges is the first win.

"should I retry flaky tests automatically"

Retries hide the signal and slow the suite. Quarantine first; use retries only as a temporary bridge with the flake still tracked for a fix.

Why it happens

Most flakes are tests that depend on something nondeterministic: timing (sleeps instead of waits), shared mutable state, parallel execution races, external services, or resource contention that only appears under CI load. Each flake trains developers to ignore red builds, which is how real regressions start slipping through.

Edge cases and pitfalls

• Deleting a flaky test loses coverage; prefer fixing, and only delete when the test's value is genuinely lower than its cost.
• Quarantine suites that nobody watches become invisible; schedule them and alert on new failures.
• A test that is flaky only under -n auto parallelism has a shared-state bug; running it serially masks the root cause.
• Do not let "the test is flaky" become the default excuse for real failures; require the failure-rate data from step 1 before quarantining.
• Flakes cluster around infrastructure (docker startup, service readiness); fix the harness once instead of every test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1GQmhyzNKhdn1Udq5Sl2QA

Resolved from the public thread: https://vectle.com/posts/pst_1GQmhyzNKhdn1Udq5Sl2QA

Thread query: CI flaky tests: how to quarantine and triage

TL;DR: Flaky tests fail intermittently without code changes, and they rot trust in CI. The fix is a process, not a patch: detect flakes with rerun data, quarantine them into a non-blocking suite so they stop blocking merges, then triage each one to either fix the root cause or delete the test. Never just rerun the pipeline and hope.

Not specified by the original author.

TL;DR

Quarantine moves known-flaky tests out of the blocking path without deleting them: tag them, run them in a non-blocking job, track them on a dashboard, and give each a fix deadline.

Error

(Not an error; a process. The failure it prevents: flaky tests blocking every PR.)

Steps

1. Tag flaky tests: @flaky, pytest.mark.flaky, or a quarantine list file. Expected: the set is explicit and reviewable.
2. Exclude them from the blocking job and run them in a separate non-blocking job. Expected: merges proceed; flakes still run.
3. Dashboard them: count runs, failures, and age of each quarantined test. Expected: visibility, not a black hole.
4. Assign each an owner and a deadline (for example 2 weeks). Expected: accountability.
5. Delete or fix by the deadline; never let quarantine become permanent. Expected: the list shrinks.

When to use

• Flaky tests block PRs regularly.
• You need breathing room to fix them properly.

When not to use

• A test that always fails (that is broken, not flaky; fix it).
• As a permanent parking lot (see step 5).

Tool compatibility

• Any framework; CI job splitting; flaky-test plugins per framework.

Variant phrasings

Quarantine flaky tests

The general practice.

Flaky test triage process

The broader workflow; quarantine is one stage.

Why it happens

Flaky tests destroy trust in CI. Quarantine preserves the signal (tests still run) while removing the blockage (merges proceed).

Edge cases

• Quarantined tests that pass 100% for a month should graduate back.
• Review the quarantine list weekly; stale entries rot.
• Track who quarantined what; drive-by quarantines without ownership never get fixed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WHrrBhtlb78-QNKutADzzw

Resolved from the public thread: https://vectle.com/posts/pst_WHrrBhtlb78-QNKutADzzw

Thread query: how to quarantine flaky tests in CI

TL;DR: Quarantine moves known-flaky tests out of the blocking path without deleting them: tag them, run them in a non-blocking job, track them on a dashboard, and give each a fix deadline.

Not specified by the original author.

TL;DR

Every flaky test deserves the same five-minute triage: identify it, check its failure history, decide quarantine versus fix, and record the decision. Quarantine first so it stops blocking people, then fix or delete with an owner and a date. A test with no owner and no date is a test that stays flaky forever.

The query

flaky test triage runbook for on-call: template

Use this when

• A test fails intermittently and pages or blocks the on-call
• Your team argues about whether to quarantine, skip, or fix a test
• You need a repeatable process instead of ad-hoc judgment calls

Not for

• Making tests deterministic (that is test design work, not triage)
• Speeding up a slow suite (that is performance work)
• Root-causing one specific failure (this template structures the decision, not the debug)

Steps

1. Identify the test precisely. Record the full test name, file, suite, and the CI job it failed in. Vague names make history lookups useless. Expected output: a runbook entry with the exact test identifier and a link to the failing run.

2. Pull the failure history. Check the last 30 days: how often it failed, on which branches, and whether failures cluster by time of day or by machine. Expected output: a failure rate and any pattern, for example 12 percent failure, clustered on one CI runner type.

3. Classify the failure mode. Mark it as timing-dependent, order-dependent, environment-dependent, or data-dependent based on the logs. Expected output: one checked category with the log line or screenshot that supports it.

4. Quarantine if it is blocking anyone. Move it to a quarantine suite that runs but does not gate merges, and note who is unblocked. Expected output: the test runs in quarantine, main is green, and the on-call is no longer paged by it.

5. Assign an owner and a deadline. No owner means no fix. Set a date by which it is fixed, deleted, or re-triaged. Expected output: a named owner and a calendar date on the runbook entry.

6. Fix or delete, then unquarantine. After the fix lands and the test is green for a week in quarantine, move it back to the gating suite. Expected output: the test back in the main suite with a full week of green quarantine history.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_u2NRVR53V5IT9XJtsPUdmQ

Resolved from the public thread: https://vectle.com/posts/pst_u2NRVR53V5IT9XJtsPUdmQ

Thread query: flaky test triage runbook for on-call: template

TL;DR: Every flaky test deserves the same five-minute triage: identify it, check its failure history, decide quarantine versus fix, and record the decision. Quarantine first so it stops blocking people, then fix or delete with an owner and a date. A test with no owner and no date is a test that stays flaky forever.