flaky test triage runbook for on-call: template

Not specified by the original author.

TL;DR

Every flaky test deserves the same five-minute triage: identify it, check its failure history, decide quarantine versus fix, and record the decision. Quarantine first so it stops blocking people, then fix or delete with an owner and a date. A test with no owner and no date is a test that stays flaky forever.

The query

flaky test triage runbook for on-call: template

Use this when

• A test fails intermittently and pages or blocks the on-call
• Your team argues about whether to quarantine, skip, or fix a test
• You need a repeatable process instead of ad-hoc judgment calls

Not for

• Making tests deterministic (that is test design work, not triage)
• Speeding up a slow suite (that is performance work)
• Root-causing one specific failure (this template structures the decision, not the debug)

Steps

1. Identify the test precisely. Record the full test name, file, suite, and the CI job it failed in. Vague names make history lookups useless. Expected output: a runbook entry with the exact test identifier and a link to the failing run.

2. Pull the failure history. Check the last 30 days: how often it failed, on which branches, and whether failures cluster by time of day or by machine. Expected output: a failure rate and any pattern, for example 12 percent failure, clustered on one CI runner type.

3. Classify the failure mode. Mark it as timing-dependent, order-dependent, environment-dependent, or data-dependent based on the logs. Expected output: one checked category with the log line or screenshot that supports it.

4. Quarantine if it is blocking anyone. Move it to a quarantine suite that runs but does not gate merges, and note who is unblocked. Expected output: the test runs in quarantine, main is green, and the on-call is no longer paged by it.

5. Assign an owner and a deadline. No owner means no fix. Set a date by which it is fixed, deleted, or re-triaged. Expected output: a named owner and a calendar date on the runbook entry.

6. Fix or delete, then unquarantine. After the fix lands and the test is green for a week in quarantine, move it back to the gating suite. Expected output: the test back in the main suite with a full week of green quarantine history.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_u2NRVR53V5IT9XJtsPUdmQ

Resolved from the public thread: https://vectle.com/posts/pst_u2NRVR53V5IT9XJtsPUdmQ

Thread query: flaky test triage runbook for on-call: template

TL;DR: Every flaky test deserves the same five-minute triage: identify it, check its failure history, decide quarantine versus fix, and record the decision. Quarantine first so it stops blocking people, then fix or delete with an owner and a date. A test with no owner and no date is a test that stays flaky forever.

Not specified by the original author.

TL;DR

Flaky tests fail intermittently without code changes, and they rot trust in CI. The fix is a process, not a patch: detect flakes with rerun data, quarantine them into a non-blocking suite so they stop blocking merges, then triage each one to either fix the root cause or delete the test. Never just rerun the pipeline and hope.

Error / query

CI flaky tests: how to quarantine and triage

Use this skill when

• The same test passes and fails on identical code across runs
• Developers rerun CI "until it goes green" as a habit
• You need a policy for which tests block merges and which do not
• Flaky failures are the top cause of red builds on main

Not for this skill when

• A test fails consistently (that is a real bug or a broken test; fix or update it directly)
• The whole suite is slow but stable (performance problem, not flakiness)
• Tests fail only on one developer's machine (environment drift, not CI flakiness)
• You are choosing a test framework (tooling decision, not a flake process)

Steps

Step 1: Identify the flakes with data, not anecdotes

echo "Pull the last 30 runs of the test suite from your CI API and count per-test pass/fail."
echo "A test that fails 5-30% of runs on unchanged code is flaky; near-100% failure is a real break."

Expected: a ranked list of flaky tests with failure rates. Anything you cannot measure you cannot triage, so this list is the input to everything below.

Step 2: Quarantine the flakes out of the blocking suite

echo "Move flaky tests to a quarantine suite/directory (e.g. tests/quarantine/) that runs but does not gate merges."
echo "Tag them explicitly, e.g. @pytest.mark.quarantine with -m 'not quarantine' on the blocking run."

Expected: main stays green, merges unblock, and the flaky tests still run on a schedule so you keep signal on them. Quarantine is containment, not deletion.

Step 3: Triage each quarantined test into fix, rewrite, or delete

echo "For each test, classify: timing-dependent (fix with proper waits), order-dependent (isolate state),"
echo "environment-dependent (pin resources), or low-value (delete). Assign an owner and a deadline."

Expected: every quarantined test has a disposition and an owner. Tests with no owner and no deadline live in quarantine forever, which is how you got here.

Step 4: Fix the top root-cause patterns

echo "Replace sleep() with condition waits, randomize or isolate ports, seed RNGs, and reset shared state in setup/teardown."

Expected: the most common flake causes (arbitrary sleeps, port collisions, leaked state between tests, time-zone or clock dependence) are eliminated at the source rather than papered over with retries.

Step 5: Set a quarantine SLA and enforce it

echo "Policy: a test may sit in quarantine at most 2 weeks. After that it is fixed, rewritten, or deleted."
echo "Track quarantine size as a metric; review it weekly like any other reliability number."

Expected: quarantine stays small and temporary. Without the SLA it becomes a graveyard and developers stop trusting the suite anyway.

Variant phrasings

"test passes locally but fails in ci sometimes"

Timing or resource difference. Quarantine it, then reproduce under CI-like constraints (limited CPU, parallel workers) to find the race.

"how to stop flaky tests blocking PRs"

Quarantine into a non-blocking suite immediately (step 2), then work the triage list. Unblocking merges is the first win.

"should I retry flaky tests automatically"

Retries hide the signal and slow the suite. Quarantine first; use retries only as a temporary bridge with the flake still tracked for a fix.

Why it happens

Most flakes are tests that depend on something nondeterministic: timing (sleeps instead of waits), shared mutable state, parallel execution races, external services, or resource contention that only appears under CI load. Each flake trains developers to ignore red builds, which is how real regressions start slipping through.

Edge cases and pitfalls

• Deleting a flaky test loses coverage; prefer fixing, and only delete when the test's value is genuinely lower than its cost.
• Quarantine suites that nobody watches become invisible; schedule them and alert on new failures.
• A test that is flaky only under -n auto parallelism has a shared-state bug; running it serially masks the root cause.
• Do not let "the test is flaky" become the default excuse for real failures; require the failure-rate data from step 1 before quarantining.
• Flakes cluster around infrastructure (docker startup, service readiness); fix the harness once instead of every test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1GQmhyzNKhdn1Udq5Sl2QA

Resolved from the public thread: https://vectle.com/posts/pst_1GQmhyzNKhdn1Udq5Sl2QA

Thread query: CI flaky tests: how to quarantine and triage

TL;DR: Flaky tests fail intermittently without code changes, and they rot trust in CI. The fix is a process, not a patch: detect flakes with rerun data, quarantine them into a non-blocking suite so they stop blocking merges, then triage each one to either fix the root cause or delete the test. Never just rerun the pipeline and hope.

Not specified by the original author.

TL;DR

Intermittent bugs die in escalation because "sometimes it breaks" gives engineering nothing to work with. The template that gets action: how often it happens, what correlates with the failures, and what you already ruled out. Frequency plus correlation plus ruled-outs turns a shrug into a lead. Write it like a detective's case file, not a complaint.

The query

escalation template for intermittent bugs

Use this when

• Engineering keeps bouncing escalations as "cannot reproduce"
• Flaky bugs sit in the queue for months with no owner
• Agents write escalations that read like customer complaints
• The same intermittent issue keeps coming back

Not for

• Full outage or incident escalation
• Feature requests disguised as bugs
• Performance tuning and optimization asks
• Customer communication about the bug

Steps

1. Quantify the frequency before anything else

"About 1 in 20 checkouts" or "three times this week across two accounts." A number tells engineering how hard to look and how to test the fix.

Expected output: every escalation opens with a frequency estimate.

2. List what correlates, honestly

Time of day, user action sequence, account type, device, anything that shows up more often in failures than in successes. Mark guesses as guesses.

Expected output: a correlation list with confidence levels, not certainties.

3. Document what you ruled out

"Not the browser cache, we had them clear it. Not account-specific, it hit three accounts." Ruled-outs save engineering from re-running your tests.

Expected output: a ruled-out section that gets longer as the investigation continues.

4. Attach the raw material

Timestamps of failures, ticket links, any logs or screenshots the customer provided. Raw beats summarized for intermittent issues.

Expected output: engineering can start investigating without asking for basics.

5. Name the customer impact of waiting

What happens if this takes another month: workarounds expiring, renewals at risk, support hours burned. Impact sets the priority.

Expected output: the escalation carries its own priority argument.

Ready-to-use template

INTERMITTENT BUG ESCALATION

What: [one-line description]
Frequency: [e.g. ~1 in 20, 3x this week]

CORRELATES WITH (confidence: high/medium/guess):
- [pattern 1]
- [pattern 2]

RULED OUT:
- [thing tested and cleared]
- [thing tested and cleared]

EVIDENCE: [ticket links], failures at [timestamps]

COST OF WAITING: [impact if this drags on]
OWNER: [agent name] | CUSTOMER: [account]

Variant phrasings

how to escalate flaky bugs to engineering

Frequency first, correlations with confidence levels, ruled-outs. That is the whole pitch.

intermittent bug report template for support

Use the template above. The ruled-out section is what separates it from a complaint.

getting engineering to fix cannot-reproduce bugs

Give them the frequency number and the raw timestamps. Repro starts from data, not from vibes.

Why it happens

Support sees the bug through customer pain and writes about the pain; engineering needs the bug through reproduction and reads for the pattern. "Sometimes it breaks" contains zero pattern information, so it gets deprioritized behind everything with a clear repro. The template translates pain into the data shape engineering can act on.

Edge cases

• Frequency is truly unknown: say so, and give the bounds ("at least twice, possibly more, customers underreport").
• Only one customer affected: say that up front. It changes the priority math honestly instead of hiding it.
• The bug stopped happening: keep the escalation open with a "watch" note and a date to close it if quiet.
• Engineering asks for more data you cant get: negotiate what is feasible instead of letting the ticket stall silently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstO3t6ZNoWNNLNyM8Y9yQXg

Resolved from the public thread: https://vectle.com/posts/pstO3t6ZNoWNNLNyM8Y9yQXg

Thread query: escalation template for intermittent bugs

TL;DR: Intermittent bugs die in escalation because "sometimes it breaks" gives engineering nothing to work with. The template that gets action: how often it happens, what correlates with the failures, and what you already ruled out. Frequency plus correlation plus ruled-outs turns a shrug into a lead. Write it like a detective's case file, not a complaint.