flaky test triage runbook for on-call: template

TL;DR

Every flaky test deserves the same five-minute triage: identify it, check its failure history, decide quarantine versus fix, and record the decision. Quarantine first so it stops blocking people, then fix or delete with an owner and a date. A test with no owner and no date is a test that stays flaky forever.

The query

flaky test triage runbook for on-call: template

Use this when

• A test fails intermittently and pages or blocks the on-call
• Your team argues about whether to quarantine, skip, or fix a test
• You need a repeatable process instead of ad-hoc judgment calls

Not for

• Making tests deterministic (that is test design work, not triage)
• Speeding up a slow suite (that is performance work)
• Root-causing one specific failure (this template structures the decision, not the debug)

Steps

1. Identify the test precisely. Record the full test name, file, suite, and the CI job it failed in. Vague names make history lookups useless.

Expected output: a runbook entry with the exact test identifier and a link to the failing run.

1. Pull the failure history. Check the last 30 days: how often it failed, on which branches, and whether failures cluster by time of day or by machine.

Expected output: a failure rate and any pattern, for example 12 percent failure, clustered on one CI runner type.

1. Classify the failure mode. Mark it as timing-dependent, order-dependent, environment-dependent, or data-dependent based on the logs.

Expected output: one checked category with the log line or screenshot that supports it.

1. Quarantine if it is blocking anyone. Move it to a quarantine suite that runs but does not gate merges, and note who is unblocked.

Expected output: the test runs in quarantine, main is green, and the on-call is no longer paged by it.

1. Assign an owner and a deadline. No owner means no fix. Set a date by which it is fixed, deleted, or re-triaged.

Expected output: a named owner and a calendar date on the runbook entry.

1. Fix or delete, then unquarantine. After the fix lands and the test is green for a week in quarantine, move it back to the gating suite.

Expected output: the test back in the main suite with a full week of green quarantine history.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_u2NRVR53V5IT9XJtsPUdmQ