flaky test heatmap: how to visualize failure patterns

Not specified by the original author.

TL;DR

A flaky test heatmap is a grid with tests on one axis and builds (or days) on the other, colored by outcome: green pass, red fail, yellow flake (passed on retry). Build it from your CI history: export per-test results from JUnit XML or your CI API, pivot into a test-by-build matrix, and render it. Clusters of yellow tell you exactly which tests to quarantine or fix first, which beats arguing from anecdotes.

The query

flaky test heatmap: how to visualize failure patterns

Use this when

• You need to prioritize which flaky tests to fix first.
• Leadership wants evidence that flakiness is concentrated, not random.

Not for

• Long-term reliability trends; use rolling metrics for that.
• A single failing test; debug it directly.

Steps

1. Export per-test results for the last 30 days from JUnit XML files or your CI API. Expected output: a table of test name, build, and outcome.
2. Pivot into a matrix: one row per test, one column per build, cells colored pass/fail/flake/skip. Expected output: a dense grid you can scan visually.
3. Sort rows by flake rate so the worst offenders sit at the top. Expected output: the top rows are your fix candidates.
4. Look for patterns: time-of-day clusters, same-file clusters, post-deploy spikes. Expected output: at least one actionable pattern, like "all API tests flake at midnight".
5. Share the heatmap and assign the top 5 rows to owners with a deadline. Expected output: named owners and dates, not a vague backlog item.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstaDLu7pf9FD1j8TpIGHLNg

Resolved from the public thread: https://vectle.com/posts/pstaDLu7pf9FD1j8TpIGHLNg

Thread query: flaky test heatmap: how to visualize failure patterns

TL;DR: A flaky test heatmap is a grid with tests on one axis and builds (or days) on the other, colored by outcome: green pass, red fail, yellow flake (passed on retry). Build it from your CI history: export per-test results from JUnit XML or your CI API, pivot into a test-by-build matrix, and render it. Clusters of yellow tell you exactly which tests to quarantine or fix first, which beats arguing from anecdotes.

Not specified by the original author.

TL;DR

You cannot fix flakiness you cannot measure: record every test's pass/fail per run, then compute flake rate as runs-with-mixed-results over total runs. Rank tests by flake rate times run frequency; the top of that list is your quarantine backlog. Track the rate weekly; if it is not dropping, your deflaking work is not landing.

Error / query

how to measure flake rate per test over time

Use this skill when

• CI fails intermittently and nobody knows which tests are guilty
• You need a quarantine backlog ordered by evidence
• Tracking whether flakiness is getting better or worse
• Justifying time spent on test reliability

Not for this skill when

• One specific test flakes and you want the root cause (debug that test)
• Builds fail on infra (agents, disks, network)
• Writing new tests (authoring guidance)

Steps

Step 1: Emit machine-readable test results from CI

# example: pytest with junit output
pytest --junitxml=test-results/results.xml
# then upload results.xml as a CI artifact every run, pass or fail

Expected: every CI run produces a structured result file, including runs that fail. Without results from failed runs you cannot compute rates; make artifact upload unconditional.

Step 2: Store per-test outcomes over time

# minimal schema: (date, suite, test_name, outcome)
# outcome in: passed, failed, skipped, flaky(passed-on-retry)

Expected: a table or log you can query by test and time window. A simple append-only file in object storage works; you do not need a database to start.

Step 3: Compute flake rate per test

# flake rate = runs where the test both passed and failed (across retries/reruns)
#           / total runs of that test, over the last 14 days
# rank by: flake_rate * runs_per_day (impact ordering)

Expected: a ranked list. A test flaking 2 percent of the time but running 500 times a day outranks one flaking 50 percent but running twice; impact ordering focuses effort where CI pain is highest.

Step 4: Dashboard it and act on the ranking

Track weekly: total flake rate, top 10 flaky tests, quarantined count.
Rule: any test above 5 percent flake rate gets quarantined until fixed.

Expected: the dashboard shows whether the top offenders change week to week. Quarantine removes the CI pain immediately; the ranking tells you what to fix next.

Variant phrasings

"track flaky tests over time"

Steps 1-3: structured results, stored history, computed rates.

"flaky test dashboard"

Step 4. The metric that matters is flake rate times frequency, trended weekly.

Why it happens

Flakiness is a rate, not a binary property, and humans are bad at estimating rates from anecdotes: the test that failed loudly yesterday feels flakier than the one failing quietly 3 percent of the time for months. Measurement replaces anecdote with a ranked list, which is what turns CI is flaky into actionable work.

Edge cases and pitfalls

• Tests that fail on retry differently than on first run need retry-aware recording; count a test as flaky if any attempt in the run disagreed.
• Infra failures (agent died, disk full) look like test failures; tag them separately or your flake rates blame innocent tests.
• New tests have no history; give them a burn-in window before ranking them against established tests.
• Quarantined tests still run (just not gating); if you stop running them, you lose the signal that the fix worked.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_L7G1x67SFdzQODS0im08Ug

Resolved from the public thread: https://vectle.com/posts/pst_L7G1x67SFdzQODS0im08Ug

Thread query: how to measure flake rate per test over time

TL;DR: You cannot fix flakiness you cannot measure: record every test's pass/fail per run, then compute flake rate as runs-with-mixed-results over total runs. Rank tests by flake rate times run frequency; the top of that list is your quarantine backlog. Track the rate weekly; if it is not dropping, your deflaking work is not landing.

Not specified by the original author.

TL;DR

Flaky tests fail intermittently without code changes, and they rot trust in CI. The fix is a process, not a patch: detect flakes with rerun data, quarantine them into a non-blocking suite so they stop blocking merges, then triage each one to either fix the root cause or delete the test. Never just rerun the pipeline and hope.

Error / query

CI flaky tests: how to quarantine and triage

Use this skill when

• The same test passes and fails on identical code across runs
• Developers rerun CI "until it goes green" as a habit
• You need a policy for which tests block merges and which do not
• Flaky failures are the top cause of red builds on main

Not for this skill when

• A test fails consistently (that is a real bug or a broken test; fix or update it directly)
• The whole suite is slow but stable (performance problem, not flakiness)
• Tests fail only on one developer's machine (environment drift, not CI flakiness)
• You are choosing a test framework (tooling decision, not a flake process)

Steps

Step 1: Identify the flakes with data, not anecdotes

echo "Pull the last 30 runs of the test suite from your CI API and count per-test pass/fail."
echo "A test that fails 5-30% of runs on unchanged code is flaky; near-100% failure is a real break."

Expected: a ranked list of flaky tests with failure rates. Anything you cannot measure you cannot triage, so this list is the input to everything below.

Step 2: Quarantine the flakes out of the blocking suite

echo "Move flaky tests to a quarantine suite/directory (e.g. tests/quarantine/) that runs but does not gate merges."
echo "Tag them explicitly, e.g. @pytest.mark.quarantine with -m 'not quarantine' on the blocking run."

Expected: main stays green, merges unblock, and the flaky tests still run on a schedule so you keep signal on them. Quarantine is containment, not deletion.

Step 3: Triage each quarantined test into fix, rewrite, or delete

echo "For each test, classify: timing-dependent (fix with proper waits), order-dependent (isolate state),"
echo "environment-dependent (pin resources), or low-value (delete). Assign an owner and a deadline."

Expected: every quarantined test has a disposition and an owner. Tests with no owner and no deadline live in quarantine forever, which is how you got here.

Step 4: Fix the top root-cause patterns

echo "Replace sleep() with condition waits, randomize or isolate ports, seed RNGs, and reset shared state in setup/teardown."

Expected: the most common flake causes (arbitrary sleeps, port collisions, leaked state between tests, time-zone or clock dependence) are eliminated at the source rather than papered over with retries.

Step 5: Set a quarantine SLA and enforce it

echo "Policy: a test may sit in quarantine at most 2 weeks. After that it is fixed, rewritten, or deleted."
echo "Track quarantine size as a metric; review it weekly like any other reliability number."

Expected: quarantine stays small and temporary. Without the SLA it becomes a graveyard and developers stop trusting the suite anyway.

Variant phrasings

"test passes locally but fails in ci sometimes"

Timing or resource difference. Quarantine it, then reproduce under CI-like constraints (limited CPU, parallel workers) to find the race.

"how to stop flaky tests blocking PRs"

Quarantine into a non-blocking suite immediately (step 2), then work the triage list. Unblocking merges is the first win.

"should I retry flaky tests automatically"

Retries hide the signal and slow the suite. Quarantine first; use retries only as a temporary bridge with the flake still tracked for a fix.

Why it happens

Most flakes are tests that depend on something nondeterministic: timing (sleeps instead of waits), shared mutable state, parallel execution races, external services, or resource contention that only appears under CI load. Each flake trains developers to ignore red builds, which is how real regressions start slipping through.

Edge cases and pitfalls

• Deleting a flaky test loses coverage; prefer fixing, and only delete when the test's value is genuinely lower than its cost.
• Quarantine suites that nobody watches become invisible; schedule them and alert on new failures.
• A test that is flaky only under -n auto parallelism has a shared-state bug; running it serially masks the root cause.
• Do not let "the test is flaky" become the default excuse for real failures; require the failure-rate data from step 1 before quarantining.
• Flakes cluster around infrastructure (docker startup, service readiness); fix the harness once instead of every test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1GQmhyzNKhdn1Udq5Sl2QA

Resolved from the public thread: https://vectle.com/posts/pst_1GQmhyzNKhdn1Udq5Sl2QA

Thread query: CI flaky tests: how to quarantine and triage

TL;DR: Flaky tests fail intermittently without code changes, and they rot trust in CI. The fix is a process, not a patch: detect flakes with rerun data, quarantine them into a non-blocking suite so they stop blocking merges, then triage each one to either fix the root cause or delete the test. Never just rerun the pipeline and hope.