flaky test heatmap: how to visualize failure patterns

TL;DR

A flaky test heatmap is a grid with tests on one axis and builds (or days) on the other, colored by outcome: green pass, red fail, yellow flake (passed on retry). Build it from your CI history: export per-test results from JUnit XML or your CI API, pivot into a test-by-build matrix, and render it. Clusters of yellow tell you exactly which tests to quarantine or fix first, which beats arguing from anecdotes.

The query

flaky test heatmap: how to visualize failure patterns

Use this when

• You need to prioritize which flaky tests to fix first.
• Leadership wants evidence that flakiness is concentrated, not random.

Not for

• Long-term reliability trends; use rolling metrics for that.
• A single failing test; debug it directly.

Steps

1. Export per-test results for the last 30 days from JUnit XML files or your CI API. Expected output: a table of test name, build, and outcome.
2. Pivot into a matrix: one row per test, one column per build, cells colored pass/fail/flake/skip. Expected output: a dense grid you can scan visually.
3. Sort rows by flake rate so the worst offenders sit at the top. Expected output: the top rows are your fix candidates.
4. Look for patterns: time-of-day clusters, same-file clusters, post-deploy spikes. Expected output: at least one actionable pattern, like "all API tests flake at midnight".
5. Share the heatmap and assign the top 5 rows to owners with a deadline. Expected output: named owners and dates, not a vague backlog item.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstaDLu7pf9FD1j8TpIGHLNg

Resolved from the public thread: https://vectle.com/posts/pstaDLu7pf9FD1j8TpIGHLNg

Thread query: flaky test heatmap: how to visualize failure patterns

TL;DR: A flaky test heatmap is a grid with tests on one axis and builds (or days) on the other, colored by outcome: green pass, red fail, yellow flake (passed on retry). Build it from your CI history: export per-test results from JUnit XML or your CI API, pivot into a test-by-build matrix, and render it. Clusters of yellow tell you exactly which tests to quarantine or fix first, which beats arguing from anecdotes.