how to attribute a flaky test to a recent commit

TL;DR

Find when the flake started in CI history, then run git bisect with a loop script that repeats the test enough times to trigger the flake. The bisect lands on the first bad commit; confirm by running the flake loop on that commit and its parent.

Problem

A test fails intermittently and nobody knows which recent change introduced the instability.

Steps

1. Pin down the start: check CI history for the test and note the first failing run's date, then list commits in that window with git log --since="2 weeks ago" -- path/to/test.py plus the source files it covers.

Expected: you have a short list of candidate commits and a date the flake began.

1. Write a loop script that runs the single test enough times to fail reliably on the bad commit:

   #!/bin/bash
   for i in $(seq 1 20); do
     pytest tests/test_checkout.py::test_discount -q || exit 1
   done

Expected: on a known-flaky checkout of the code, the script exits 1 within a few minutes.

1. Bisect with the script as the test command:

   git bisect start [bad-commit] [good-commit]
   git bisect run ./flake_loop.sh
   git bisect reset

(Replace [bad-commit] / [good-commit] with real hashes.) Expected: bisect prints the first bad commit hash.

1. Verify: check out the flagged commit and its parent, run the loop on each.

Expected: the loop fails on the commit and passes on the parent (give the parent extra repetitions before trusting a pass).

When to use

• A test that used to be green started flaking after recent merges.
• You have CI history showing roughly when failures began.
• The flake reproduces locally at least sometimes.

When not to use

• The test never passed (not a regression; debug it directly).
• The flake depends on external services or wall-clock time (bisect still works, but the loop must control those variables).
• You already suspect one commit (just test that commit and its parent; skip the bisect).

Tool compatibility

• git 2.x bisect run; works with any test runner (pytest, go test, rspec) as the loop body.

Variant phrasings

find which commit made a test flaky

CI history narrows the window, git bisect run with a repetition loop finds the commit.

git bisect for intermittent test failures

The trick is the loop: bisect needs a deterministic-ish signal, so repeat the test inside the bisect command.

Why it happens

Flakes come from ordering, timing, or shared-state assumptions that a code change subtly alters (a new parallel worker, a slower query, a reordered seed). Bisect works because the change is in history even when the failure is not deterministic.

Edge cases

• The flake needs 50+ runs to appear: bisect multiplies that cost across roughly log2(N) commits; narrow the window with CI history first.
• Bisect lands on a merge commit: test each parent leg separately to find the guilty branch.
• The loop is flaky on the good commit too: increase repetitions or fix the loop's own determinism before trusting bisect.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuoYZJFT8e4AZ4EOrtHjRg