how to bisect a flaky test to find the root cause

TL;DR

Bisect by halving: run half the suite, keep the half that flakes, repeat until one test remains, then halve the test itself. The minimal reproducer reveals the cause.

Error

(Not an error; a debugging method. The situation: a test flakes and nobody knows why.)

Steps

1. Confirm the flake reproduces: run the file 10 times. Expected: at least 2 failures.
2. Halve the file: run the first half 5 times, then the second half. Keep the half that flakes. Expected: the flaky half identified.
3. Repeat until one test remains. Expected: a single flaky test.
4. Halve the test: comment out half its steps, run 5 times, keep the flaky half. Expected: the minimal flaky interaction.
5. Read the minimal case: the cause (timing, shared state, ordering) is now obvious. Fix it. Expected: the fix is small and targeted.

When to use

• Unknown-cause flakiness.
• A test that flakes in a big file.

When not to use

• Consistent failures (just debug it).
• You already suspect the cause (test the hypothesis directly).

Tool compatibility

• Any framework; --repeat-each or shell loops for repetition.

Variant phrasings

Bisect flaky test

The method name; halving is the technique.

Find minimal reproducer for flaky test

The goal; the bisection produces it.

Why it happens

Flakes hide in interactions: test A's leftover state breaks test B. Halving isolates the interaction mechanically.

Edge cases

• Order-dependent flakes need the pair, not the single test; bisect pairs when singles pass.
• Bisection needs many runs; automate the loop.
• Some flakes need the full suite's timing; if bisection loses the flake, it is load-dependent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_I4v-TyNcFSaMYLrNnyxYXQ