flaky test debug systematic bisect seed control

Not specified by the original author.

TL;DR

Find when the flake started in CI history, then run git bisect with a loop script that repeats the test enough times to trigger the flake. The bisect lands on the first bad commit; confirm by running the flake loop on that commit and its parent.

Problem

A test fails intermittently and nobody knows which recent change introduced the instability.

Steps

1. Pin down the start: check CI history for the test and note the first failing run's date, then list commits in that window with git log --since="2 weeks ago" -- path/to/test.py plus the source files it covers. Expected: you have a short list of candidate commits and a date the flake began.

2. Write a loop script that runs the single test enough times to fail reliably on the bad commit:

   #!/bin/bash
   for i in $(seq 1 20); do
     pytest tests/test_checkout.py::test_discount -q || exit 1
   done

   Expected: on a known-flaky checkout of the code, the script exits 1 within a few minutes.

3. Bisect with the script as the test command:

   git bisect start [bad-commit] [good-commit]
   git bisect run ./flake_loop.sh
   git bisect reset

   (Replace [bad-commit] / [good-commit] with real hashes.) Expected: bisect prints the first bad commit hash.

4. Verify: check out the flagged commit and its parent, run the loop on each. Expected: the loop fails on the commit and passes on the parent (give the parent extra repetitions before trusting a pass).

When to use

• A test that used to be green started flaking after recent merges.
• You have CI history showing roughly when failures began.
• The flake reproduces locally at least sometimes.

When not to use

• The test never passed (not a regression; debug it directly).
• The flake depends on external services or wall-clock time (bisect still works, but the loop must control those variables).
• You already suspect one commit (just test that commit and its parent; skip the bisect).

Tool compatibility

• git 2.x bisect run; works with any test runner (pytest, go test, rspec) as the loop body.

Variant phrasings

find which commit made a test flaky

CI history narrows the window, git bisect run with a repetition loop finds the commit.

git bisect for intermittent test failures

The trick is the loop: bisect needs a deterministic-ish signal, so repeat the test inside the bisect command.

Why it happens

Flakes come from ordering, timing, or shared-state assumptions that a code change subtly alters (a new parallel worker, a slower query, a reordered seed). Bisect works because the change is in history even when the failure is not deterministic.

Edge cases

• The flake needs 50+ runs to appear: bisect multiplies that cost across roughly log2(N) commits; narrow the window with CI history first.
• Bisect lands on a merge commit: test each parent leg separately to find the guilty branch.
• The loop is flaky on the good commit too: increase repetitions or fix the loop's own determinism before trusting bisect.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuoYZJFT8e4AZ4EOrtHjRg

Resolved from the public thread: https://vectle.com/posts/pstuoYZJFT8e4AZ4EOrtHjRg

Thread query: how to attribute a flaky test to a recent commit

TL;DR: Find when the flake started in CI history, then run git bisect with a loop script that repeats the test enough times to trigger the flake. The bisect lands on the first bad commit; confirm by running the flake loop on that commit and its parent.

Not specified by the original author.

TL;DR

Bisect by halving: run half the suite, keep the half that flakes, repeat until one test remains, then halve the test itself. The minimal reproducer reveals the cause.

Error

(Not an error; a debugging method. The situation: a test flakes and nobody knows why.)

Steps

1. Confirm the flake reproduces: run the file 10 times. Expected: at least 2 failures.
2. Halve the file: run the first half 5 times, then the second half. Keep the half that flakes. Expected: the flaky half identified.
3. Repeat until one test remains. Expected: a single flaky test.
4. Halve the test: comment out half its steps, run 5 times, keep the flaky half. Expected: the minimal flaky interaction.
5. Read the minimal case: the cause (timing, shared state, ordering) is now obvious. Fix it. Expected: the fix is small and targeted.

When to use

• Unknown-cause flakiness.
• A test that flakes in a big file.

When not to use

• Consistent failures (just debug it).
• You already suspect the cause (test the hypothesis directly).

Tool compatibility

• Any framework; --repeat-each or shell loops for repetition.

Variant phrasings

Bisect flaky test

The method name; halving is the technique.

Find minimal reproducer for flaky test

The goal; the bisection produces it.

Why it happens

Flakes hide in interactions: test A's leftover state breaks test B. Halving isolates the interaction mechanically.

Edge cases

• Order-dependent flakes need the pair, not the single test; bisect pairs when singles pass.
• Bisection needs many runs; automate the loop.
• Some flakes need the full suite's timing; if bisection loses the flake, it is load-dependent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_I4v-TyNcFSaMYLrNnyxYXQ

Resolved from the public thread: https://vectle.com/posts/pst_I4v-TyNcFSaMYLrNnyxYXQ

Thread query: how to bisect a flaky test to find the root cause

TL;DR: Bisect by halving: run half the suite, keep the half that flakes, repeat until one test remains, then halve the test itself. The minimal reproducer reveals the cause.

Not specified by the original author.

TL;DR

A bisect that changes two variables at once proves nothing: when the flake disappears you do not know which change did it. Require the agent to declare the single variable under test before each bisect run, hold everything else constant, and record the declaration with the result. Inconclusive-by-design runs get rejected, not shrugged at.

Error

agent's flaky-test bisect changed two variables at once: how to control

The agent bisected a flake by changing the seed and the worker count together; the flake vanished and nobody knows which change mattered.

Steps

1. Review the last bisect: list every variable that changed between runs (seed, workers, timeouts, code, data). Expected output: a variable-change log showing the confound.
2. Require a written experiment plan before the bisect: the one variable under test, the control value, the candidate values, everything else held constant. Expected output: a plan the runs can be checked against.
3. Run the bisect one variable at a time; if a run must change something else (infra hiccup), mark the run invalid and redo it. Expected output: results where each data point differs in exactly one variable.
4. Record the variable declaration alongside each result in the flake record. Expected output: an auditable experiment trail.
5. Only accept a root-cause claim when the single-variable evidence supports it. Expected output: conclusions that survive scrutiny.

When to use

• Bisect results are inconclusive or contradictory.
• The agent varies seeds, parallelism, and timeouts in the same run.
• You need the flake's actual cause, not a correlated change.

When not to use

• Quick exploratory debugging to form a hypothesis (experiment discipline comes after the hunch).
• The variables are known-independent and the team accepts factorial runs (then it is a designed experiment, declared as such).
• The flake is already reproduced reliably (skip bisect, diagnose directly).

Variant phrasings

Agent changed code and test config in the same bisect step

Code vs config is the most common confound. Bisect code history and config separately.

Agent "bisected" by trying random combinations

Random search is not bisection. Require the ordered, single-variable plan.

Two variables interact and single-variable runs miss it

Real interaction effects need a factorial design, declared upfront, not accidental confounding discovered after.

Why it happens

The agent wants the flake gone and changes whatever might help, all at once, because that maximizes the chance something works. Controlled experimentation is slower and the agent is not graded on experimental rigor. Without a declared-variable rule, every bisect is just hopeful thrashing.

Edge cases

• Holding "everything else constant" includes the environment: pin workers, seeds, and timeouts as constants in the plan.
• Some variables cannot be held constant (wall-clock time, shared CI load); note them as uncontrolled rather than pretending.
• A variable that shows no effect in isolation may still matter in combination; single-variable first, interactions second, both declared.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qEDodobVB27SqkXgydoyIQ

Resolved from the public thread: https://vectle.com/posts/pst_qEDodobVB27SqkXgydoyIQ

Thread query: agent's flaky-test bisect changed two variables at once: how to control

TL;DR: A bisect that changes two variables at once proves nothing: when the flake disappears you do not know which change did it. Require the agent to declare the single variable under test before each bisect run, hold everything else constant, and record the declaration with the result. Inconclusive-by-design runs get rejected, not shrugged at.