agent's flaky-test bisect changed two variables at once: how to control

TL;DR

A bisect that changes two variables at once proves nothing: when the flake disappears you do not know which change did it. Require the agent to declare the single variable under test before each bisect run, hold everything else constant, and record the declaration with the result. Inconclusive-by-design runs get rejected, not shrugged at.

Error

agent's flaky-test bisect changed two variables at once: how to control

The agent bisected a flake by changing the seed and the worker count together; the flake vanished and nobody knows which change mattered.

Steps

1. Review the last bisect: list every variable that changed between runs (seed, workers, timeouts, code, data). Expected output: a variable-change log showing the confound.
2. Require a written experiment plan before the bisect: the one variable under test, the control value, the candidate values, everything else held constant. Expected output: a plan the runs can be checked against.
3. Run the bisect one variable at a time; if a run must change something else (infra hiccup), mark the run invalid and redo it. Expected output: results where each data point differs in exactly one variable.
4. Record the variable declaration alongside each result in the flake record. Expected output: an auditable experiment trail.
5. Only accept a root-cause claim when the single-variable evidence supports it. Expected output: conclusions that survive scrutiny.

When to use

• Bisect results are inconclusive or contradictory.
• The agent varies seeds, parallelism, and timeouts in the same run.
• You need the flake's actual cause, not a correlated change.

When not to use

• Quick exploratory debugging to form a hypothesis (experiment discipline comes after the hunch).
• The variables are known-independent and the team accepts factorial runs (then it is a designed experiment, declared as such).
• The flake is already reproduced reliably (skip bisect, diagnose directly).

Variant phrasings

Agent changed code and test config in the same bisect step

Code vs config is the most common confound. Bisect code history and config separately.

Agent "bisected" by trying random combinations

Random search is not bisection. Require the ordered, single-variable plan.

Two variables interact and single-variable runs miss it

Real interaction effects need a factorial design, declared upfront, not accidental confounding discovered after.

Why it happens

The agent wants the flake gone and changes whatever might help, all at once, because that maximizes the chance something works. Controlled experimentation is slower and the agent is not graded on experimental rigor. Without a declared-variable rule, every bisect is just hopeful thrashing.

Edge cases

• Holding "everything else constant" includes the environment: pin workers, seeds, and timeouts as constants in the plan.
• Some variables cannot be held constant (wall-clock time, shared CI load); note them as uncontrolled rather than pretending.
• A variable that shows no effect in isolation may still matter in combination; single-variable first, interactions second, both declared.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qEDodobVB27SqkXgydoyIQ