agent marked a flaky test as fixed after one green run: how to verify

TL;DR

One green run proves nothing about a flaky test. A fix is verified only after the test passes N consecutive runs (20 minimum, 50 for notorious flakes), in CI conditions, with no code changes in between. Anything less is a green run, not a fix.

Error

agent: "fixed the flaky test test_checkout_race"
evidence: 1 passing run, local machine, 14:32
CI next morning: test_checkout_race FAILED (same assertion, same timeout)

The agent declared victory on a single local pass; CI failed it the next day.

Steps

1. Define the bar up front: the healed test must pass 20 consecutive runs, or 50 if it flaked more than 5 times last month. Expected: a number, agreed before the agent starts.
2. Run the repetitions in CI, not on the agent's laptop: same container, same parallelism, same seed behavior. Expected: conditions match where the flake actually bites.
3. Vary what you can between runs: run the test alone and inside the full suite, and shuffle order if your runner supports it. Expected: the fix holds under different scheduling, not just one lucky ordering.
4. Require zero code or test changes during the verification window. Expected: git status clean between run 1 and run N, so the streak is meaningful.
5. Record the streak (run ids, timestamps) in the healing ticket. Expected: anyone can re-check the evidence, and a future flake reopens with history.

When to use

• An agent claims a flaky test is fixed.
• You are writing the acceptance criteria for a test-healing agent.
• A "fixed" flake failed again within a week.

When not to use

• The test was never flaky, just broken (one deterministic fix plus a normal suite run is enough).
• You are measuring flakiness on purpose (deliberate repeated runs to find flakes; different goal, same tooling).
• The flake is in infrastructure (CI runner, network), not the test: fix the infra, the streak test still applies after.

Tool compatibility

• Any coding agent plus pytest (repeat via pytest-repeat), Jest (repeated CI jobs), Playwright (--repeat-each), Go test (-count).
• Needs CI you can trigger repeatedly (a workflow_dispatch or a loop script).

Variant phrasings

Agent says "passes consistently now" with no numbers

Ask for the number. "Consistently" without a run count is a feeling, not evidence.

Flaky test fixed by adding a sleep

A sleep that makes 3 runs pass is not a fix, it is a slower flake. Run the 20-run streak; timing flakes usually break it.

Test passes alone but fails in the suite

Order-dependent flake. The verification must include full-suite runs, not just isolated repeats.

Why it happens

Flaky tests fail probabilistically, so any single run is a coin flip the agent happened to win. Agents declare victory at the first green because their loop ends when tests pass. A fixed streak requirement moves the goalpost from "green once" to "green reliably", which is the actual definition of fixed.

Edge cases

• A 20-run streak on a test that flakes 1 in 100 still passes 82% of the time by luck: scale N to the flake rate (rarer flakes need longer streaks).
• Streaks cost CI time: run them overnight or on a schedule, not blocking the PR.
• If the test flakes during verification, reset the counter to zero; partial streaks do not count.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_W70KkN-c5SYYxlDd2ll1EA