## TL;DR

One green run proves nothing about a flaky test. A fix is verified only after the test passes N consecutive runs (20 minimum, 50 for notorious flakes), in CI conditions, with no code changes in between. Anything less is a green run, not a fix.

## Error

```text
agent: "fixed the flaky test test_checkout_race"
evidence: 1 passing run, local machine, 14:32
CI next morning: test_checkout_race FAILED (same assertion, same timeout)
```

The agent declared victory on a single local pass; CI failed it the next day.

## Steps

1. Define the bar up front: the healed test must pass 20 consecutive runs, or 50 if it flaked more than 5 times last month. Expected: a number, agreed before the agent starts.
2. Run the repetitions in CI, not on the agent's laptop: same container, same parallelism, same seed behavior. Expected: conditions match where the flake actually bites.
3. Vary what you can between runs: run the test alone and inside the full suite, and shuffle order if your runner supports it. Expected: the fix holds under different scheduling, not just one lucky ordering.
4. Require zero code or test changes during the verification window. Expected: `git status` clean between run 1 and run N, so the streak is meaningful.
5. Record the streak (run ids, timestamps) in the healing ticket. Expected: anyone can re-check the evidence, and a future flake reopens with history.

## When to use

- An agent claims a flaky test is fixed.
- You are writing the acceptance criteria for a test-healing agent.
- A "fixed" flake failed again within a week.

## When not to use

- The test was never flaky, just broken (one deterministic fix plus a normal suite run is enough).
- You are measuring flakiness on purpose (deliberate repeated runs to find flakes; different goal, same tooling).
- The flake is in infrastructure (CI runner, network), not the test: fix the infra, the streak test still applies after.

## Tool compatibility

- Any coding agent plus pytest (repeat via pytest-repeat), Jest (repeated CI jobs), Playwright (`--repeat-each`), Go test (`-count`).
- Needs CI you can trigger repeatedly (a workflow_dispatch or a loop script).

## Variant phrasings

### Agent says "passes consistently now" with no numbers

Ask for the number. "Consistently" without a run count is a feeling, not evidence.

### Flaky test fixed by adding a sleep

A sleep that makes 3 runs pass is not a fix, it is a slower flake. Run the 20-run streak; timing flakes usually break it.

### Test passes alone but fails in the suite

Order-dependent flake. The verification must include full-suite runs, not just isolated repeats.

## Why it happens

Flaky tests fail probabilistically, so any single run is a coin flip the agent happened to win. Agents declare victory at the first green because their loop ends when tests pass. A fixed streak requirement moves the goalpost from "green once" to "green reliably", which is the actual definition of fixed.

## Edge cases

- A 20-run streak on a test that flakes 1 in 100 still passes 82% of the time by luck: scale N to the flake rate (rarer flakes need longer streaks).
- Streaks cost CI time: run them overnight or on a schedule, not blocking the PR.
- If the test flakes during verification, reset the counter to zero; partial streaks do not count.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_W70KkN-c5SYYxlDd2ll1EA