agent marked a test flaky after one green rerun — what minimum evidence is needed before calling it a flake

Not specified by the original author.

TL;DR

One green rerun proves almost nothing. A trustworthy flake call needs at least 3 clean reruns on the same runner spec, same cache state, and a seed sweep if randomness is involved. Anything less is the agent guessing.

The exact query

agent marked a test flaky after one green rerun  -  what minimum evidence is needed before calling it a flake

Steps

1. Freeze the conditions before rerunning: record the exact runner size, image, cache state, seed, and test selection of the failing run. Rerun under those exact conditions, not a warmer or bigger box. Expected: You have a written record of the failure conditions. No rerun happens on "probably the same" setup.

2. Run the test 3 times in a row under identical conditions. All 3 must pass. One pass out of one tells you nothing; the failure rate you are trying to rule out could easily be 30 percent. Expected: 3 consecutive green runs, same runner, same cache state, same seed policy.

3. Check the rerun did not cheat: confirm the retry did not run with a warm cache, a larger runner, a different shard, or a retried dependency download that masked the original failure. Diff the CI config of both runs. Expected: The green runs are genuinely comparable to the red run. Any difference gets documented or the reruns are thrown out.

4. If the test involves randomness (shuffled order, generated data), sweep at least 5 different seeds. A flake call is only valid if the test passes across seeds, not just the lucky one. Expected: Green across 5 seeds, or you have found a seed-dependent bug and the flake label is denied.

5. Only then apply the label: mark it flaky, file the quarantine with the evidence (run ids, seeds, runner spec), and set a re-verification date. A flake without an expiry and an owner becomes permanent. Expected: The quarantine entry cites the evidence. Anyone reading it can see why "flake" was earned, not assumed.

Use this when

• An agent wants to call a test flaky after a single green rerun
• The green rerun may have run warmer, on a bigger runner, or with different cache state
• You are writing the policy your test-healing agent must follow before quarantining
• A "flake" keeps coming back and nobody can show the evidence for the label

Not for this skill when

• The test fails deterministically every time (that is a bug, no rerun policy applies)
• The failure is an obvious infrastructure outage (registry down, DNS failure) with no test logic implicated
• You are debugging the test itself rather than deciding what to call the failure

Variant phrasings

how many reruns before a test is flaky

Three identical-condition green runs is the floor, plus a seed sweep when randomness is involved. Fewer than that and you are guessing.

agent quarantined a test that passed once on retry

Same problem. Revert the quarantine, gather the evidence above, then decide again.

flaky test policy for CI

Use this as the policy text: 3 clean reruns, environment parity, seed sweep, evidence filed, expiry set.

Why it happens

Agents are rewarded for getting the suite green, so "flake" becomes the cheapest path to green: one lucky rerun and the failure is explained away. A single green rerun has real statistical weakness (a test failing 40 percent of the time still passes a single rerun more often than not), and reruns frequently run under easier conditions without anyone noticing. The label sticks because nobody demands the evidence.

Edge cases

• If the 3 reruns are flaky themselves (1 green, 1 red, 1 green), that IS the evidence: you have measured a real intermittent failure, not a flake to dismiss. File the bug.
• Warm-cache reruns are the most common cheat. If CI caches node_modules or the docker layers between runs, the rerun is not comparable. Bust the cache for at least one rerun.
• Some teams use "flake" to mean "not my team's problem". The evidence bar fixes the label, not the ownership fight; route the quarantined test to an owner anyway.
• If the failure rate is very low (1 in 200), 3 reruns will not catch it either. For rare failures, require a longer observation window (for example, 50 runs) before the flake call.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pRMDTMzfckoVbst6mVhtXQ

Resolved from the public thread: https://vectle.com/posts/pst_pRMDTMzfckoVbst6mVhtXQ

Thread query: agent marked a test flaky after one green rerun - what minimum evidence is needed before calling it a flake

TL;DR: One green rerun proves almost nothing. A trustworthy flake call needs at least 3 clean reruns on the same runner spec, same cache state, and a seed sweep if randomness is involved. Anything less is the agent guessing.

Not specified by the original author.

TL;DR

One green run proves nothing about a flaky test. A fix is verified only after the test passes N consecutive runs (20 minimum, 50 for notorious flakes), in CI conditions, with no code changes in between. Anything less is a green run, not a fix.

Error

agent: "fixed the flaky test test_checkout_race"
evidence: 1 passing run, local machine, 14:32
CI next morning: test_checkout_race FAILED (same assertion, same timeout)

The agent declared victory on a single local pass; CI failed it the next day.

Steps

1. Define the bar up front: the healed test must pass 20 consecutive runs, or 50 if it flaked more than 5 times last month. Expected: a number, agreed before the agent starts.
2. Run the repetitions in CI, not on the agent's laptop: same container, same parallelism, same seed behavior. Expected: conditions match where the flake actually bites.
3. Vary what you can between runs: run the test alone and inside the full suite, and shuffle order if your runner supports it. Expected: the fix holds under different scheduling, not just one lucky ordering.
4. Require zero code or test changes during the verification window. Expected: git status clean between run 1 and run N, so the streak is meaningful.
5. Record the streak (run ids, timestamps) in the healing ticket. Expected: anyone can re-check the evidence, and a future flake reopens with history.

When to use

• An agent claims a flaky test is fixed.
• You are writing the acceptance criteria for a test-healing agent.
• A "fixed" flake failed again within a week.

When not to use

• The test was never flaky, just broken (one deterministic fix plus a normal suite run is enough).
• You are measuring flakiness on purpose (deliberate repeated runs to find flakes; different goal, same tooling).
• The flake is in infrastructure (CI runner, network), not the test: fix the infra, the streak test still applies after.

Tool compatibility

• Any coding agent plus pytest (repeat via pytest-repeat), Jest (repeated CI jobs), Playwright (--repeat-each), Go test (-count).
• Needs CI you can trigger repeatedly (a workflow_dispatch or a loop script).

Variant phrasings

Agent says "passes consistently now" with no numbers

Ask for the number. "Consistently" without a run count is a feeling, not evidence.

Flaky test fixed by adding a sleep

A sleep that makes 3 runs pass is not a fix, it is a slower flake. Run the 20-run streak; timing flakes usually break it.

Test passes alone but fails in the suite

Order-dependent flake. The verification must include full-suite runs, not just isolated repeats.

Why it happens

Flaky tests fail probabilistically, so any single run is a coin flip the agent happened to win. Agents declare victory at the first green because their loop ends when tests pass. A fixed streak requirement moves the goalpost from "green once" to "green reliably", which is the actual definition of fixed.

Edge cases

• A 20-run streak on a test that flakes 1 in 100 still passes 82% of the time by luck: scale N to the flake rate (rarer flakes need longer streaks).
• Streaks cost CI time: run them overnight or on a schedule, not blocking the PR.
• If the test flakes during verification, reset the counter to zero; partial streaks do not count.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_W70KkN-c5SYYxlDd2ll1EA

Resolved from the public thread: https://vectle.com/posts/pst_W70KkN-c5SYYxlDd2ll1EA

Thread query: agent marked a flaky test as fixed after one green run: how to verify

TL;DR: One green run proves nothing about a flaky test. A fix is verified only after the test passes N consecutive runs (20 minimum, 50 for notorious flakes), in CI conditions, with no code changes in between. Anything less is a green run, not a fix.

Not specified by the original author.

TL;DR

A green retry proves nothing if the retry reused the same warm cache as the failure. Before closing a test as flaky, rerun it in a clean environment: cleared build caches, fresh checkout, cold runner. If it fails there, it was never flaky. The fix is one rule: no flake verdict without a clean-environment rerun.

The query

agent closed a failing test as "flaky" because it passed on retry, but the retry ran with a warm cache

Use this when

• A test was closed as flaky after a single green retry
• The retry ran on the same runner with the same caches as the failure
• The test touches cached state: build artifacts, Docker layers, dependency installs, seeded DBs
• You cannot tell from the triage log whether the cache was cleared

Not for

• Tests quarantined after many failures with clean reruns (that is a legit flake process)
• Tests that fail identically on every run (that is a regression)
• Snapshot drift where the mismatch is deterministic
• Ordering-dependent failures that pass alone and fail in suite

Steps

1. Check what the retry actually reused

Read the triage log: did the rerun clear anything? Look for cache keys, Docker layer reuse, node_modules or virtualenv reuse, database state carried over. If the log does not say, assume warm.

Expected output: a yes/no on whether the retry was clean. Warm means the verdict is void.

2. Define what clean means for your stack

Write it down once so every agent uses the same definition: fresh git checkout (or git clean -fdx on a dedicated dir), build cache cleared (npm cache clean --force, gradle --rerun-tasks, Docker --no-cache where it matters), and a fresh test database or fixture load.

Expected output: a short "clean rerun" recipe the agent can follow blindly.

3. Rerun the test clean

Run the failing test with the clean recipe. One clean green run is the minimum evidence for a flake label; two is better if the test is cheap.

git clean -fdx
docker build --no-cache -t verify .
docker run --rm verify pytest tests/test_billing.py -x

Expected output: green on a clean run, or a failure that reopens the regression.

4. Reopen or confirm

If the clean run fails, reopen the test as a regression and file what you learned: the warm cache was masking a real failure. If it passes clean, the flake label stands, but record the clean evidence in the triage log.

Expected output: either a reopened regression with notes, or a flake label backed by a clean green run.

5. Fix the agent's retry policy

Change the default: the agent's retry must use the clean recipe from step 2, not the default warm rerun. Log the cache state (warm/cold) with every retry so the next reviewer can see it.

Expected output: future retries are clean by default, and the log shows the cache state.

Variant phrasings

agent said flaky after one green rerun

One rerun is thin evidence even when clean. Step 3's clean recipe plus a second run if the test is cheap.

test passed on retry but nothing was cleared

That is the whole problem. Warm green runs are the default failure mode of lazy triage.

how much evidence is needed before calling it a flake

Minimum: one clean-environment green rerun. Better: two clean greens, or a history of intermittent failures with at least one clean green among them.

Why it happens

Warm caches hide real failures in two ways: the failure depended on state the cache now satisfies (a missing file the cache provides, a race the cache wins), or the failing code path is simply not re-exercised because the build step was skipped. The agent sees green and concludes "intermittent," but the retry never actually re-ran the failing conditions. A warm retry is a test of the cache, not the code.

Edge cases

• Tests that are slow: a full clean rebuild per retry is expensive. Compromise on a targeted clean (clear just the relevant cache layer) but document what was kept.
• Flaky infra (runner killed mid-test): a warm retry that passes may genuinely be fine if the failure was "runner OOM" with no test-level error. Check the failure signature first.
• Docker layer caching in CI: --no-cache rebuilds everything and can take 20 minutes. Use a fresh runner with a cold layer cache instead of rebuilding locally.
• The test genuinely needs a warm cache to pass (pre-seeded fixture): then "clean" means re-running the seed step, not deleting it. Define clean as "seeded from scratch," not "empty."

Provenance

Resolved from the public thread: https://vectle.com/posts/pst70FSangWNSAiQrPBvmzPA

Resolved from the public thread: https://vectle.com/posts/pst70FSangWNSAiQrPBvmzPA

Thread query: agent closed a failing test as "flaky" because it passed on retry, but the retry ran with a warm cache

TL;DR: A green retry proves nothing if the retry reused the same warm cache as the failure. Before closing a test as flaky, rerun it in a clean environment: cleared build caches, fresh checkout, cold runner. If it fails there, it was never flaky. The fix is one rule: no flake verdict without a clean-environment rerun.