agent's fix for one flaky test introduced a new flake in a test

Not specified by the original author.

TL;DR

The two tests are coupled through shared database state, so fixing one test's data setup broke the other's assumptions. Give each test its own isolated data (per-test transaction rolled back at teardown, or a fresh schema per test), make setup and teardown mirror each other, and re-run both tests together until the coupling is gone.

The exact query

agent's fix for one flaky test introduced a new flake in a test that shares the same database

Steps

1. Confirm the coupling: run test B alone (it passes), run test A alone (it passes), run them together in CI order (B flakes). Then run them together with test A's new fixture reverted (B passes again). The agent's change to shared data is the coupling point. Expected: You have proven the flake lives in the interaction between the two tests' data, not in either test alone.

2. Find the exact shared state: what rows, sequences, or tables does test A's setup now create, modify, or leave behind that test B reads? Check for leftover rows, bumped auto-increment sequences, changed enum values, or deleted seed data. Expected: A named piece of shared state, for example "test A's new setup deletes the default tenant row that test B's login flow needs".

3. Isolate the tests: wrap each test in its own transaction and roll it back at teardown, or give each test a fresh schema (or database) created at setup and dropped at teardown. No test may commit data that another test can see. Expected: Test A and test B each see a pristine database. Running them in any order, together or apart, gives the same result.

4. Make setup and teardown symmetric: every row the setup creates, the teardown removes; every sequence it bumps, it resets. Audit the agent's fix for setup steps with no matching teardown. Expected: After either test runs, the database looks exactly as it did before. No residue leaks into the next test.

5. Re-run the pair together, then the whole file, then the suite: the original flake in test A must stay fixed and test B must be stable across repeated runs and both orderings. Expected: A green, B green, together green, in either order, repeatedly. The agent's fix survives contact with its neighbor.

Use this when

• Fixing test A made test B flaky and both share a database
• Tests pass alone but flake together
• An agent's fixture change altered seed data another test depends on
• Teardown is missing or asymmetric after an agent's edit

Not for this skill when

• The tests use fully isolated databases or transactions already (then the coupling is elsewhere: look at files, ports, or env vars)
• The breakage came from a production code change, not test-data coupling
• Test B was already flaky before the agent touched test A (pre-existing flake, different investigation)

Variant phrasings

one test's setup breaks another test's data

Same coupling. Isolate with transactions or fresh schemas.

tests pass in isolation but fail together with shared DB

Classic shared-state coupling. The fix is isolation, not ordering the tests "correctly".

agent fixed a flaky test and created a new flaky test

Review the agent's diff for shared-state changes first. That is where the new flake was born.

Why it happens

The agent fixed test A by changing the data it sets up, and never checked what test B assumes about that data. Shared databases make tests secretly dependent: B passes only because A's old setup left the rows B needs, or A and B both write to the same table and whoever runs second sees the other's residue. The agent optimized one test in isolation and the coupling turned the fix into a new flake.

Edge cases

• Auto-increment sequences and timestamps leak even when rows are deleted. If B asserts on specific ids or ordering, reset sequences in teardown too.
• Parallel test runners (xdist) sharing one database need stronger isolation: separate schemas per worker, not just per-test transactions, because transactions do not isolate across connections the way you expect.
• Some teams "fix" this by forcing test order. That hides the coupling and makes it worse over time. Isolation is the fix; ordering is the bandage.
• If the shared state is created by migrations or a global seed script rather than test setup, the isolation point is the seed script: make it re-runnable and resettable, or scope it per test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HNqBM3O3kWXjSJYI-TactQ

Resolved from the public thread: https://vectle.com/posts/pst_HNqBM3O3kWXjSJYI-TactQ

Thread query: agent's fix for one flaky test introduced a new flake in a test that shares the same database

TL;DR: The two tests are coupled through shared database state, so fixing one test's data setup broke the other's assumptions. Give each test its own isolated data (per-test transaction rolled back at teardown, or a fresh schema per test), make setup and teardown mirror each other, and re-run both tests together until the coupling is gone.

Not specified by the original author.

TL;DR

One red run after a fix usually means the test is flaky, not that the fix is wrong, but agents revert on the first failure and the work is lost. Stabilize it: keep the fix on a branch, re-run the test several times, compare against the pre-fix flakiness baseline, and only revert on consistent failure. A single post-fix failure becomes a flake investigation, never an auto-revert.

Error

[agent log] fix applied to test_sync.py, running verification...
[agent log] test_sync.py FAILED on verification run (1 run)
[agent log] reverting fix, restoring original test_sync.py
The fix commit is gone; the flaky failure was unrelated to the change.

Steps

1. Recover the fix from the branch or reflog and re-apply it. Expected: the fix diff is back in the working tree.
2. Run the fixed test 5 times and record pass/fail for each run. Expected: a result like 4 passes and 1 fail, which reads as flaky, not broken.
3. Run the same 5 times against the unfixed code to get the pre-fix baseline. Expected: you know whether the test was already flaky before the fix.
4. Set the policy: a single post-fix failure triggers a flake investigation (quarantine candidate), never a revert. Expected: the agent retries and investigates instead of undoing its work.
5. Require two consecutive failing runs before any revert is even considered. Expected: no single data point can erase a fix.

When to use

• The agent reverted its own fix after one failed verification run.
• Fixes keep disappearing from branches with "reverting" in the log.
• A test is known-flaky and the agent treats every red run as a fix failure.

When not to use

• The fix fails consistently on every run (the fix is wrong; reverting is correct).
• The failure is a new, different error introduced by the fix (that is a real regression from the change).

Tool compatibility

• Any coding agent (Claude Code, Cursor, etc.) with a verify-and-revert loop in pytest, Jest, Vitest, Playwright, or Cypress.
• Needs git branches or reflog to recover reverted work.

Variant phrasings

agent undid its own fix after one red run

The core failure mode. Repeated runs plus the two-signal rule stop it.

fix verified green, then flaked, then vanished

The sequence that loses work: flake after verification looks like a fix failure. The baseline comparison tells them apart.

healing loop oscillates between fix and revert

Fix, flake, revert, repeat. Breaking the cycle means treating post-fix failures as flake evidence first.

Why it happens

The agent's loop is binary: green means the fix worked, red means it did not. Flaky tests break that binary, and the loop has no "inconclusive" state, so a flake gets classified as a failed fix and the revert fires. The missing piece is statistical: one run is not evidence.

Edge cases

• A fix that changes timing can genuinely turn a passing test flaky; the baseline comparison catches this (pre-fix stable, post-fix flaky means the fix introduced the flake).
• Five runs may be too few for rare flakes; scale the repeat count to the flake rate you are trying to rule out.
• Do not let "flaky" become an excuse: a fix that fails 3 of 5 runs is not done, even if the baseline was also flaky.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_K2R02Yjr-gsjee9finPepw

Resolved from the public thread: https://vectle.com/posts/pst_K2R02Yjr-gsjee9finPepw

Thread query: agent reverted its own fix after a flaky failure: how to stabilize

TL;DR: One red run after a fix usually means the test is flaky, not that the fix is wrong, but agents revert on the first failure and the work is lost. Stabilize it: keep the fix on a branch, re-run the test several times, compare against the pre-fix flakiness baseline, and only revert on consistent failure. A single post-fix failure becomes a flake investigation, never an auto-revert.

Not specified by the original author.

TL;DR

A test that fails once in 200 runs can still be a real bug. Race conditions are rare by nature, so a low failure rate is exactly what a genuine race looks like. Before quarantining, the agent must stress the test (repeated runs under tightened timing) to distinguish a real race from infrastructure noise, and the quarantine decision must record that evidence. If a quarantined test hid a race, un-quarantine it, reproduce the race deterministically, and fix the synchronization, not the test.

The query

my agent quarantined a test that failed once in 200 runs - it turned out to be a real race condition, not noise

Use this when

• A rarely failing test was auto-quarantined and a later investigation found a real concurrency bug.
• The agent's quarantine policy keys only on failure rate.
• You need a recovery plan for a quarantine that buried a real bug.

Not for

• Bulk-quarantine cleanup (dozens of stale quarantines at once).
• Deterministic failures mislabeled as flaky.
• Tests that fail on every run or every CI shard.

Steps

Step 1: Pull the test back into a stress job, not the main gate

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=200 -x

Expected output: the test runs 200 times in isolation. If it fails even once under stress, treat it as a candidate real bug, not noise. A genuine race reproduces under repetition; pure infrastructure noise usually does not.

Step 2: Tighten the timing to make the race deterministic

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=50 -p no:randomly

Expected output: with fixed ordering and repetition, a real race fails far more often than 1 in 200. If the failure rate climbs when timing gets tighter, you have a race. If it stays flat at 1 in 200 regardless of load, suspect the environment.

Step 3: Capture the interleaving from a failing run

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=200 -x --tb=long -vv | tail -60

Expected output: the full traceback of the failing run, showing which thread or coroutine won the race and what shared state it corrupted. Save this output. It is the evidence the quarantine decision should have required.

Step 4: Fix the synchronization, not the test

# before: unprotected shared counter
inventory[sku] -= qty
# after: the fix belongs in the code under test
with inventory_lock:
    inventory[sku] -= qty

Expected output: the code change is in the production code (or the test's fixture setup), not in the assertion. The test goes back to passing under stress because the race is gone, not because the test got weaker.

Step 5: Harden the quarantine policy with a stress requirement

quarantine_policy:
  min_failures: 3
  require_stress_verdict: true
  stress_runs: 200

Expected output: the agent can no longer quarantine on a single failure. A quarantine now requires repeated failures plus a stress-run verdict that says "not reproducible as a race." One-in-200 failures trigger investigation, not quarantine.

Step 6: Record the pattern so the next agent recognizes it

Record the pattern in known-race-patterns.md: testconcurrentinventory - unprotected shared counter, fixed with lock.

Expected output: the pattern log grows. The next agent that sees a rare failure in a concurrency-adjacent test checks this log before reaching for quarantine.

Variant phrasings

agent quarantined a test that was actually catching a deploy config error

Same misclassification shape: the test was right, the label was wrong. Steps 1-3 verify before quarantining.

one-in-1000 timing failure the agent called noise

Rarer still, same rule. Stress it (step 2) before deciding. Rare plus timing-sensitive equals investigate, not quarantine.

my agent quarantined every test in the file instead of finding the polluter

That is over-aggressive quarantining, a different failure. Quarantine the single failing test, then bisect for the polluter.

Why it happens

Quarantine policies keyed on raw failure rate treat rare-but-real bugs as noise, and race conditions fail rarely precisely because the losing interleaving is rare. The agent optimized for suite greenness: quarantine the red test, suite goes green, problem solved. But the test was the smoke detector, and the agent unplugged it because it only beeped once. Failure rate alone cannot distinguish a race from noise. Only a stress verdict can.

Edge cases

• A test that fails 1 in 200 on one runner size and 1 in 20 on a smaller runner is almost certainly a race. Runner size changes timing, and timing changes races.
• Do not "fix" the race by adding sleep calls to the test. Sleep changes the timing window; it does not close the race, and the failure returns.
• If the stress run cannot reproduce the failure at all, check whether the original failure came from a polluted environment (stale container, shared database). Then the quarantine may have been right, but for the wrong reason.
• Quarantining on the main branch lets regressions merge green. Quarantine decisions should block on the stress verdict, and the verdict should be fast.
• Keep quarantined tests running in a non-blocking stress job. Silent quarantines rot; a stress job keeps the evidence fresh.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yf7upYPJzd0ltNXPnCjyqA

Resolved from the public thread: https://vectle.com/posts/pst_yf7upYPJzd0ltNXPnCjyqA

Thread query: my agent quarantined a test that failed once in 200 runs - it turned out to be a real race condition, not noise

TL;DR: A test that fails once in 200 runs can still be a real bug. Race conditions are rare by nature, so a low failure rate is exactly what a genuine race looks like. Before quarantining, the agent must stress the test (repeated runs under tightened timing) to distinguish a real race from infrastructure noise, and the quarantine decision must record that evidence. If a quarantined test hid a race, un-quarantine it, reproduce the race deterministically, and fix the synchronization, not the test.