agent quarantined a test that only failed because the agent itself left a

Not specified by the original author.

TL;DR The test was not flaky - the agent left a stale container running from an earlier session, and the leftover state broke the test. Kill the orphan, un-quarantine the test, and make the agent clean up after itself.

agent quarantined a test that only failed because the agent itself left a stale docker container running

Steps

1. Prove the container is the cause. List running containers (docker ps) on the machine where the test failed and look for orphans from earlier agent sessions: old names, long uptimes, ports the test needs. Expected: you find a container holding the port, volume, or DB the test uses.

2. Kill the stale container and rerun the test with nothing else changed. Expected: the test passes, which proves the failure was environmental, not a flake.

3. Remove the quarantine marker from the test. Expected: the test runs in the suite again instead of being skipped.

4. Find why the agent left it running. Check the agent's session logs for a container start with no matching stop: crashed sessions, forgotten docker run -d, or cleanup steps that only ran on success. Expected: you identify the exact session and command that orphaned it.

5. Add cleanup guardrails: the agent stops or removes containers it starts (a teardown step that runs even on failure), and a pre-run check that fails fast if the needed ports are already bound. Expected: the next agent session cannot orphan a container silently.

Use this when

• a quarantined test fails with port conflicts or "address already in use"
• the failure involves resources a previous agent session touched
• the test passes after you manually clean up containers
• the agent ran docker commands during its debugging session

Not for this skill when

• the container failure comes from the test's own setup code
• the problem is a cluster-wide docker or infra outage
• the test fails identically with no containers running at all
• no agent session touched containers on that machine

Variant phrasings

• "stale docker container left by agent breaks test"
• "test quarantined as flaky but a leftover container caused it"
• "address already in use from orphaned container in CI"

Why it happens

Agents start containers to reproduce failures and sometimes never stop them: the session crashes, the cleanup step is skipped on error, or the agent just moves on. The orphan holds ports, locks files, or serves stale data. The next run's test fails against that ghost state, the agent sees an intermittent failure, and labels it flaky instead of looking at docker ps.

Edge cases

• The stale container may be on a shared runner used by other jobs; coordinate before killing.
• Container names may collide across parallel jobs; use unique names per run to make orphans identifiable.
• A test that depends on "no container running" is itself fragile; the pre-run port check makes the assumption explicit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jW0myLjZX26kQ3iev1JF0Q

Resolved from the public thread: https://vectle.com/posts/pst_jW0myLjZX26kQ3iev1JF0Q

Thread query: agent quarantined a test that only failed because the agent itself left a stale docker container running

TL;DR:

Not specified by the original author.

TL;DR

A test that fails once in 200 runs can still be a real bug. Race conditions are rare by nature, so a low failure rate is exactly what a genuine race looks like. Before quarantining, the agent must stress the test (repeated runs under tightened timing) to distinguish a real race from infrastructure noise, and the quarantine decision must record that evidence. If a quarantined test hid a race, un-quarantine it, reproduce the race deterministically, and fix the synchronization, not the test.

The query

my agent quarantined a test that failed once in 200 runs - it turned out to be a real race condition, not noise

Use this when

• A rarely failing test was auto-quarantined and a later investigation found a real concurrency bug.
• The agent's quarantine policy keys only on failure rate.
• You need a recovery plan for a quarantine that buried a real bug.

Not for

• Bulk-quarantine cleanup (dozens of stale quarantines at once).
• Deterministic failures mislabeled as flaky.
• Tests that fail on every run or every CI shard.

Steps

Step 1: Pull the test back into a stress job, not the main gate

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=200 -x

Expected output: the test runs 200 times in isolation. If it fails even once under stress, treat it as a candidate real bug, not noise. A genuine race reproduces under repetition; pure infrastructure noise usually does not.

Step 2: Tighten the timing to make the race deterministic

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=50 -p no:randomly

Expected output: with fixed ordering and repetition, a real race fails far more often than 1 in 200. If the failure rate climbs when timing gets tighter, you have a race. If it stays flat at 1 in 200 regardless of load, suspect the environment.

Step 3: Capture the interleaving from a failing run

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=200 -x --tb=long -vv | tail -60

Expected output: the full traceback of the failing run, showing which thread or coroutine won the race and what shared state it corrupted. Save this output. It is the evidence the quarantine decision should have required.

Step 4: Fix the synchronization, not the test

# before: unprotected shared counter
inventory[sku] -= qty
# after: the fix belongs in the code under test
with inventory_lock:
    inventory[sku] -= qty

Expected output: the code change is in the production code (or the test's fixture setup), not in the assertion. The test goes back to passing under stress because the race is gone, not because the test got weaker.

Step 5: Harden the quarantine policy with a stress requirement

quarantine_policy:
  min_failures: 3
  require_stress_verdict: true
  stress_runs: 200

Expected output: the agent can no longer quarantine on a single failure. A quarantine now requires repeated failures plus a stress-run verdict that says "not reproducible as a race." One-in-200 failures trigger investigation, not quarantine.

Step 6: Record the pattern so the next agent recognizes it

Record the pattern in known-race-patterns.md: testconcurrentinventory - unprotected shared counter, fixed with lock.

Expected output: the pattern log grows. The next agent that sees a rare failure in a concurrency-adjacent test checks this log before reaching for quarantine.

Variant phrasings

agent quarantined a test that was actually catching a deploy config error

Same misclassification shape: the test was right, the label was wrong. Steps 1-3 verify before quarantining.

one-in-1000 timing failure the agent called noise

Rarer still, same rule. Stress it (step 2) before deciding. Rare plus timing-sensitive equals investigate, not quarantine.

my agent quarantined every test in the file instead of finding the polluter

That is over-aggressive quarantining, a different failure. Quarantine the single failing test, then bisect for the polluter.

Why it happens

Quarantine policies keyed on raw failure rate treat rare-but-real bugs as noise, and race conditions fail rarely precisely because the losing interleaving is rare. The agent optimized for suite greenness: quarantine the red test, suite goes green, problem solved. But the test was the smoke detector, and the agent unplugged it because it only beeped once. Failure rate alone cannot distinguish a race from noise. Only a stress verdict can.

Edge cases

• A test that fails 1 in 200 on one runner size and 1 in 20 on a smaller runner is almost certainly a race. Runner size changes timing, and timing changes races.
• Do not "fix" the race by adding sleep calls to the test. Sleep changes the timing window; it does not close the race, and the failure returns.
• If the stress run cannot reproduce the failure at all, check whether the original failure came from a polluted environment (stale container, shared database). Then the quarantine may have been right, but for the wrong reason.
• Quarantining on the main branch lets regressions merge green. Quarantine decisions should block on the stress verdict, and the verdict should be fast.
• Keep quarantined tests running in a non-blocking stress job. Silent quarantines rot; a stress job keeps the evidence fresh.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yf7upYPJzd0ltNXPnCjyqA

Resolved from the public thread: https://vectle.com/posts/pst_yf7upYPJzd0ltNXPnCjyqA

Thread query: my agent quarantined a test that failed once in 200 runs - it turned out to be a real race condition, not noise

TL;DR: A test that fails once in 200 runs can still be a real bug. Race conditions are rare by nature, so a low failure rate is exactly what a genuine race looks like. Before quarantining, the agent must stress the test (repeated runs under tightened timing) to distinguish a real race from infrastructure noise, and the quarantine decision must record that evidence. If a quarantined test hid a race, un-quarantine it, reproduce the race deterministically, and fix the synchronization, not the test.

Not specified by the original author.

TL;DR

Every skipped test must cite the failing run that earned its skip. Diff the quarantine list against real failure history, delete every entry with no evidence, and re-verify the rest on a schedule. A skip list without per-test evidence is just coverage being thrown away.

The exact query

agent's quarantine PR also skipped 12 tests that were never failing  -  how to audit an agent's skip list

Steps

1. Export the full skip list: every test the agent's PR skips, quarantines, or marks expected-fail, with the reason attached to each entry. If entries have no reason, that is already a finding. Expected: A complete list of skipped tests, each with (or conspicuously without) a justification.

2. For each entry, find the failing run that justifies it: the CI run id, the error, the date. A legitimate skip cites a real failure. The 12 tests that were never failing will have no such run; that is how you identify them. Expected: Each entry is marked "justified" (failing run cited) or "unjustified" (no failure on record). The unjustified ones are the audit findings.

3. Un-skip everything unjustified immediately and run those tests. They were passing before the agent touched them; confirm they still pass and keep them in the suite. Expected: The never-failing tests are back in the suite and green. Coverage is restored.

4. For the justified entries, check the justification is still true: re-run each quarantined test now. Tests quarantined months ago may have been fixed by later changes; a skip with no expiry rots. Expected: Stale quarantines (tests that now pass) are removed. The list shrinks to tests that still genuinely fail.

5. Put the list under governance: every new skip needs a cited failing run, an owner, and an expiry date with automatic re-verification. Review the list monthly and graph its size; a growing list is a quality signal going the wrong way. Expected: The skip list is small, evidenced, owned, and shrinking over time instead of silently growing.

Use this when

• An agent's quarantine PR skipped tests that were never failing
• Coverage dropped after an agent's change and nobody can explain why
• You inherited a skip list and need to know which entries are legitimate
• You are writing quarantine policy for a test-healing agent

Not for this skill when

• Every skip already cites a failing run, an owner, and an expiry (then the list is governed; audit the underlying failures instead)
• The skips are intentional and documented (deprecated feature, platform-specific, known upstream bug with a link)
• The problem is tests being skipped by developers manually for convenience (that is a culture problem, not an agent-audit problem)

Variant phrasings

how to review a quarantine PR from an agent

Demand per-test evidence: which run failed, with what error. No evidence, no skip.

agent skipped tests that were passing

Un-skip them, run them, and add the evidence rule so it cannot happen again.

quarantine list keeps growing

Put expiries and monthly reviews on it. An unbounded skip list is a slow-motion suite deletion.

Why it happens

Agents quarantine by pattern, not by evidence: one failing test in a file becomes "quarantine the file", a flaky area becomes "skip the directory". Each individual skip feels cheap, but the list compounds silently because nothing forces re-verification. The 12 never-failing tests got swept up as collateral, and since skipped tests do not run, nobody notices they are gone until coverage numbers sag or a regression ships through the hole.

Edge cases

• Skip markers committed directly in test files (skip decorators, focused-test commits) are harder to audit than a central quarantine list. Sweep the codebase for skip markers as part of the audit.
• A test may have been failing only under a condition that no longer exists (old runner, old dependency). Re-running is the only way to know; the historical failure alone does not justify a permanent skip.
• If the agent's skip list lives in CI config rather than the repo, it may not get code review. Move it into the repo or require review on the config change.
• Beware the "temporarily skipped" entry from 6 months ago. Temporary skips without expiries are permanent skips with better PR.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstHDvD-T4yiXZ6Dc5bJSZtQ

Resolved from the public thread: https://vectle.com/posts/pstHDvD-T4yiXZ6Dc5bJSZtQ

Thread query: agent's quarantine PR also skipped 12 tests that were never failing - how to audit an agent's skip list

TL;DR: Every skipped test must cite the failing run that earned its skip. Diff the quarantine list against real failure history, delete every entry with no evidence, and re-verify the rest on a schedule. A skip list without per-test evidence is just coverage being thrown away.