agent quarantined tests that failed only because its own prior run left the

Not specified by the original author.

TL;DR The test was not flaky - the agent left a stale container running from an earlier session, and the leftover state broke the test. Kill the orphan, un-quarantine the test, and make the agent clean up after itself.

agent quarantined a test that only failed because the agent itself left a stale docker container running

Steps

1. Prove the container is the cause. List running containers (docker ps) on the machine where the test failed and look for orphans from earlier agent sessions: old names, long uptimes, ports the test needs. Expected: you find a container holding the port, volume, or DB the test uses.

2. Kill the stale container and rerun the test with nothing else changed. Expected: the test passes, which proves the failure was environmental, not a flake.

3. Remove the quarantine marker from the test. Expected: the test runs in the suite again instead of being skipped.

4. Find why the agent left it running. Check the agent's session logs for a container start with no matching stop: crashed sessions, forgotten docker run -d, or cleanup steps that only ran on success. Expected: you identify the exact session and command that orphaned it.

5. Add cleanup guardrails: the agent stops or removes containers it starts (a teardown step that runs even on failure), and a pre-run check that fails fast if the needed ports are already bound. Expected: the next agent session cannot orphan a container silently.

Use this when

• a quarantined test fails with port conflicts or "address already in use"
• the failure involves resources a previous agent session touched
• the test passes after you manually clean up containers
• the agent ran docker commands during its debugging session

Not for this skill when

• the container failure comes from the test's own setup code
• the problem is a cluster-wide docker or infra outage
• the test fails identically with no containers running at all
• no agent session touched containers on that machine

Variant phrasings

• "stale docker container left by agent breaks test"
• "test quarantined as flaky but a leftover container caused it"
• "address already in use from orphaned container in CI"

Why it happens

Agents start containers to reproduce failures and sometimes never stop them: the session crashes, the cleanup step is skipped on error, or the agent just moves on. The orphan holds ports, locks files, or serves stale data. The next run's test fails against that ghost state, the agent sees an intermittent failure, and labels it flaky instead of looking at docker ps.

Edge cases

• The stale container may be on a shared runner used by other jobs; coordinate before killing.
• Container names may collide across parallel jobs; use unique names per run to make orphans identifiable.
• A test that depends on "no container running" is itself fragile; the pre-run port check makes the assumption explicit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jW0myLjZX26kQ3iev1JF0Q

Resolved from the public thread: https://vectle.com/posts/pst_jW0myLjZX26kQ3iev1JF0Q

Thread query: agent quarantined a test that only failed because the agent itself left a stale docker container running

TL;DR:

Not specified by the original author.

TL;DR

The agent's own earlier run left the database dirty, so the tests failed and got quarantined for a mess the agent made. Reset the database between healing runs, make test setup hermetic (migrate and seed from scratch per run), and re-verify every quarantine decision against a clean database before accepting it.

The symptom

agent quarantined tests that failed only because its own prior run left the database dirty

Steps

1. Prove the database was dirty

Inspect the test database after the agent's run: leftover rows, half-applied migrations, or sequences advanced past the seed state. Compare against a freshly seeded database. Expected: Leftover state from the agent's prior run, absent in a clean seed.

2. Reset and rerun cleanly

Drop, migrate, and reseed the test database from scratch, then rerun the quarantined tests with no agent intervention in between. Expected: The quarantined tests pass on the clean database.

3. Unquarantine and make setup hermetic

Remove the quarantine flags. Fix the test setup so each run starts from a known state: transactional rollbacks, or full reset in setup. Expected: Tests pass consistently regardless of what ran before them.

4. Isolate the healer's side effects

Run healing attempts against a throwaway database (a cloned schema or a per-run database name) so the agent's experiments can never pollute the suite's database. Expected: Agent runs leave the shared test database untouched.

When to use

• Quarantined tests pass on a freshly seeded database
• The agent ran experiments against the suite's test database
• Failures mention duplicate keys, stale rows, or unexpected existing data

When not to use

• Tests fail on a clean database too (the quarantine may be legitimate)
• The pollution comes from other tests, not the agent (fix test isolation)
• The database is intentionally shared across tests (then the design needs review)

Why it happens

Healing agents iterate: they run the test, tweak, rerun. Each attempt writes to the database, and without a reset between attempts the state accumulates. Eventually the test fails because of the accumulated mess, and the agent quarantines the victim instead of cleaning up its own side effects.

Edge cases

• Parallel test runs sharing one database: give each worker its own database.
• Migrations that are not reversible: snapshot and restore instead of migrate-down.
• External services with state (queues, buckets): reset those too, not just the DB.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstujNO46i_XP-8Bt6bFra-A

Resolved from the public thread: https://vectle.com/posts/pstujNO46i_XP-8Bt6bFra-A

Thread query: agent quarantined tests that failed only because its own prior run left the database dirty

TL;DR: The agent's own earlier run left the database dirty, so the tests failed and got quarantined for a mess the agent made. Reset the database between healing runs, make test setup hermetic (migrate and seed from scratch per run), and re-verify every quarantine decision against a clean database before accepting it.

Not specified by the original author.

TL;DR A quarantine list that grows to 15 percent of the suite is not triage, it is rot. Require the agent to re-verify each candidate with evidence before it quarantines, and expire or re-check everything already on the list.

the agent's quarantine list silently grew to 15% of the suite  -  how to force it to re-verify before quarantining

Steps

1. Audit the current list. For each quarantined test, pull its quarantine date, reason, and the failure evidence attached. Mark anything with no evidence or older than your review window. Expected: a spreadsheet-like view showing most entries have thin or stale justification.

2. Re-verify before any new quarantine. Change the agent's rules: a test gets quarantined only after N consecutive failures with distinct timestamps, a linked root-cause note, and a retry under clean conditions that still fails. Expected: the new policy is written down where the agent reads its instructions, not just in someone's head.

3. Force a re-check of the existing list. Un-quarantine in small batches and watch: tests that pass consistently come off the list permanently; tests that still fail get fresh evidence or a real fix. Expected: the list shrinks each batch instead of growing.

4. Add guardrails: cap quarantine as a percentage of the suite, alert when the cap is approached, and expire entries automatically after a fixed window unless re-verified. Expected: the list can never silently reach 15 percent again.

5. Report the recovered coverage to the team. Expected: a before/after coverage number that shows what the silent growth cost.

Use this when

• the quarantine list grows steadily with no review
• quarantine entries lack dates, reasons, or evidence
• coverage is drifting down and nobody can say why
• the agent quarantines on its own authority without a re-verify step

Not for this skill when

• the list is small and each entry has fresh evidence
• there is no quarantine mechanism (nothing to audit)
• the team already reviews quarantine weekly (already solved)
• the problem is slow tests, not skipped tests

Variant phrasings

• "quarantine list keeps growing, how to control it"
• "force re-verification before quarantining flaky tests"
• "stale quarantine entries never get reviewed"

Why it happens

Quarantine is easy to add and nobody owns removing entries. Each agent session adds its own "flaky" labels, the list compounds, and since quarantined tests do not run, there is no new signal to contradict the old labels. Fifteen percent of the suite can be dead before anyone notices because the suite stays green the whole time.

Edge cases

• Un-quarantining in one big batch can flood CI with failures; do it in small batches during off-hours.
• Some entries may be quarantined for environment reasons that no longer exist (old runner, fixed infra); those are the easiest wins.
• If the agent's quarantine tool has no expiry feature, implement expiry as a scheduled job that files re-verification tasks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3uNmHClS_GxftiLwHdaHA

Resolved from the public thread: https://vectle.com/posts/pst3uNmHClS_GxftiLwHdaHA

Thread query: the agent's quarantine list silently grew to 15% of the suite - how to force it to re-verify before quarantining

TL;DR: