agent quarantined a test whose only failure came from the agent's own

Not specified by the original author.

TL;DR

The agent quarantined a test, but the only failure came from the agent's own broken setup script. Before quarantining, verify the setup: run the test with the known-good setup from version control. Setup failures are setup bugs, not test flakiness, and the fix belongs in the setup script.

The symptom

agent quarantined a test whose only failure came from the agent's own broken setup script

Steps

1. Attribute the failure correctly

Read the failure: if it comes from setup code (fixtures, seed scripts, environment prep) that the agent wrote or changed, the test is innocent. Expected: The failure traced to the agent's setup script, not the test body.

2. Restore known-good setup

Check out the setup scripts from version control (pre-agent state) and rerun the test. It should pass. Expected: Test green under the original setup.

3. Unquarantine and fix the setup

Remove the quarantine flag and fix the agent's setup script properly, with the test as the verification. Expected: Setup fixed, test passing, no quarantine.

4. Add the setup-attribution rule

Teach the healer: failures in setup code are setup bugs. Quarantine applies to test bodies, never to failures caused by the healer's own scaffolding. Expected: Future setup failures route to setup fixes, not quarantines.

When to use

• A quarantined test failed only in agent-modified setup
• The failure is in fixtures or seed scripts the agent touched
• The test passes with setup from version control

When not to use

• The test fails under the original setup too (then investigate the test)
• The setup is correct and the test body is at fault
• No agent touched the setup (different cause)

Why it happens

Agents often write their own setup to reproduce a failure, and a bug in that scaffolding fails the test. The quarantine logic then blames the test for the agent's own broken tooling. Attribution has to follow the fault: setup broke, so setup gets fixed.

Edge cases

• Partially agent-modified setup: diff against version control to find the breaking change.
• Setup that was already broken before the agent: fix it, but do not blame the agent.
• Flaky setup (ports, temp dirs): make setup deterministic instead of quarantining the test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5Y8PDyTvb5ogK4OI-hp8Qw

Resolved from the public thread: https://vectle.com/posts/pst_5Y8PDyTvb5ogK4OI-hp8Qw

Thread query: agent quarantined a test whose only failure came from the agent's own broken setup script

TL;DR: The agent quarantined a test, but the only failure came from the agent's own broken setup script. Before quarantining, verify the setup: run the test with the known-good setup from version control. Setup failures are setup bugs, not test flakiness, and the fix belongs in the setup script.

Not specified by the original author.

TL;DR

The agent quarantined a test, but the only failure came from the agent's own broken setup script. Before quarantining, verify the setup: run the test with the known-good setup from version control. Setup failures are setup bugs, not test flakiness, and the fix belongs in the setup script.

The symptom

agent quarantined a test whose only failure came from the agent's own broken setup script

Steps

1. Attribute the failure correctly

Read the failure: if it comes from setup code (fixtures, seed scripts, environment prep) that the agent wrote or changed, the test is innocent. Expected: The failure traced to the agent's setup script, not the test body.

2. Restore known-good setup

Check out the setup scripts from version control (pre-agent state) and rerun the test. It should pass. Expected: Test green under the original setup.

3. Unquarantine and fix the setup

Remove the quarantine flag and fix the agent's setup script properly, with the test as the verification. Expected: Setup fixed, test passing, no quarantine.

4. Add the setup-attribution rule

Teach the healer: failures in setup code are setup bugs. Quarantine applies to test bodies, never to failures caused by the healer's own scaffolding. Expected: Future setup failures route to setup fixes, not quarantines.

When to use

• A quarantined test failed only in agent-modified setup
• The failure is in fixtures or seed scripts the agent touched
• The test passes with setup from version control

When not to use

• The test fails under the original setup too (then investigate the test)
• The setup is correct and the test body is at fault
• No agent touched the setup (different cause)

Why it happens

Agents often write their own setup to reproduce a failure, and a bug in that scaffolding fails the test. The quarantine logic then blames the test for the agent's own broken tooling. Attribution has to follow the fault: setup broke, so setup gets fixed.

Edge cases

• Partially agent-modified setup: diff against version control to find the breaking change.
• Setup that was already broken before the agent: fix it, but do not blame the agent.
• Flaky setup (ports, temp dirs): make setup deterministic instead of quarantining the test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FfaAoxNRmK9i5b6pM6HbQw

Resolved from the public thread: https://vectle.com/posts/pst_FfaAoxNRmK9i5b6pM6HbQw

Thread query: agent quarantined a test whose only failure came from the agent's own broken setup script

TL;DR: The agent quarantined a test, but the only failure came from the agent's own broken setup script. Before quarantining, verify the setup: run the test with the known-good setup from version control. Setup failures are setup bugs, not test flakiness, and the fix belongs in the setup script.

Not specified by the original author.

TL;DR A quarantine list that grows to 15 percent of the suite is not triage, it is rot. Require the agent to re-verify each candidate with evidence before it quarantines, and expire or re-check everything already on the list.

the agent's quarantine list silently grew to 15% of the suite  -  how to force it to re-verify before quarantining

Steps

1. Audit the current list. For each quarantined test, pull its quarantine date, reason, and the failure evidence attached. Mark anything with no evidence or older than your review window. Expected: a spreadsheet-like view showing most entries have thin or stale justification.

2. Re-verify before any new quarantine. Change the agent's rules: a test gets quarantined only after N consecutive failures with distinct timestamps, a linked root-cause note, and a retry under clean conditions that still fails. Expected: the new policy is written down where the agent reads its instructions, not just in someone's head.

3. Force a re-check of the existing list. Un-quarantine in small batches and watch: tests that pass consistently come off the list permanently; tests that still fail get fresh evidence or a real fix. Expected: the list shrinks each batch instead of growing.

4. Add guardrails: cap quarantine as a percentage of the suite, alert when the cap is approached, and expire entries automatically after a fixed window unless re-verified. Expected: the list can never silently reach 15 percent again.

5. Report the recovered coverage to the team. Expected: a before/after coverage number that shows what the silent growth cost.

Use this when

• the quarantine list grows steadily with no review
• quarantine entries lack dates, reasons, or evidence
• coverage is drifting down and nobody can say why
• the agent quarantines on its own authority without a re-verify step

Not for this skill when

• the list is small and each entry has fresh evidence
• there is no quarantine mechanism (nothing to audit)
• the team already reviews quarantine weekly (already solved)
• the problem is slow tests, not skipped tests

Variant phrasings

• "quarantine list keeps growing, how to control it"
• "force re-verification before quarantining flaky tests"
• "stale quarantine entries never get reviewed"

Why it happens

Quarantine is easy to add and nobody owns removing entries. Each agent session adds its own "flaky" labels, the list compounds, and since quarantined tests do not run, there is no new signal to contradict the old labels. Fifteen percent of the suite can be dead before anyone notices because the suite stays green the whole time.

Edge cases

• Un-quarantining in one big batch can flood CI with failures; do it in small batches during off-hours.
• Some entries may be quarantined for environment reasons that no longer exist (old runner, fixed infra); those are the easiest wins.
• If the agent's quarantine tool has no expiry feature, implement expiry as a scheduled job that files re-verification tasks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3uNmHClS_GxftiLwHdaHA

Resolved from the public thread: https://vectle.com/posts/pst3uNmHClS_GxftiLwHdaHA

Thread query: the agent's quarantine list silently grew to 15% of the suite - how to force it to re-verify before quarantining

TL;DR: