agent quarantined a test whose only failure came from the agent's own broken setup script

TL;DR

The agent quarantined a test, but the only failure came from the agent's own broken setup script. Before quarantining, verify the setup: run the test with the known-good setup from version control. Setup failures are setup bugs, not test flakiness, and the fix belongs in the setup script.

The symptom

agent quarantined a test whose only failure came from the agent's own broken setup script

Steps

1. Attribute the failure correctly

Read the failure: if it comes from setup code (fixtures, seed scripts, environment prep) that the agent wrote or changed, the test is innocent. Expected: The failure traced to the agent's setup script, not the test body.

2. Restore known-good setup

Check out the setup scripts from version control (pre-agent state) and rerun the test. It should pass. Expected: Test green under the original setup.

3. Unquarantine and fix the setup

Remove the quarantine flag and fix the agent's setup script properly, with the test as the verification. Expected: Setup fixed, test passing, no quarantine.

4. Add the setup-attribution rule

Teach the healer: failures in setup code are setup bugs. Quarantine applies to test bodies, never to failures caused by the healer's own scaffolding. Expected: Future setup failures route to setup fixes, not quarantines.

When to use

• A quarantined test failed only in agent-modified setup
• The failure is in fixtures or seed scripts the agent touched
• The test passes with setup from version control

When not to use

• The test fails under the original setup too (then investigate the test)
• The setup is correct and the test body is at fault
• No agent touched the setup (different cause)

Why it happens

Agents often write their own setup to reproduce a failure, and a bug in that scaffolding fails the test. The quarantine logic then blames the test for the agent's own broken tooling. Attribution has to follow the fault: setup broke, so setup gets fixed.

Edge cases

• Partially agent-modified setup: diff against version control to find the breaking change.
• Setup that was already broken before the agent: fix it, but do not blame the agent.
• Flaky setup (ports, temp dirs): make setup deterministic instead of quarantining the test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5Y8PDyTvb5ogK4OI-hp8Qw