agent quarantined tests that failed only because its own prior run left the database dirty

TL;DR

The agent's own earlier run left the database dirty, so the tests failed and got quarantined for a mess the agent made. Reset the database between healing runs, make test setup hermetic (migrate and seed from scratch per run), and re-verify every quarantine decision against a clean database before accepting it.

The symptom

agent quarantined tests that failed only because its own prior run left the database dirty

Steps

1. Prove the database was dirty

Inspect the test database after the agent's run: leftover rows, half-applied migrations, or sequences advanced past the seed state. Compare against a freshly seeded database. Expected: Leftover state from the agent's prior run, absent in a clean seed.

2. Reset and rerun cleanly

Drop, migrate, and reseed the test database from scratch, then rerun the quarantined tests with no agent intervention in between. Expected: The quarantined tests pass on the clean database.

3. Unquarantine and make setup hermetic

Remove the quarantine flags. Fix the test setup so each run starts from a known state: transactional rollbacks, or full reset in setup. Expected: Tests pass consistently regardless of what ran before them.

4. Isolate the healer's side effects

Run healing attempts against a throwaway database (a cloned schema or a per-run database name) so the agent's experiments can never pollute the suite's database. Expected: Agent runs leave the shared test database untouched.

When to use

• Quarantined tests pass on a freshly seeded database
• The agent ran experiments against the suite's test database
• Failures mention duplicate keys, stale rows, or unexpected existing data

When not to use

• Tests fail on a clean database too (the quarantine may be legitimate)
• The pollution comes from other tests, not the agent (fix test isolation)
• The database is intentionally shared across tests (then the design needs review)

Why it happens

Healing agents iterate: they run the test, tweak, rerun. Each attempt writes to the database, and without a reset between attempts the state accumulates. Eventually the test fails because of the accumulated mess, and the agent quarantines the victim instead of cleaning up its own side effects.

Edge cases

• Parallel test runs sharing one database: give each worker its own database.
• Migrations that are not reversible: snapshot and restore instead of migrate-down.
• External services with state (queues, buckets): reset those too, not just the DB.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstujNO46i_XP-8Bt6bFra-A