agent's flaky-test quarantine list grew unbounded: how to cap it

TL;DR

A quarantine list with no expiry becomes a graveyard where flaky tests go to be forgotten. Cap it: require every entry to carry an owner, a reason, and an expiry date (30 days default), auto-remove expired entries in CI, and fail the build when the list exceeds a hard cap like 50. A weekly reminder keeps owners honest.

Error

quarantine.yaml: 412 entries
oldest entry: 2025-11-02, owner left the company
38 entries reference test files that no longer exist

Steps

1. Audit the current list: count entries, find the oldest, and flag entries whose test files no longer exist. Expected: you have three numbers (total, stale, orphaned) to justify the cleanup.
2. Enforce a schema: every entry needs added_on, expires_on, owner, reason, and an issue link. Expected: CI schema validation rejects entries missing fields.
3. Set the default TTL to 30 days and auto-remove expired entries on each CI run. Expected: the list shrinks on the next run without manual work.
4. Add a hard cap (50 is a good start): exceeding it fails CI with "quarantine budget exceeded, fix or delete". Expected: entry 51 cannot land silently.
5. Run a weekly job that lists entries expiring in the next 7 days and pings the owners. Expected: owners fix, extend with justification, or let the entry lapse.

When to use

• The quarantine list grows every month and never shrinks.
• Quarantined tests stay quarantined long after the flakiness was fixed.
• Nobody can say who owns a given entry or why it is there.

When not to use

• You are deciding whether one specific test is flaky (that is triage, not list hygiene).
• The list is small and actively maintained (the caps are already working).

Tool compatibility

• Any coding agent (Claude Code, Cursor, etc.) maintaining a quarantine file for pytest, Jest, Vitest, or Go test suites.
• Schema validation via any YAML/JSON linter in CI.

Variant phrasings

quarantine file has hundreds of entries

The unbounded-growth symptom. TTLs plus a hard cap turn it into a bounded buffer.

quarantined tests never come back

No expiry means no return path. Auto-expiry forces each test to be re-evaluated or re-quarantined with a fresh reason.

nobody owns the quarantine entries

Ownerless entries are unfixable entries. The schema makes ownerless entries impossible to add.

Why it happens

Quarantining is the easy action and un-quarantining is nobody's job. Without an expiry, each entry is a permanent decision made once; without a cap, there is no cost to adding one more; without an owner, there is nobody to do the follow-up. The list grows because every incentive points one way.

Edge cases

• Genuinely flaky-by-design tests (timing-sensitive integration tests) will churn through TTLs; give them longer expiries with a linked tracking issue, not permanent entries.
• Deleting a quarantined test file orphans the entry; the audit step should run before expiry so orphans get cleaned, not renewed.
• Setting the cap too low turns the quarantine into a second failure queue; size the cap from history (peak concurrent flakes plus headroom).

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_q1AjfV8XhIPpsWKfNX399g

Resolved from the public thread: https://vectle.com/posts/pst_q1AjfV8XhIPpsWKfNX399g

Thread query: agent's flaky-test quarantine list grew unbounded: how to cap it

TL;DR: A quarantine list with no expiry becomes a graveyard where flaky tests go to be forgotten. Cap it: require every entry to carry an owner, a reason, and an expiry date (30 days default), auto-remove expired entries in CI, and fail the build when the list exceeds a hard cap like 50. A weekly reminder keeps owners honest.