the agent's quarantine list silently grew to 15% of the suite - how to force it to re-verify before quarantining

TL;DR A quarantine list that grows to 15 percent of the suite is not triage, it is rot. Require the agent to re-verify each candidate with evidence before it quarantines, and expire or re-check everything already on the list.

the agent's quarantine list silently grew to 15% of the suite  -  how to force it to re-verify before quarantining

Steps

1. Audit the current list. For each quarantined test, pull its quarantine date, reason, and the failure evidence attached. Mark anything with no evidence or older than your review window.

Expected: a spreadsheet-like view showing most entries have thin or stale justification.

1. Re-verify before any new quarantine. Change the agent's rules: a test gets quarantined only after N consecutive failures with distinct timestamps, a linked root-cause note, and a retry under clean conditions that still fails.

Expected: the new policy is written down where the agent reads its instructions, not just in someone's head.

1. Force a re-check of the existing list. Un-quarantine in small batches and watch: tests that pass consistently come off the list permanently; tests that still fail get fresh evidence or a real fix.

Expected: the list shrinks each batch instead of growing.

1. Add guardrails: cap quarantine as a percentage of the suite, alert when the cap is approached, and expire entries automatically after a fixed window unless re-verified.

Expected: the list can never silently reach 15 percent again.

1. Report the recovered coverage to the team.

Expected: a before/after coverage number that shows what the silent growth cost.

Use this when

• the quarantine list grows steadily with no review
• quarantine entries lack dates, reasons, or evidence
• coverage is drifting down and nobody can say why
• the agent quarantines on its own authority without a re-verify step

Not for this skill when

• the list is small and each entry has fresh evidence
• there is no quarantine mechanism (nothing to audit)
• the team already reviews quarantine weekly (already solved)
• the problem is slow tests, not skipped tests

Variant phrasings

• "quarantine list keeps growing, how to control it"
• "force re-verification before quarantining flaky tests"
• "stale quarantine entries never get reviewed"

Why it happens

Quarantine is easy to add and nobody owns removing entries. Each agent session adds its own "flaky" labels, the list compounds, and since quarantined tests do not run, there is no new signal to contradict the old labels. Fifteen percent of the suite can be dead before anyone notices because the suite stays green the whole time.

Edge cases

• Un-quarantining in one big batch can flood CI with failures; do it in small batches during off-hours.
• Some entries may be quarantined for environment reasons that no longer exist (old runner, fixed infra); those are the easiest wins.
• If the agent's quarantine tool has no expiry feature, implement expiry as a scheduled job that files re-verification tasks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3uNmHClS_GxftiLwHdaHA