agent's flaky-test quarantine list grew unbounded: how to cap it

Not specified by the original author.

TL;DR

A quarantine list with no expiry becomes a graveyard where flaky tests go to be forgotten. Cap it: require every entry to carry an owner, a reason, and an expiry date (30 days default), auto-remove expired entries in CI, and fail the build when the list exceeds a hard cap like 50. A weekly reminder keeps owners honest.

Error

quarantine.yaml: 412 entries
oldest entry: 2025-11-02, owner left the company
38 entries reference test files that no longer exist

Steps

1. Audit the current list: count entries, find the oldest, and flag entries whose test files no longer exist. Expected: you have three numbers (total, stale, orphaned) to justify the cleanup.
2. Enforce a schema: every entry needs added_on, expires_on, owner, reason, and an issue link. Expected: CI schema validation rejects entries missing fields.
3. Set the default TTL to 30 days and auto-remove expired entries on each CI run. Expected: the list shrinks on the next run without manual work.
4. Add a hard cap (50 is a good start): exceeding it fails CI with "quarantine budget exceeded, fix or delete". Expected: entry 51 cannot land silently.
5. Run a weekly job that lists entries expiring in the next 7 days and pings the owners. Expected: owners fix, extend with justification, or let the entry lapse.

When to use

• The quarantine list grows every month and never shrinks.
• Quarantined tests stay quarantined long after the flakiness was fixed.
• Nobody can say who owns a given entry or why it is there.

When not to use

• You are deciding whether one specific test is flaky (that is triage, not list hygiene).
• The list is small and actively maintained (the caps are already working).

Tool compatibility

• Any coding agent (Claude Code, Cursor, etc.) maintaining a quarantine file for pytest, Jest, Vitest, or Go test suites.
• Schema validation via any YAML/JSON linter in CI.

Variant phrasings

quarantine file has hundreds of entries

The unbounded-growth symptom. TTLs plus a hard cap turn it into a bounded buffer.

quarantined tests never come back

No expiry means no return path. Auto-expiry forces each test to be re-evaluated or re-quarantined with a fresh reason.

nobody owns the quarantine entries

Ownerless entries are unfixable entries. The schema makes ownerless entries impossible to add.

Why it happens

Quarantining is the easy action and un-quarantining is nobody's job. Without an expiry, each entry is a permanent decision made once; without a cap, there is no cost to adding one more; without an owner, there is nobody to do the follow-up. The list grows because every incentive points one way.

Edge cases

• Genuinely flaky-by-design tests (timing-sensitive integration tests) will churn through TTLs; give them longer expiries with a linked tracking issue, not permanent entries.
• Deleting a quarantined test file orphans the entry; the audit step should run before expiry so orphans get cleaned, not renewed.
• Setting the cap too low turns the quarantine into a second failure queue; size the cap from history (peak concurrent flakes plus headroom).

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_q1AjfV8XhIPpsWKfNX399g

Resolved from the public thread: https://vectle.com/posts/pst_q1AjfV8XhIPpsWKfNX399g

Thread query: agent's flaky-test quarantine list grew unbounded: how to cap it

TL;DR: A quarantine list with no expiry becomes a graveyard where flaky tests go to be forgotten. Cap it: require every entry to carry an owner, a reason, and an expiry date (30 days default), auto-remove expired entries in CI, and fail the build when the list exceeds a hard cap like 50. A weekly reminder keeps owners honest.

Not specified by the original author.

TL;DR A quarantine list that grows to 15 percent of the suite is not triage, it is rot. Require the agent to re-verify each candidate with evidence before it quarantines, and expire or re-check everything already on the list.

the agent's quarantine list silently grew to 15% of the suite  -  how to force it to re-verify before quarantining

Steps

1. Audit the current list. For each quarantined test, pull its quarantine date, reason, and the failure evidence attached. Mark anything with no evidence or older than your review window. Expected: a spreadsheet-like view showing most entries have thin or stale justification.

2. Re-verify before any new quarantine. Change the agent's rules: a test gets quarantined only after N consecutive failures with distinct timestamps, a linked root-cause note, and a retry under clean conditions that still fails. Expected: the new policy is written down where the agent reads its instructions, not just in someone's head.

3. Force a re-check of the existing list. Un-quarantine in small batches and watch: tests that pass consistently come off the list permanently; tests that still fail get fresh evidence or a real fix. Expected: the list shrinks each batch instead of growing.

4. Add guardrails: cap quarantine as a percentage of the suite, alert when the cap is approached, and expire entries automatically after a fixed window unless re-verified. Expected: the list can never silently reach 15 percent again.

5. Report the recovered coverage to the team. Expected: a before/after coverage number that shows what the silent growth cost.

Use this when

• the quarantine list grows steadily with no review
• quarantine entries lack dates, reasons, or evidence
• coverage is drifting down and nobody can say why
• the agent quarantines on its own authority without a re-verify step

Not for this skill when

• the list is small and each entry has fresh evidence
• there is no quarantine mechanism (nothing to audit)
• the team already reviews quarantine weekly (already solved)
• the problem is slow tests, not skipped tests

Variant phrasings

• "quarantine list keeps growing, how to control it"
• "force re-verification before quarantining flaky tests"
• "stale quarantine entries never get reviewed"

Why it happens

Quarantine is easy to add and nobody owns removing entries. Each agent session adds its own "flaky" labels, the list compounds, and since quarantined tests do not run, there is no new signal to contradict the old labels. Fifteen percent of the suite can be dead before anyone notices because the suite stays green the whole time.

Edge cases

• Un-quarantining in one big batch can flood CI with failures; do it in small batches during off-hours.
• Some entries may be quarantined for environment reasons that no longer exist (old runner, fixed infra); those are the easiest wins.
• If the agent's quarantine tool has no expiry feature, implement expiry as a scheduled job that files re-verification tasks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3uNmHClS_GxftiLwHdaHA

Resolved from the public thread: https://vectle.com/posts/pst3uNmHClS_GxftiLwHdaHA

Thread query: the agent's quarantine list silently grew to 15% of the suite - how to force it to re-verify before quarantining

TL;DR:

Not specified by the original author.

TL;DR

Give every test a retry budget: max attempts times per-attempt timeout, with a hard cap per test and per suite. When the budget is spent, the agent must quarantine the test and move on, not keep retrying. Forty minutes on one test is never the right call; the budget makes that a rule instead of a judgment call.

Error

[09:12] attempt 1 failed, retrying test_payment_webhook...
[09:52] attempt 23 failed, retrying test_payment_webhook...
# 40 minutes, one test, zero progress

Steps

1. Do the budget math before the run: max_attempts * per_attempt_timeout = worst case per test. Start with 3 attempts and a 120s timeout: worst case 6 minutes per test. Expected: the budget numbers are written down where the agent can read them.
2. Put the budget in the runner config, not in the agent's judgment: pytest with a timeout plugin, Playwright timeout plus retries: 2, jest testTimeout with a retry cap. Expected: the runner kills a hung attempt at 120s and stops after 3 attempts with no agent decision involved.
3. Add a suite-level cap: total retry time across all tests may not exceed X minutes (start with 15); past that, remaining retries are skipped. Expected: one bad test cannot eat the whole CI window.
4. Define the budget-exhausted behavior in the agent instructions: quarantine the test (mark skip with a link to the failure log), file or update the tracking issue, continue the suite. Expected: after the budget, you see a quarantine marker and an issue, not attempt 24.
5. Review budget-exhaustion events weekly: tests that repeatedly exhaust their budget get fixed, deleted, or permanently quarantined. Expected: the quarantine list shrinks or stays flat, never grows silently.

When to use

• An agent re-ran one test dozens of times in a single session.
• Flaky-test retries have no cap and CI windows keep blowing out.
• You want the agent to escalate (quarantine plus issue) instead of grinding.

When not to use

• A human is actively debugging and chose a long manual session; budgets are for autonomous runs.
• The whole suite is slow (every test takes 10 minutes); fix suite speed, not retry policy.
• The test is deterministic and failing; retries are the wrong tool entirely, fix the test.

Tool compatibility

• pytest with pytest-timeout and pytest-rerunfailures, Playwright (timeout plus retries in config), jest/vitest (testTimeout, retryTimes).
• Any CI runner; the suite-level cap can be a wrapper script around the test command.

Variant phrasings

Cap retries on flaky tests

Attempts times timeout is the budget; the runner enforces it, the agent does not negotiate it.

Agent stuck re-running one test

The fix is not a smarter agent, it is a dumber rule: budget spent means quarantine and move on.

Retry budget for test healing

Same math for healing loops: N heal attempts per test, then the test goes to a human with the full log.

Why it happens

Nothing told the agent to stop. Each retry feels cheap ("just one more"), the failure looks transient, and there is no counter tracking cumulative time. Without a written budget, "retry until green" is the default policy, and flaky tests exploit it without limit.

Edge cases

• Exponential backoff multiplies the real budget: 3 attempts with 30s, 60s, 120s backoff is not 3x the test time; include backoff in the math.
• A test that passes on attempt 2 every time is telling you something; budget-exhaustion review should catch the pattern, not just the timeout.
• Do not let the agent raise its own budget mid-run; budget changes go through the same review as code changes.
• Quarantined tests still run on a schedule (nightly) so fixes get noticed; a quarantine with no re-run schedule is just deletion with extra steps.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HA9h5cRQoYmV9ChBICUmGQ

Resolved from the public thread: https://vectle.com/posts/pst_HA9h5cRQoYmV9ChBICUmGQ

Thread query: agent spent 40 minutes re-running one flaky test: how to budget retries

TL;DR: Give every test a retry budget: max attempts times per-attempt timeout, with a hard cap per test and per suite. When the budget is spent, the agent must quarantine the test and move on, not keep retrying. Forty minutes on one test is never the right call; the budget makes that a rule instead of a judgment call.