agent spent 40 minutes re-running one flaky test: how to budget retries

TL;DR

Give every test a retry budget: max attempts times per-attempt timeout, with a hard cap per test and per suite. When the budget is spent, the agent must quarantine the test and move on, not keep retrying. Forty minutes on one test is never the right call; the budget makes that a rule instead of a judgment call.

Error

[09:12] attempt 1 failed, retrying test_payment_webhook...
[09:52] attempt 23 failed, retrying test_payment_webhook...
# 40 minutes, one test, zero progress

Steps

1. Do the budget math before the run: max_attempts * per_attempt_timeout = worst case per test. Start with 3 attempts and a 120s timeout: worst case 6 minutes per test. Expected: the budget numbers are written down where the agent can read them.
2. Put the budget in the runner config, not in the agent's judgment: pytest with a timeout plugin, Playwright timeout plus retries: 2, jest testTimeout with a retry cap. Expected: the runner kills a hung attempt at 120s and stops after 3 attempts with no agent decision involved.
3. Add a suite-level cap: total retry time across all tests may not exceed X minutes (start with 15); past that, remaining retries are skipped. Expected: one bad test cannot eat the whole CI window.
4. Define the budget-exhausted behavior in the agent instructions: quarantine the test (mark skip with a link to the failure log), file or update the tracking issue, continue the suite. Expected: after the budget, you see a quarantine marker and an issue, not attempt 24.
5. Review budget-exhaustion events weekly: tests that repeatedly exhaust their budget get fixed, deleted, or permanently quarantined. Expected: the quarantine list shrinks or stays flat, never grows silently.

When to use

• An agent re-ran one test dozens of times in a single session.
• Flaky-test retries have no cap and CI windows keep blowing out.
• You want the agent to escalate (quarantine plus issue) instead of grinding.

When not to use

• A human is actively debugging and chose a long manual session; budgets are for autonomous runs.
• The whole suite is slow (every test takes 10 minutes); fix suite speed, not retry policy.
• The test is deterministic and failing; retries are the wrong tool entirely, fix the test.

Tool compatibility

• pytest with pytest-timeout and pytest-rerunfailures, Playwright (timeout plus retries in config), jest/vitest (testTimeout, retryTimes).
• Any CI runner; the suite-level cap can be a wrapper script around the test command.

Variant phrasings

Cap retries on flaky tests

Attempts times timeout is the budget; the runner enforces it, the agent does not negotiate it.

Agent stuck re-running one test

The fix is not a smarter agent, it is a dumber rule: budget spent means quarantine and move on.

Retry budget for test healing

Same math for healing loops: N heal attempts per test, then the test goes to a human with the full log.

Why it happens

Nothing told the agent to stop. Each retry feels cheap ("just one more"), the failure looks transient, and there is no counter tracking cumulative time. Without a written budget, "retry until green" is the default policy, and flaky tests exploit it without limit.

Edge cases

• Exponential backoff multiplies the real budget: 3 attempts with 30s, 60s, 120s backoff is not 3x the test time; include backoff in the math.
• A test that passes on attempt 2 every time is telling you something; budget-exhaustion review should catch the pattern, not just the timeout.
• Do not let the agent raise its own budget mid-run; budget changes go through the same review as code changes.
• Quarantined tests still run on a schedule (nightly) so fixes get noticed; a quarantine with no re-run schedule is just deletion with extra steps.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HA9h5cRQoYmV9ChBICUmGQ