agent added retries at the test level instead of fixing the flaky test database - the suite now takes twice as long
Fixes retry band-aids that double suite time instead of fixing the flaky fixture. Use when test-level retries were added over an unstable test database. Covers finding the DB flake, fixing setup/teardown, and removing the retries. Not for retries around genuinely external services.
TL;DR
The agent papered over a flaky test database with test-level retries, and the suite now takes twice as long. Retries treat the symptom. Fix the database flake itself (racy setup, shared state, slow seeding), then remove the retries and verify the suite is fast and green.
The symptom
agent added retries at the test level instead of fixing the flaky test database - the suite now takes twice as longSteps
1. Find the database flake
Run the retried tests without retries and watch the DB layer: racy migrations, shared rows between tests, or seeding that sometimes fails. Expected: The specific DB instability identified.
2. Fix setup and teardown
Make the DB setup deterministic: transactional tests, isolated schemas per test, or reliable seeding with verification. Expected: DB setup that works the same way every run.
3. Remove the retries
Delete the test-level retry wrappers. They were compensating for the flake you just fixed. Expected: No retry logic left in the test code.
4. Verify time and stability
Run the suite several times: it should be back to its original duration and consistently green. Expected: Suite time halved; no flakes across runs.
When to use
- Test-level retries were added in a heal run
- Suite duration roughly doubled
- The retries wrap database-backed tests
When not to use
- Retries guard genuinely external services (keep those, with limits)
- The DB flake cannot be fixed (then retries are a deliberate, documented choice)
- No retries exist (different cause for slowness)
Why it happens
Retries multiply the cost of every run: a 3-retry policy on a flaky test triples its worst-case time, and applied broadly it doubles the suite. Worse, retries normalize the flake: nobody fixes the database because the suite is 'green'. The debt compounds in CI minutes.
Edge cases
- Retry budgets: if retries must stay, cap them and alert when they fire.
- Distinguish infra flakes (retryable) from logic flakes (fixable): different policies.
- Measure retry hit rate; a high rate means the underlying flake is getting worse.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_9psUFDmTxWasZoQ1untYbg