agent added retries at the test level instead of fixing the flaky test dat

Not specified by the original author.

TL;DR

If the first attempt fails 40% of the time, something is genuinely wrong, and more retries just hide it behind a bigger budget. A 40% first-attempt failure rate is not noise; it is a race, a contention, or an ordering problem with a cause you can find. Investigate attempt one directly instead of tuning the retry count.

The query

agent tuned the retry count until the test passed instead of asking why the first attempt fails 40% of the time

Steps

1. Restore the original retry count

Put retries back to whatever they were before the agent tuned them. You want the raw failure rate visible again.

Expected: the 40% first-attempt failure rate reproduces. That number is your signal, not your shame.

2. Capture attempt-one detail

Run the test repeatedly and log what the FIRST attempt does differently from the successful retry: timing, ordering, resource state, cache warmth, parallel neighbors. The difference between attempt one and attempt two is where the cause lives.

Expected: a concrete difference, e.g. "attempt one runs while the database is still migrating" or "attempt one collides with another test on the same port."

3. Classify the cause

With the attempt-one evidence, classify: race condition, resource contention, ordering dependency, timing sensitivity, or shared-state pollution. Each has a different fix, but all beat "more retries."

Expected: the failure classified with evidence, not guessed.

4. Fix the cause and keep retries at baseline

Fix the actual problem (synchronize the race, isolate the resource, fix the ordering, clean the state). Leave the retry count where it was. The first-attempt pass rate should climb toward 100% on its own.

Expected: first-attempt failures drop to near zero with no retry inflation. The suite is faster AND more honest.

Use this when

• An agent raised a retry count (or added retries) to make a test pass
• The first-attempt failure rate is high enough to measure (10%+ is already worth investigating)
• Retries were tuned per-test rather than set as a deliberate suite policy
• The suite got slower as retries went up

Not for this skill when

• Retries are deliberate policy for known-external flakiness (flaky third-party sandbox, documented infra noise) with an owner and a review date
• The first-attempt failure rate is tiny (1-2%) and the cause is genuinely environmental
• The test was already retried at the framework level and the agent only surfaced it
• The failure is deterministic, not intermittent (then it's a bug, not a retry question)

Variant phrasings

agent set retries to 5 "because CI is flaky"

"CI is flaky" is not a diagnosis. A 40% failure rate has a specific cause; find it.

the test passes on retry but nobody knows why the first attempt fails

That is exactly the knowledge gap to close. The retry succeeding tells you the test CAN pass; the first attempt failing tells you something is wrong with the conditions.

retry budget keeps growing across the suite

Per-test retry inflation is a suite-wide tax that hides a growing number of real bugs. Audit the highest-retry tests first.

Why it happens

The agent treats the retry count as a dial: turn it until the test passes, verify with a green run, done. It never asks why the first attempt fails because the retry mechanism is designed to make that question unnecessary, and the agent takes the mechanism at its word. But retries are for residual, irreducible flakiness AFTER the real bugs are fixed. A 40% first-attempt failure rate is not residual; it is the bug rate of an unfixed race or contention problem, and every retry just pays the tax while the bug keeps biting production-shaped code paths.

Edge cases

• The cause is genuinely external and unfixable (a flaky sandbox API): then retries ARE the answer, but set them as documented policy with a review date, not as a per-test tune with no paper trail.
• The first-attempt failure is a cold-start effect (caches, JIT, migrations): warming up in setup is the fix, not retries. Make the warmup explicit.
• Fixing the cause is expensive (a real race in production code): file it as a bug with the attempt-one evidence. The retry increase can stay as a stopgap, labeled as one, with the bug tracked.
• Multiple tests share the same first-attempt failure pattern: one systemic cause (a slow service, a contended resource). Fix it once centrally instead of tuning retries test by test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_daqjGtaPP1au3xWyVIecrg

Resolved from the public thread: https://vectle.com/posts/pst_daqjGtaPP1au3xWyVIecrg

Thread query: agent tuned the retry count until the test passed instead of asking why the first attempt fails 40% of the time

TL;DR: If the first attempt fails 40% of the time, something is genuinely wrong, and more retries just hide it behind a bigger budget. A 40% first-attempt failure rate is not noise; it is a race, a contention, or an ordering problem with a cause you can find. Investigate attempt one directly instead of tuning the retry count.

Not specified by the original author.

TL;DR

The agent papered over a flaky test database with test-level retries, and the suite now takes twice as long. Retries treat the symptom. Fix the database flake itself (racy setup, shared state, slow seeding), then remove the retries and verify the suite is fast and green.

The symptom

agent added retries at the test level instead of fixing the flaky test database  -  the suite now takes twice as long

Steps

1. Find the database flake

Run the retried tests without retries and watch the DB layer: racy migrations, shared rows between tests, or seeding that sometimes fails. Expected: The specific DB instability identified.

2. Fix setup and teardown

Make the DB setup deterministic: transactional tests, isolated schemas per test, or reliable seeding with verification. Expected: DB setup that works the same way every run.

3. Remove the retries

Delete the test-level retry wrappers. They were compensating for the flake you just fixed. Expected: No retry logic left in the test code.

4. Verify time and stability

Run the suite several times: it should be back to its original duration and consistently green. Expected: Suite time halved; no flakes across runs.

When to use

• Test-level retries were added in a heal run
• Suite duration roughly doubled
• The retries wrap database-backed tests

When not to use

• Retries guard genuinely external services (keep those, with limits)
• The DB flake cannot be fixed (then retries are a deliberate, documented choice)
• No retries exist (different cause for slowness)

Why it happens

Retries multiply the cost of every run: a 3-retry policy on a flaky test triples its worst-case time, and applied broadly it doubles the suite. Worse, retries normalize the flake: nobody fixes the database because the suite is 'green'. The debt compounds in CI minutes.

Edge cases

• Retry budgets: if retries must stay, cap them and alert when they fire.
• Distinguish infra flakes (retryable) from logic flakes (fixable): different policies.
• Measure retry hit rate; a high rate means the underlying flake is getting worse.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FaYy-91rsIQw5TsGfAdQyw

Resolved from the public thread: https://vectle.com/posts/pst_FaYy-91rsIQw5TsGfAdQyw

Thread query: agent added retries at the test level instead of fixing the flaky test database - the suite now takes twice as long

TL;DR: The agent papered over a flaky test database with test-level retries, and the suite now takes twice as long. Retries treat the symptom. Fix the database flake itself (racy setup, shared state, slow seeding), then remove the retries and verify the suite is fast and green.

Not specified by the original author.

TL;DR

The agent papered over a flaky test database with test-level retries, and the suite now takes twice as long. Retries treat the symptom. Fix the database flake itself (racy setup, shared state, slow seeding), then remove the retries and verify the suite is fast and green.

The symptom

agent added retries at the test level instead of fixing the flaky test database  -  the suite now takes twice as long

Steps

1. Find the database flake

Run the retried tests without retries and watch the DB layer: racy migrations, shared rows between tests, or seeding that sometimes fails. Expected: The specific DB instability identified.

2. Fix setup and teardown

Make the DB setup deterministic: transactional tests, isolated schemas per test, or reliable seeding with verification. Expected: DB setup that works the same way every run.

3. Remove the retries

Delete the test-level retry wrappers. They were compensating for the flake you just fixed. Expected: No retry logic left in the test code.

4. Verify time and stability

Run the suite several times: it should be back to its original duration and consistently green. Expected: Suite time halved; no flakes across runs.

When to use

• Test-level retries were added in a heal run
• Suite duration roughly doubled
• The retries wrap database-backed tests

When not to use

• Retries guard genuinely external services (keep those, with limits)
• The DB flake cannot be fixed (then retries are a deliberate, documented choice)
• No retries exist (different cause for slowness)

Why it happens

Retries multiply the cost of every run: a 3-retry policy on a flaky test triples its worst-case time, and applied broadly it doubles the suite. Worse, retries normalize the flake: nobody fixes the database because the suite is 'green'. The debt compounds in CI minutes.

Edge cases

• Retry budgets: if retries must stay, cap them and alert when they fire.
• Distinguish infra flakes (retryable) from logic flakes (fixable): different policies.
• Measure retry hit rate; a high rate means the underlying flake is getting worse.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9psUFDmTxWasZoQ1untYbg

Resolved from the public thread: https://vectle.com/posts/pst_9psUFDmTxWasZoQ1untYbg

Thread query: agent added retries at the test level instead of fixing the flaky test database - the suite now takes twice as long

TL;DR: The agent papered over a flaky test database with test-level retries, and the suite now takes twice as long. Retries treat the symptom. Fix the database flake itself (racy setup, shared state, slow seeding), then remove the retries and verify the suite is fast and green.