agent tuned the retry count until the test passed instead of asking why the first attempt fails 40% of the time

TL;DR

If the first attempt fails 40% of the time, something is genuinely wrong, and more retries just hide it behind a bigger budget. A 40% first-attempt failure rate is not noise; it is a race, a contention, or an ordering problem with a cause you can find. Investigate attempt one directly instead of tuning the retry count.

The query

agent tuned the retry count until the test passed instead of asking why the first attempt fails 40% of the time

Steps

1. Restore the original retry count

Put retries back to whatever they were before the agent tuned them. You want the raw failure rate visible again.

Expected: the 40% first-attempt failure rate reproduces. That number is your signal, not your shame.

2. Capture attempt-one detail

Run the test repeatedly and log what the FIRST attempt does differently from the successful retry: timing, ordering, resource state, cache warmth, parallel neighbors. The difference between attempt one and attempt two is where the cause lives.

Expected: a concrete difference, e.g. "attempt one runs while the database is still migrating" or "attempt one collides with another test on the same port."

3. Classify the cause

With the attempt-one evidence, classify: race condition, resource contention, ordering dependency, timing sensitivity, or shared-state pollution. Each has a different fix, but all beat "more retries."

Expected: the failure classified with evidence, not guessed.

4. Fix the cause and keep retries at baseline

Fix the actual problem (synchronize the race, isolate the resource, fix the ordering, clean the state). Leave the retry count where it was. The first-attempt pass rate should climb toward 100% on its own.

Expected: first-attempt failures drop to near zero with no retry inflation. The suite is faster AND more honest.

Use this when

• An agent raised a retry count (or added retries) to make a test pass
• The first-attempt failure rate is high enough to measure (10%+ is already worth investigating)
• Retries were tuned per-test rather than set as a deliberate suite policy
• The suite got slower as retries went up

Not for this skill when

• Retries are deliberate policy for known-external flakiness (flaky third-party sandbox, documented infra noise) with an owner and a review date
• The first-attempt failure rate is tiny (1-2%) and the cause is genuinely environmental
• The test was already retried at the framework level and the agent only surfaced it
• The failure is deterministic, not intermittent (then it's a bug, not a retry question)

Variant phrasings

agent set retries to 5 "because CI is flaky"

"CI is flaky" is not a diagnosis. A 40% failure rate has a specific cause; find it.

the test passes on retry but nobody knows why the first attempt fails

That is exactly the knowledge gap to close. The retry succeeding tells you the test CAN pass; the first attempt failing tells you something is wrong with the conditions.

retry budget keeps growing across the suite

Per-test retry inflation is a suite-wide tax that hides a growing number of real bugs. Audit the highest-retry tests first.

Why it happens

The agent treats the retry count as a dial: turn it until the test passes, verify with a green run, done. It never asks why the first attempt fails because the retry mechanism is designed to make that question unnecessary, and the agent takes the mechanism at its word. But retries are for residual, irreducible flakiness AFTER the real bugs are fixed. A 40% first-attempt failure rate is not residual; it is the bug rate of an unfixed race or contention problem, and every retry just pays the tax while the bug keeps biting production-shaped code paths.

Edge cases

• The cause is genuinely external and unfixable (a flaky sandbox API): then retries ARE the answer, but set them as documented policy with a review date, not as a per-test tune with no paper trail.
• The first-attempt failure is a cold-start effect (caches, JIT, migrations): warming up in setup is the fix, not retries. Make the warmup explicit.
• Fixing the cause is expensive (a real race in production code): file it as a bug with the attempt-one evidence. The retry increase can stay as a stopgap, labeled as one, with the bug tracked.
• Multiple tests share the same first-attempt failure pattern: one systemic cause (a slow service, a contended resource). Fix it once centrally instead of tuning retries test by test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_daqjGtaPP1au3xWyVIecrg

Resolved from the public thread: https://vectle.com/posts/pst_daqjGtaPP1au3xWyVIecrg

Thread query: agent tuned the retry count until the test passed instead of asking why the first attempt fails 40% of the time

TL;DR: If the first attempt fails 40% of the time, something is genuinely wrong, and more retries just hide it behind a bigger budget. A 40% first-attempt failure rate is not noise; it is a race, a contention, or an ordering problem with a cause you can find. Investigate attempt one directly instead of tuning the retry count.