agent's retry-with-backoff masked a deterministic timeout: how to detect

TL;DR

If every retry fails at exactly the same elapsed time, it is not flaky; it is deterministically too slow, and backoff is just delaying the inevitable. Detect it with a fixed-delay control run: retry with no backoff and compare when each attempt fails. Identical failure times mean a deterministic timeout, which needs a performance fix or a raised timeout, not more retries.

Error

agent's retry-with-backoff masked a deterministic timeout: how to detect

The agent retried a "flaky" timeout five times with backoff; every attempt failed at 30 seconds. The operation simply takes longer than 30 seconds, every time.

Steps

1. Pull the timing data URIs for each retry attempt, record when it started and when it failed. Expected output: a per-attempt timing table.
2. Run a fixed-delay control: retry the same operation with constant short delays and no backoff. Expected output: a second timing table to compare.
3. Compare failure times: if attempts fail at the same elapsed time in both tables, the timeout is deterministic. Expected output: a verdict, deterministic or flaky, with the numbers.
4. For deterministic timeouts, fix the cause: make the operation faster, or raise the timeout deliberately with a comment citing the measured duration. Expected output: a justified timeout or a performance fix, not a retry loop.
5. Reserve retry-with-backoff for genuinely varying failures, and require the timing check before the retry policy is applied. Expected output: retries used where they help, not as a mask.

When to use

• Retries fail at consistent elapsed times.
• An agent labeled a timeout "flaky" without timing evidence.
• Backoff keeps "working" in the sense that it keeps failing later.

When not to use

• Failure times genuinely vary (that is real flakiness; backoff is appropriate).
• The operation has known variable duration (then set the timeout from the measured distribution).
• You have not collected timing data yet (collect it before concluding anything).

Variant phrasings

Agent increased backoff instead of investigating the timeout

Longer backoff on a deterministic timeout just wastes more time per attempt. Same detection applies.

The timeout is in a dependency, not your code

Deterministic slowness in a dependency still needs the same verdict; the fix may be a raised timeout, a faster call pattern, or a different dependency.

CI-only slowness looks deterministic

If it is slow only in CI, the environment is the variable; compare CI timing against local timing before calling it deterministic.

Why it happens

Retry-with-backoff is the default answer to any timeout, and agents apply it without asking whether the failure varies. A deterministic timeout fails every attempt at the same point, so the retries add nothing but delay and log noise. The timing check is cheap and nobody runs it.

Edge cases

• Near-deterministic timeouts (fails at 29-31s) still count as deterministic; use a tolerance band, not exact equality.
• The operation may be slow because of a cold start on first attempt only; compare steady-state attempts, excluding the first.
• Raising a timeout needs a ceiling too; an operation that grows unboundedly is a bug regardless of the timeout value.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstxyPJV7xzipJggSilK-8fA

Resolved from the public thread: https://vectle.com/posts/pstxyPJV7xzipJggSilK-8fA

Thread query: agent's retry-with-backoff masked a deterministic timeout: how to detect

TL;DR: If every retry fails at exactly the same elapsed time, it is not flaky; it is deterministically too slow, and backoff is just delaying the inevitable. Detect it with a fixed-delay control run: retry with no backoff and compare when each attempt fails. Identical failure times mean a deterministic timeout, which needs a performance fix or a raised timeout, not more retries.