usage-warehouse query timed out mid-scoring: agent got stuck retrying with no backoff

TL;DR

A timeout followed by instant identical retries is worse than no retry: the warehouse is signalling overload and the agent multiplies the load. Add exponential backoff with jitter, shrink the query's cost (narrower window, pre-aggregation), and open a circuit breaker after repeated failures so the agent waits for recovery instead of digging the hole.

usage-warehouse query timed out mid-scoring: agent got stuck retrying with no backoff

Use this when

• Logs show the same query timing out and retrying immediately many times
• Warehouse load spikes correlate with the agent's retries
• The run eventually fails anyway after the retry storm

Not for this skill when

• The warehouse had a genuine outage and the platform team owns recovery
• The query is malformed and errors instantly rather than timing out

Steps

1. Extract the retry pattern from logs: attempts, intervals, and total added load. Verify: the storm is characterised with numbers
2. Implement exponential backoff with jitter and a hard attempt cap for warehouse calls. Verify: a forced timeout in testing now backs off visibly and stops
3. Reduce the query's cost: bound the date window, aggregate in SQL, or read a pre-built usage table. Verify: the retried query is cheaper than the original by an order of magnitude
4. Add a circuit breaker: after consecutive failures, pause scoring and alert instead of retrying. Verify: the breaker trips in a failure drill and notifies a human
5. Resume with a single probe query before reopening the full run. Verify: recovery is verified by the probe, not assumed

Variant phrasings

agent retry storm warehouse timeout

A related phrasing for the same underlying problem.

no backoff on timed out query

A related phrasing for the same underlying problem.

circuit breaker for scoring agent

A related phrasing for the same underlying problem.

Compatibility: Compatibility: agent tool layers over Snowflake, BigQuery, or Redshift with configurable retry policies, current versions.

Why it happens

Retries are a bet that the failure is transient; instant identical retries bet the same way at maximum cost. Backoff, cheaper queries, and a breaker align the agent's behaviour with the warehouse's reality: overload clears when load stops, not when it quintuples.

Edge cases / pitfalls

• Backoff parameters belong in configuration, not code, so operations can tune them during an incident.
• A circuit breaker needs a half-open probe state, or the agent never learns the warehouse recovered.
• Log every retry with its attempt number and delay; without that, the next storm is undiagnosable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_maNXDt32aIunm6Nao17rYQ