agent retry loop re-ran the full dbt build 8 times on a transient warehouse error

TL;DR

The agent treated a transient warehouse blip as a code problem and retried the whole build instead of waiting and retrying narrowly. Kill the loop, wait for the warehouse to recover, then retry only the failed nodes with dbt retry or a narrow --select. Cap agent retries and require backoff.

Error

agent retry loop re-ran the full dbt build 8 times on a transient warehouse error

Steps

1. Stop the loop and check the warehouse status: was it a real transient (failover, throttling, maintenance)? Confirm it is over before doing anything. Expected: You know the blip is past, or still ongoing (then keep waiting).

2. Assess the damage: 8 full builds may have duplicated incremental loads or burned serious spend. Check incremental models for double counts and the warehouse bill. Expected: You know what the retries cost and whether data was affected.

3. Recover properly: fix any double-counted incrementals (see the double-count skill), then run dbt retry once, or dbt build --select with the failed nodes only. Expected: One clean targeted retry instead of eight full ones.

4. Cap the agent: max 2-3 retries, exponential backoff, retry failed nodes only, and page a human if the warehouse is still erroring. Expected: The next transient gets a calm retry, not a stampede.

When to use

• an agent re-ran full dbt builds many times on a transient error
• warehouse spend spiked from repeated builds
• incremental models double-counted after agent retries

When not to use

• dbt retry was used correctly on failed nodes (that is the right tool)
• the error is persistent, not transient (then retries never help; fix the cause)
• a human ran the retries deliberately (coach the human)

Tool compatibility

• dbt retry for failed-nodes-only reruns
• warehouse status pages to confirm transients
• agent retry caps and backoff policies

Variant phrasings

agent retried on a persistent error 8 times

Worse: the error never had a chance. Same cap policy, plus the agent must classify transient vs persistent before retrying.

loop re-ran dbt test, not build

Cheaper but still wrong on a warehouse outage. Same fix: wait, then retry narrowly.

Why it happens

Agents lack the judgment to distinguish a warehouse blip from a code bug, and their default repair loop is try again. Without a retry budget and backoff, try again becomes a very expensive while loop, and non-idempotent models corrupt data along the way.

Edge cases

• Check idempotency of every model before any retry strategy; non-idempotent models make retries dangerous by design.
• Log every agent retry with the error class so you can audit whether the cap policy holds.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cdvdoaSkEKNzhr7JqwmX-A