agent looped dbt test on the same failing test 12 times with no changes

TL;DR

Retesting unchanged code against unchanged data cannot change the result, so a loop of 12 identical dbt test runs is pure waste. Break the loop with a change-or-stop rule: hash the relevant files and the test definition before each run, and if nothing changed since the last failure, stop and escalate with the failing rows instead of running again. Retest only after an edit or a data refresh.

agent looped dbt test on the same failing test 12 times with no changes

Use this when

• agent logs show repeated dbt test invocations on the same failing test with no edits between them
• the loop burned time or warehouse credits without new information
• each run returns the identical failing row count

Not for this skill when

• the data is actively loading, so results legitimately change between runs; coordinate with the load instead of capping retries
• a human is deliberately soak-testing for flakiness, that needs a defined sample size and capture of failing rows
• the test is cheap and the loop is a orchestrator retry with backoff on infrastructure errors, distinguish error types first

Steps

1. Inspect the agent loop: identify what triggers each retry and whether any state (files, data, config) changes between attempts. Log a hash of the model, test definition, and relevant config per attempt. Verify: you can prove attempts are identical, or find the one input that actually varies.

2. Add a stop rule to the agent: after two identical failures with identical input hashes, stop retrying, capture store_failures output and the test SQL, and escalate to a human or a different strategy. Verify: the agent halts after the second identical failure with evidence attached.

3. Require a change token before any retest: a file edit, a config change, or an explicit data-refresh signal must occur between test runs; otherwise skip the run and report why. Verify: retests only happen after a recorded change.

4. Replay the incident inputs through the fixed loop in a sandbox project or dry run and confirm it stops and reports instead of looping. Verify: the fixed agent stops with a clear failure report on the same inputs.

Variant phrasings

loop alternates between two tests

The agent may be fixing one test by breaking another. Hash and report the whole affected model test set, not one test at a time.

loop caused by orchestrator retries

If Airflow or CI retries the whole step on any failure, set retries to zero for test steps and let the agent own the decision.

Compatibility: Agent harness design around dbt test; store_failures test config (dbt 1.3 and later) for capturing failing rows. Applies to any automation invoking dbt in a loop.

Why it happens

A deterministic test on fixed inputs returns the same result forever. Agents loop because their retry logic treats failure as transient by default and because nothing in a bare dbt test invocation tells them the inputs are unchanged. Without an explicit change signal and a stop rule, the loop is the rational output of the harness design, not bad luck.

Edge cases / pitfalls

• Warehouse-side changes (someone fixing data manually) are real change signals too; allow an explicit human resume instead of a permanent block.
• Identical failing counts can still hide changing rows; compare failing row samples, not counts only.
• Cap total loop spend as a backstop metric even with stop rules, and alert on the cap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vRdGNl1oNYJEmTbM7Ta-TQ