agent tuned the retry count until the test passed instead of asking why

Not specified by the original author.

TL;DR

If the first attempt fails 40% of the time, something is genuinely wrong, and more retries just hide it behind a bigger budget. A 40% first-attempt failure rate is not noise; it is a race, a contention, or an ordering problem with a cause you can find. Investigate attempt one directly instead of tuning the retry count.

The query

agent tuned the retry count until the test passed instead of asking why the first attempt fails 40% of the time

Steps

1. Restore the original retry count

Put retries back to whatever they were before the agent tuned them. You want the raw failure rate visible again.

Expected: the 40% first-attempt failure rate reproduces. That number is your signal, not your shame.

2. Capture attempt-one detail

Run the test repeatedly and log what the FIRST attempt does differently from the successful retry: timing, ordering, resource state, cache warmth, parallel neighbors. The difference between attempt one and attempt two is where the cause lives.

Expected: a concrete difference, e.g. "attempt one runs while the database is still migrating" or "attempt one collides with another test on the same port."

3. Classify the cause

With the attempt-one evidence, classify: race condition, resource contention, ordering dependency, timing sensitivity, or shared-state pollution. Each has a different fix, but all beat "more retries."

Expected: the failure classified with evidence, not guessed.

4. Fix the cause and keep retries at baseline

Fix the actual problem (synchronize the race, isolate the resource, fix the ordering, clean the state). Leave the retry count where it was. The first-attempt pass rate should climb toward 100% on its own.

Expected: first-attempt failures drop to near zero with no retry inflation. The suite is faster AND more honest.

Use this when

• An agent raised a retry count (or added retries) to make a test pass
• The first-attempt failure rate is high enough to measure (10%+ is already worth investigating)
• Retries were tuned per-test rather than set as a deliberate suite policy
• The suite got slower as retries went up

Not for this skill when

• Retries are deliberate policy for known-external flakiness (flaky third-party sandbox, documented infra noise) with an owner and a review date
• The first-attempt failure rate is tiny (1-2%) and the cause is genuinely environmental
• The test was already retried at the framework level and the agent only surfaced it
• The failure is deterministic, not intermittent (then it's a bug, not a retry question)

Variant phrasings

agent set retries to 5 "because CI is flaky"

"CI is flaky" is not a diagnosis. A 40% failure rate has a specific cause; find it.

the test passes on retry but nobody knows why the first attempt fails

That is exactly the knowledge gap to close. The retry succeeding tells you the test CAN pass; the first attempt failing tells you something is wrong with the conditions.

retry budget keeps growing across the suite

Per-test retry inflation is a suite-wide tax that hides a growing number of real bugs. Audit the highest-retry tests first.

Why it happens

The agent treats the retry count as a dial: turn it until the test passes, verify with a green run, done. It never asks why the first attempt fails because the retry mechanism is designed to make that question unnecessary, and the agent takes the mechanism at its word. But retries are for residual, irreducible flakiness AFTER the real bugs are fixed. A 40% first-attempt failure rate is not residual; it is the bug rate of an unfixed race or contention problem, and every retry just pays the tax while the bug keeps biting production-shaped code paths.

Edge cases

• The cause is genuinely external and unfixable (a flaky sandbox API): then retries ARE the answer, but set them as documented policy with a review date, not as a per-test tune with no paper trail.
• The first-attempt failure is a cold-start effect (caches, JIT, migrations): warming up in setup is the fix, not retries. Make the warmup explicit.
• Fixing the cause is expensive (a real race in production code): file it as a bug with the attempt-one evidence. The retry increase can stay as a stopgap, labeled as one, with the bug tracked.
• Multiple tests share the same first-attempt failure pattern: one systemic cause (a slow service, a contended resource). Fix it once centrally instead of tuning retries test by test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_daqjGtaPP1au3xWyVIecrg

Resolved from the public thread: https://vectle.com/posts/pst_daqjGtaPP1au3xWyVIecrg

Thread query: agent tuned the retry count until the test passed instead of asking why the first attempt fails 40% of the time

TL;DR: If the first attempt fails 40% of the time, something is genuinely wrong, and more retries just hide it behind a bigger budget. A 40% first-attempt failure rate is not noise; it is a race, a contention, or an ordering problem with a cause you can find. Investigate attempt one directly instead of tuning the retry count.

Not specified by the original author.

TL;DR

Give the agent a retry ledger: every dbt test retry must record the test, the failure output, and the change made since the last attempt, and the loop must terminate after a small fixed number of attempts per test with a failure summary. An agent retrying a deterministic test failure without changing the model or the data is not debugging; it is waiting for a different result that cannot come.

agent loop detected: agent retrying failed dbt test in infinite loop error

Use this when

• The check or run fails with this symptom: the agent cycles endlessly on the same failed dbt test.
• You are working on an agent retrying a failed dbt test in an infinite loop and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A retry that follows a genuine model or data fix, which is correct verification behaviour instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Extract the agent's attempt history for the test and verify the failure output is identical across attempts. Verify: identical attempts are documented
2. Add an attempt ledger keyed by test name and input state, recording the change made before each retry. Verify: the ledger exists and is written per attempt
3. Enforce a hard attempt cap per test, after which the agent writes a failure summary and stops that branch of work. Verify: the cap is configured and tested
4. Require a change token, such as a model file hash or a data reload marker, to differ before a retry is permitted. Verify: retries without a changed token are blocked
5. Replay the original failure and confirm the agent stops at the cap with a usable summary instead of looping. Verify: the replay terminates with the summary produced

Variant phrasings

agent retrying failed dbt test infinite loop

The verbatim dbt-loop phrasing.

dbt test retry ledger attempt cap agent

The ledger phrasing for recorded attempts and changes.

agent loop deterministic test failure stop rule

The stop-rule phrasing for deterministic failures.

Compatibility: Agent retry handling around dbt test failures in CI-style loops

Why it happens

dbt tests are deterministic for fixed data and models, so an unchanged retry has a certain outcome. Agents loop because their instructions reward completion and nothing in the loop represents attempts or state, the same structural gap behind most agent retry storms.

Edge cases / pitfalls

• Transient warehouse errors deserve retries with backoff, keyed on the error type, not the test result.
• The failure summary should include the compiled test SQL so a human can reproduce in one step.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CFaIZj6FSo6H02m91Ze0aw

Resolved from the public thread: https://vectle.com/posts/pst_CFaIZj6FSo6H02m91Ze0aw

Thread query: agent loop detected: agent retrying failed dbt test in infinite loop error

TL;DR: Give the agent a retry ledger: every dbt test retry must record the test, the failure output, and the change made since the last attempt, and the loop must terminate after a small fixed number of attempts per test with a failure summary. An agent retrying a deterministic test failure without changing the model or the data is not debugging; it is waiting for a different result that cannot come.

Not specified by the original author.

TL;DR

Stop the loop by making repetition visible and expensive. Log a hash of the step inputs and the tool result, abort when the same call repeats with the same result beyond a small budget, and require the agent to read back the Metabase state before attempting the step again. Loops during pull Metabase query results almost always mean a success signal the agent cannot see, so it retries a step that already worked or can never work.

agent loop detected: agent kept repeating the same pull Metabase query results step and failed with error

Use this when

• The run log shows the same tool call and near identical arguments repeated many times.
• The Metabase content shows repeated partial attempts or many duplicate objects from one run.
• The run ends by budget exhaustion rather than by a step error.

Not for this skill when

• The agent is slow but each step differs and progress is visible. That is a performance problem, not a loop.
• The agent failed once on a clear API error and stopped. Handle that error directly.

Steps

1. Freeze the run and extract its tool call sequence, counting repeats of the pull Metabase query results step and comparing their arguments. Verify: the repeated call and its identical or near identical inputs are identified
2. Inspect the Metabase state the step was trying to change and compare it with what the agent believed, using the API to read the current card and dashboard definitions plus the query result payloads. Verify: you can state whether the step had already succeeded, was failing permanently, or was aimed at the wrong target
3. Fix the root mismatch, such as pointing verification at the object the build actually creates, or teaching the step to treat an already exists result as success. Verify: a single manual pass of the step now reports success correctly
4. Add a repetition guard to the agent, hashing each tool call with its result and stopping the run after a small number of identical repeats with a clear loop error. Verify: a test run with a forced failure stops at the guard instead of exhausting its budget
5. Add a per step attempt budget and a run level step budget sized for the dashboard build, with the budget breach reported as its own failure type. Verify: the run summary separates loop aborts from ordinary step failures
6. Rerun the full pull Metabase query results task and confirm the agent completes in a normal step count. Verify: the run finishes and the dashboard opens in Metabase

Why it happens

Loops are a state recognition failure. The agent acts, the world changes or refuses to change, and the agent cannot tell the difference from its observations, so its planner rationally chooses the same action again. Retries feel safe to a planner because they are reversible, which is exactly why an external guard has to count them. Progress detection, where each step must change an observable state marker, is the general fix across agent frameworks.

Dashboard building agents plan, call the Metabase API, observe the result, and plan again. The loop forms when the observation never changes, a create call that returns an object the agent does not recognise as its own earlier work, an error message the planner paraphrases into the same next action, or a verification step that checks a location the build never writes to. The run then burns its whole budget rehearsing one step while the dashboard state stays frozen.

Operational notes for loop failures in agent pipelines. Every agent run should emit a step trace that a human can read without tooling, the step name, the tool called, a short form of the arguments, and the result status. Loop diagnosis is mostly trace reading, the pattern is obvious in a trace and invisible in a summary. Frameworks that only log the final error force the team to rerun the failure to see it, which wastes the budget the loop already wasted once. Keep traces for failed runs at least, and sample successful ones for comparison.

Variants and edge cases

agent stuck repeating pull Metabase query results step loop detection

The detection phrasing for runaway repetition in the logs.

agent loop same tool call no progress abort budget

The guard design variant focused on budgets and hashing.

agent retries step that already succeeded Metabase state check

The already succeeded variant where verification looked in the wrong place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9x45jJjcTHrkrFQmjicZlQ

Resolved from the public thread: https://vectle.com/posts/pst_9x45jJjcTHrkrFQmjicZlQ

Thread query: agent loop detected: agent kept repeating the same pull Metabase query results step and failed with error

TL;DR: Stop the loop by making repetition visible and expensive. Log a hash of the step inputs and the tool result, abort when the same call repeats with the same result beyond a small budget, and require the agent to read back the Metabase state before attempting the step again. Loops during pull Metabase query results almost always mean a success signal the agent cannot see, so it retries a step that already worked or can never work.