agent loop detected: re-triggering failed dag run over and over
Stops agents from re-triggering failed DAG runs in a loop: recognizing deterministic failure and switching to diagnosis. Use when an agent keeps retriggering the same failing run. Not for scheduled runs failing on their own.
TL;DR
Re-triggering a deterministically failing DAG run is the definition of insanity with extra steps: the same code on the same data fails the same way. After two identical failures, stop triggering and read the failure. Fix the cause, then trigger once to verify. Triggers are for new attempts after a fix, not for hoping.
agent loop detected: re-triggering failed dag run over and overUse this when
- An agent re-triggers a failed DAG run repeatedly.
- Each attempt fails the same way.
- No fix was applied between attempts.
Not for this skill when
- A fix was applied between attempts. That is verification, not a loop.
- Different runs fail differently. That is flakiness, not determinism.
Steps
- Compare the failure across attempts: same task, same error. Note the match. Verify: determinism is proven
- Stop triggering. Note the halt. Verify: the loop breaks
- Read the failing task's logs and identify the root cause. Note the cause. Verify: the diagnosis replaces the hoping
- Apply the fix: code, config, or data. Note the change. Verify: something actually changes
- Trigger once and confirm the run succeeds. Note the green run. Verify: the fix is verified with one trigger
Variant phrasings
agent retrying failed dag run loop
The paraphrase.
stop re-triggering airflow dag
The fix phrasing.
Compatibility: Airflow 2.x DAG run triggering via UI, API, or CLI. Loop-breaking is agent behavior.
Why it happens
Re-triggering feels like doing something, which is why agents do it: the action is available, the failure is unexplained, and trying again is the path of least resistance. But a deterministic failure has no randomness to overcome, so each retry is pure cost: compute, time, and possibly duplicated side effects. The two-identical-failures rule is the circuit breaker. It forces the agent to spend its budget on the one thing that can help: understanding the failure.
Edge cases / pitfalls
- Re-triggering can double-apply non-idempotent tasks. The loop is not just wasteful; it can corrupt data.
- A run that fails differently each time may be flaky infrastructure, where retries are legitimate. Distinguish first.
- Clearing and re-running a single failed task is cheaper than re-triggering the whole run. Prefer the smaller blast radius.
- If the agent lacks permission to fix the cause, escalate after the second identical failure instead of triggering a third.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstIgvvdYkHuF-8CHkRH0tDw