agent looped re-triggering a failed dagster asset materialization

TL;DR

A materialization that fails deterministically, bad code, bad config, missing upstream data, will fail identically forever, and an agent that relaunches on every failure observation manufactures a run queue storm and buries the original error. Break the loop: read the failed run's step logs, classify the failure as deterministic or transient, fix deterministic causes before any new launch, and give the agent a hard retry budget keyed to the asset and partition, with a check that a run is not already in flight.

agent looped re-triggering a failed dagster asset materialization

Use this when

• The runs list shows many failed runs for the same asset and partition in a short window.
• Each failure has the same error and the same stack trace.
• Downstream assets are starved or the run queue is full of retries.

Not for this skill when

• Dagster's own retry policy is handling a known transient source outage within its configured attempts. That is bounded by design.
• A human is deliberately rerunning after each fix attempt. The loop has a decision maker in it.

Steps

1. Stop further launches first, by pausing the agent or the automation doing the relaunching, and count the failed runs for the asset and partition. Note the count and the time span. Verify: the loop is stopped before diagnosis, so the evidence stops growing
2. Read the earliest failed run's step logs, not the latest, and note the root exception and whether any input changed between runs. Note the failure class: deterministic or transient. Verify: the original cause is identified without retry noise
3. If deterministic, fix the cause: the asset code, a missing config value, or the absent upstream partition. Verify the fix in a single manual materialization launched by a human decision. Note the green run id. Verify: one verified fix replaces a pile of blind retries
4. If transient, define the retry policy explicitly: maximum attempts, a backoff that grows, and a stop condition that reports a human readable failure instead of relaunching. Note the policy values in the agent's configuration. Verify: transient retries are bounded and visible
5. Add the loop guards to the agent: before launching, check for an in-progress run for the same asset and partition and for the number of failed attempts in the window, and refuse to launch past the budget. Then re-enable automation and watch one cycle. Note the guard triggers correctly on a test failure. Verify: the agent cannot recreate the storm even if its reasoning slips

Variant phrasings

agent keeps relaunching failed dagster run retry loop stop

The loop breaking phrasing.

dagster asset materialization fails same error repeated runs

The repeated failure symptom search.

Compatibility: Dagster 1.x runs launched through the GraphQL API, the UI, or automation conditions, supervised by an external agent. Run state queries used for the guards are standard Dagster API reads.

Why it happens

Agents relaunch because relaunching is the one action that sometimes works, and without a model of the failure, every observation of a failed state looks like a fresh invitation. Deterministic failures are the majority in data pipelines, since the code and the data that broke the run are still there minutes later. The system level damage compounds the original fault: queued retries consume concurrency, spam alerting, and can hammer the same struggling source the first failure was about. Budgets and in-flight checks convert an reasoning problem into a mechanical impossibility.

Edge cases / pitfalls

• Cancelling a flood of queued retry runs needs care: cancel the duplicates, keep the earliest or the one carrying a fix attempt, and record what you cancelled.
• A partition that fails while its neighbours succeed points at partition specific data, not global code. The fix and the budget should be per partition.
• Automation conditions inside Dagster can also relaunch. Check whether the loop is the agent, a Dagster automation, or both feeding each other.
• Alert on the loop pattern itself, repeated launches per asset per hour, so the next loop is caught even with a new error.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6BFNjpoTVQW7sm-jFL4rLw