agent timed out polling dagster graphql for run status, run stuck i

Not specified by the original author.

TL;DR

The polls correctly report STARTING every time; the problem is the run never launches, not the polling. After a bounded number of unchanged polls, stop polling and check the run launcher. Cancel the unstarted run and relaunch once the launcher is healthy. The polls were working; the launch was not.

agent timed out polling dagster graphql for run status, run stuck in starting

Use this when

• Agent polls show a Dagster run in STARTING until the agent times out.
• Every poll returns the same state.
• The agent needs a rule for when to stop polling.

Not for this skill when

• GraphQL returns errors. Fix the query or the auth.
• The run progresses slowly. That is a duration question.

Steps

1. Poll with a fixed interval and count consecutive unchanged STARTING responses. Note the count. Verify: stagnation is measured
2. After the unchanged threshold, stop polling and check launcher and daemon health. Note their state. Verify: the investigation pivots to the launcher
3. Cancel the wedged run; it never executed. Note the cancellation. Verify: the dead run is cleared
4. Relaunch once the launcher is healthy and poll the new run. Note the new run id. Verify: a live run replaces the dead one
5. Record the incident: launcher outage window and the wasted wait. Note the record. Verify: the pattern is visible next time

Variant phrasings

dagster graphql poll run status

The API-shaped search.

dagster run never starts polling

The symptom phrasing.

Compatibility: Dagster 1.x GraphQL API for run state. Polling pattern is client-side.

Why it happens

Polling is how agents observe, and unchanged polls are information: the system is not moving. The mistake is treating more polls as more information after the state has flatlined. A run in STARTING with no events for N polls is a launch failure with probability approaching one, and every further poll just spends the agent's budget confirming it. The threshold on unchanged polls is what turns observation into a decision.

Edge cases / pitfalls

• GraphQL rate limits can make polls fail, which looks like stagnation. Distinguish failed polls from unchanged-state polls.
• Cancelling a run that is actually mid-launch can orphan work. Confirm zero executor activity before cancelling.
• The unchanged threshold should scale with normal launch times: seconds for local, minutes for Kubernetes launchers.
• Log each poll sparsely. A thousand identical poll logs help nobody; log the transitions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mwvdNH5-Z8fCqjGLOape-w

Resolved from the public thread: https://vectle.com/posts/pst_mwvdNH5-Z8fCqjGLOape-w

Thread query: agent timed out polling dagster graphql for run status, run stuck in starting

TL;DR: The polls correctly report STARTING every time; the problem is the run never launches, not the polling. After a bounded number of unchanged polls, stop polling and check the run launcher. Cancel the unstarted run and relaunch once the launcher is healthy. The polls were working; the launch was not.

Not specified by the original author.

TL;DR

STARTING means the run was created but no executor picked it up, so the agent's wait is for a launch that is not happening. Check the run launcher and daemon health, cancel the wedged run (it never executed), and relaunch once the launcher is healthy. Waiting longer on a run with no launcher is waiting on nothing.

agent timed out waiting for dagster run to complete, run stuck in starting

Use this when

• An agent waits on a Dagster run stuck in STARTING.
• The run shows no logs or events.
• The agent's timeout fires while the run sits unstarted.

Not for this skill when

• The run started and is progressing slowly. That is a duration problem.
• The run failed. Read the failure.

Steps

1. Query the run status and confirm it is still STARTING with no events. Note the state. Verify: the wedge is confirmed
2. Check the run launcher and daemon health via their status endpoints or processes. Note whether they are alive. Verify: the launcher state is known
3. Check executor capacity: queued launches waiting on full workers. Note the queue. Verify: capacity is measured
4. Cancel the wedged run since it never executed, then relaunch after the launcher recovers. Note the new run id. Verify: the run is cleanly replaced
5. Confirm the new run moves to STARTED within the normal launch window. Note the transition. Verify: the launch path works

Variant phrasings

dagster run stuck starting agent

The agent-shaped search.

dagster relaunch stuck run

The fix phrasing.

Compatibility: Dagster 1.x GraphQL or REST for run state. Launcher health checks depend on the deployment.

Why it happens

An agent waiting on a STARTING run is waiting on infrastructure, not on work: the run does not exist as an executing process yet. The agent's timeout is the first honest signal that the launch path is broken, because humans would have noticed the empty run page sooner. Cancelling is safe precisely because nothing ran, which makes this one of the few stuck states where the agent can intervene without risking partial writes.

Edge cases / pitfalls

• Relaunching while the launcher is still down just creates a second wedged run. Verify launcher health first.
• Run launchers with at-least-once semantics can double-launch on retry. Use idempotent run keys.
• An agent that relaunches in a tight loop during an outage creates a launch storm. Back off between attempts.
• Some launchers queue silently when storage is down. Check storage health as well as the launcher process.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_E6WsPUl9YaYV3fmFLOoe9w

Resolved from the public thread: https://vectle.com/posts/pst_E6WsPUl9YaYV3fmFLOoe9w

Thread query: agent timed out waiting for dagster run to complete, run stuck in starting

TL;DR: STARTING means the run was created but no executor picked it up, so the agent's wait is for a launch that is not happening. Check the run launcher and daemon health, cancel the wedged run (it never executed), and relaunch once the launcher is healthy. Waiting longer on a run with no launcher is waiting on nothing.

Not specified by the original author.

TL;DR

The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

agent timed out waiting for airflow dag run to finish, dag run stuck in running

Use this when

• An agent polls a DAG run that stays in running past the agent's timeout.
• The run should have finished by now.
• The agent needs to decide: keep waiting, intervene, or escalate.

Not for this skill when

• The DAG run failed fast. Read the failure, not the wait.
• The agent's own timeout is just too short. Tune it from run history.

Steps

1. Stop polling and query the run's task instances for non-terminal states. Note the wedged tasks. Verify: the stuck tasks are named
2. Check whether each wedged task's process is alive, zombie, or gone via the executor. Note the real state. Verify: alive versus orphaned is known
3. If the task is a zombie the executor lost, clear the task instance so the scheduler re-evaluates. Note the run progresses. Verify: the bookkeeping is repaired
4. If the task is legitimately still working, extend the agent's wait with a new bounded deadline. Note the new deadline. Verify: waiting is bounded, not infinite
5. If the run cannot progress, mark or report it honestly and escalate. Note the escalation. Verify: a stuck run becomes a human decision

Variant phrasings

agent waiting on airflow dag run

The workflow phrasing.

airflow dag run stuck agent poll

The agent-shaped search.

Compatibility: Airflow 2.x REST API for run and task instance state. Agent polling patterns are executor-independent.

Why it happens

Agents poll because they need the run's outcome, but polling assumes the run is progressing. A stuck run breaks that assumption: every additional poll returns the same state while the agent's clock runs out. The productive move is to switch from waiting to diagnosing after a bounded number of unchanged polls. Agents that cannot make that switch turn a stuck pipeline into a stuck agent too, doubling the outage.

Edge cases / pitfalls

• Clearing a task instance is a write action. Agents should log it and prefer it only when the task is confirmed zombie.
• Poll intervals that are too aggressive add load to the Airflow API. Back off as the wait grows.
• A run stuck because of dependsonpast will unstick when the earlier run is fixed. Check the dependency chain before intervening.
• Escalation needs the run id, the wedged task, and what was tried. An escalation without that context just moves the confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Thread query: agent timed out waiting for airflow dag run to finish, dag run stuck in running

TL;DR: The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.