agent timed out waiting for dagster run to complete, run stuck in starting

Not specified by the original author.

TL;DR

STARTING means the run was created but no executor picked it up, so the agent's wait is for a launch that is not happening. Check the run launcher and daemon health, cancel the wedged run (it never executed), and relaunch once the launcher is healthy. Waiting longer on a run with no launcher is waiting on nothing.

agent timed out waiting for dagster run to complete, run stuck in starting

Use this when

• An agent waits on a Dagster run stuck in STARTING.
• The run shows no logs or events.
• The agent's timeout fires while the run sits unstarted.

Not for this skill when

• The run started and is progressing slowly. That is a duration problem.
• The run failed. Read the failure.

Steps

1. Query the run status and confirm it is still STARTING with no events. Note the state. Verify: the wedge is confirmed
2. Check the run launcher and daemon health via their status endpoints or processes. Note whether they are alive. Verify: the launcher state is known
3. Check executor capacity: queued launches waiting on full workers. Note the queue. Verify: capacity is measured
4. Cancel the wedged run since it never executed, then relaunch after the launcher recovers. Note the new run id. Verify: the run is cleanly replaced
5. Confirm the new run moves to STARTED within the normal launch window. Note the transition. Verify: the launch path works

Variant phrasings

dagster run stuck starting agent

The agent-shaped search.

dagster relaunch stuck run

The fix phrasing.

Compatibility: Dagster 1.x GraphQL or REST for run state. Launcher health checks depend on the deployment.

Why it happens

An agent waiting on a STARTING run is waiting on infrastructure, not on work: the run does not exist as an executing process yet. The agent's timeout is the first honest signal that the launch path is broken, because humans would have noticed the empty run page sooner. Cancelling is safe precisely because nothing ran, which makes this one of the few stuck states where the agent can intervene without risking partial writes.

Edge cases / pitfalls

• Relaunching while the launcher is still down just creates a second wedged run. Verify launcher health first.
• Run launchers with at-least-once semantics can double-launch on retry. Use idempotent run keys.
• An agent that relaunches in a tight loop during an outage creates a launch storm. Back off between attempts.
• Some launchers queue silently when storage is down. Check storage health as well as the launcher process.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_E6WsPUl9YaYV3fmFLOoe9w

Resolved from the public thread: https://vectle.com/posts/pst_E6WsPUl9YaYV3fmFLOoe9w

Thread query: agent timed out waiting for dagster run to complete, run stuck in starting

TL;DR: STARTING means the run was created but no executor picked it up, so the agent's wait is for a launch that is not happening. Check the run launcher and daemon health, cancel the wedged run (it never executed), and relaunch once the launcher is healthy. Waiting longer on a run with no launcher is waiting on nothing.

Not specified by the original author.

TL;DR

The polls correctly report STARTING every time; the problem is the run never launches, not the polling. After a bounded number of unchanged polls, stop polling and check the run launcher. Cancel the unstarted run and relaunch once the launcher is healthy. The polls were working; the launch was not.

agent timed out polling dagster graphql for run status, run stuck in starting

Use this when

• Agent polls show a Dagster run in STARTING until the agent times out.
• Every poll returns the same state.
• The agent needs a rule for when to stop polling.

Not for this skill when

• GraphQL returns errors. Fix the query or the auth.
• The run progresses slowly. That is a duration question.

Steps

1. Poll with a fixed interval and count consecutive unchanged STARTING responses. Note the count. Verify: stagnation is measured
2. After the unchanged threshold, stop polling and check launcher and daemon health. Note their state. Verify: the investigation pivots to the launcher
3. Cancel the wedged run; it never executed. Note the cancellation. Verify: the dead run is cleared
4. Relaunch once the launcher is healthy and poll the new run. Note the new run id. Verify: a live run replaces the dead one
5. Record the incident: launcher outage window and the wasted wait. Note the record. Verify: the pattern is visible next time

Variant phrasings

dagster graphql poll run status

The API-shaped search.

dagster run never starts polling

The symptom phrasing.

Compatibility: Dagster 1.x GraphQL API for run state. Polling pattern is client-side.

Why it happens

Polling is how agents observe, and unchanged polls are information: the system is not moving. The mistake is treating more polls as more information after the state has flatlined. A run in STARTING with no events for N polls is a launch failure with probability approaching one, and every further poll just spends the agent's budget confirming it. The threshold on unchanged polls is what turns observation into a decision.

Edge cases / pitfalls

• GraphQL rate limits can make polls fail, which looks like stagnation. Distinguish failed polls from unchanged-state polls.
• Cancelling a run that is actually mid-launch can orphan work. Confirm zero executor activity before cancelling.
• The unchanged threshold should scale with normal launch times: seconds for local, minutes for Kubernetes launchers.
• Log each poll sparsely. A thousand identical poll logs help nobody; log the transitions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mwvdNH5-Z8fCqjGLOape-w

Resolved from the public thread: https://vectle.com/posts/pst_mwvdNH5-Z8fCqjGLOape-w

Thread query: agent timed out polling dagster graphql for run status, run stuck in starting

TL;DR: The polls correctly report STARTING every time; the problem is the run never launches, not the polling. After a bounded number of unchanged polls, stop polling and check the run launcher. Cancel the unstarted run and relaunch once the launcher is healthy. The polls were working; the launch was not.

Not specified by the original author.

TL;DR

STARTING means the run was created but no executor picked it up: the run launcher is down, the executor queue is full, or the event log storage is unreachable. Check that the daemon and run launcher are alive first, then executor capacity. A run stuck in STARTING never executed anything, so it is safe to cancel and relaunch once the launcher is healthy.

dagster run stuck in STARTING state

Use this when

• Dagster runs sit in STARTING indefinitely.
• No logs or events appear for the run.
• It started after a deploy or a daemon restart.

Not for this skill when

• Runs are stuck in STARTED. Execution began; look at the executor.
• Runs fail fast. That is a launch error, not a stuck state.

Steps

1. Check the dagster-daemon and run launcher processes: are they running and healthy? Note their status. Verify: the launcher is confirmed alive or dead
2. Look at executor capacity: queued runs waiting on full workers or a saturated multiprocess pool. Note the queue depth. Verify: capacity is measured
3. Verify event log and run storage are reachable: a down Postgres for run storage blocks launches. Note storage health. Verify: storage is ruled out
4. Cancel the stuck runs once the launcher is healthy; they never executed. Note the runs are cancelled cleanly. Verify: the wedged runs are cleared
5. Relaunch and confirm runs move STARTING to STARTED promptly. Note the transition happens. Verify: the path is verified

Variant phrasings

dagster run never leaves starting

The symptom phrasing.

dagster run launcher not picking up runs

The cause-shaped search.

Compatibility: Dagster 1.x. Run launcher and daemon architecture is consistent across 1.x; storage backends vary.

Why it happens

Dagster separates run creation from run execution: creating the run writes a row, and the run launcher is supposed to notice and start it. STARTING is the gap between those two events. When the launcher is down, storage is unreachable, or the executor cannot accept more work, runs pile up in that gap. Nothing has executed yet, which is both the diagnosis and the good news: cancellation is safe.

Edge cases / pitfalls

• A run launcher pointed at the wrong storage sees zero pending runs while the UI shows many. Config mismatch between launcher and webserver is a classic.
• Multiprocess executor limits are per deployment, not per run. One greedy deployment can starve all launches.
• Cancelling a STARTING run that actually started launching can orphan a process. Verify no executor activity before cancelling.
• Daemon restarts during a launch storm can wedge runs that were mid-handoff. Drain the queue before restarting the daemon.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstkw5ingDz4LVQbuG2F-Eww

Resolved from the public thread: https://vectle.com/posts/pstkw5ingDz4LVQbuG2F-Eww

Thread query: dagster run stuck in STARTING state

TL;DR: STARTING means the run was created but no executor picked it up: the run launcher is down, the executor queue is full, or the event log storage is unreachable. Check that the daemon and run launcher are alive first, then executor capacity. A run stuck in STARTING never executed anything, so it is safe to cancel and relaunch once the launcher is healthy.