agent timed out waiting for dagster run to complete, run stuck in starting

TL;DR

STARTING means the run was created but no executor picked it up, so the agent's wait is for a launch that is not happening. Check the run launcher and daemon health, cancel the wedged run (it never executed), and relaunch once the launcher is healthy. Waiting longer on a run with no launcher is waiting on nothing.

agent timed out waiting for dagster run to complete, run stuck in starting

Use this when

• An agent waits on a Dagster run stuck in STARTING.
• The run shows no logs or events.
• The agent's timeout fires while the run sits unstarted.

Not for this skill when

• The run started and is progressing slowly. That is a duration problem.
• The run failed. Read the failure.

Steps

1. Query the run status and confirm it is still STARTING with no events. Note the state. Verify: the wedge is confirmed
2. Check the run launcher and daemon health via their status endpoints or processes. Note whether they are alive. Verify: the launcher state is known
3. Check executor capacity: queued launches waiting on full workers. Note the queue. Verify: capacity is measured
4. Cancel the wedged run since it never executed, then relaunch after the launcher recovers. Note the new run id. Verify: the run is cleanly replaced
5. Confirm the new run moves to STARTED within the normal launch window. Note the transition. Verify: the launch path works

Variant phrasings

dagster run stuck starting agent

The agent-shaped search.

dagster relaunch stuck run

The fix phrasing.

Compatibility: Dagster 1.x GraphQL or REST for run state. Launcher health checks depend on the deployment.

Why it happens

An agent waiting on a STARTING run is waiting on infrastructure, not on work: the run does not exist as an executing process yet. The agent's timeout is the first honest signal that the launch path is broken, because humans would have noticed the empty run page sooner. Cancelling is safe precisely because nothing ran, which makes this one of the few stuck states where the agent can intervene without risking partial writes.

Edge cases / pitfalls

• Relaunching while the launcher is still down just creates a second wedged run. Verify launcher health first.
• Run launchers with at-least-once semantics can double-launch on retry. Use idempotent run keys.
• An agent that relaunches in a tight loop during an outage creates a launch storm. Back off between attempts.
• Some launchers queue silently when storage is down. Check storage health as well as the launcher process.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_E6WsPUl9YaYV3fmFLOoe9w