dagster run stuck in STARTING state

TL;DR

STARTING means the run was created but no executor picked it up: the run launcher is down, the executor queue is full, or the event log storage is unreachable. Check that the daemon and run launcher are alive first, then executor capacity. A run stuck in STARTING never executed anything, so it is safe to cancel and relaunch once the launcher is healthy.

dagster run stuck in STARTING state

Use this when

• Dagster runs sit in STARTING indefinitely.
• No logs or events appear for the run.
• It started after a deploy or a daemon restart.

Not for this skill when

• Runs are stuck in STARTED. Execution began; look at the executor.
• Runs fail fast. That is a launch error, not a stuck state.

Steps

1. Check the dagster-daemon and run launcher processes: are they running and healthy? Note their status. Verify: the launcher is confirmed alive or dead
2. Look at executor capacity: queued runs waiting on full workers or a saturated multiprocess pool. Note the queue depth. Verify: capacity is measured
3. Verify event log and run storage are reachable: a down Postgres for run storage blocks launches. Note storage health. Verify: storage is ruled out
4. Cancel the stuck runs once the launcher is healthy; they never executed. Note the runs are cancelled cleanly. Verify: the wedged runs are cleared
5. Relaunch and confirm runs move STARTING to STARTED promptly. Note the transition happens. Verify: the path is verified

Variant phrasings

dagster run never leaves starting

The symptom phrasing.

dagster run launcher not picking up runs

The cause-shaped search.

Compatibility: Dagster 1.x. Run launcher and daemon architecture is consistent across 1.x; storage backends vary.

Why it happens

Dagster separates run creation from run execution: creating the run writes a row, and the run launcher is supposed to notice and start it. STARTING is the gap between those two events. When the launcher is down, storage is unreachable, or the executor cannot accept more work, runs pile up in that gap. Nothing has executed yet, which is both the diagnosis and the good news: cancellation is safe.

Edge cases / pitfalls

• A run launcher pointed at the wrong storage sees zero pending runs while the UI shows many. Config mismatch between launcher and webserver is a classic.
• Multiprocess executor limits are per deployment, not per run. One greedy deployment can starve all launches.
• Cancelling a STARTING run that actually started launching can orphan a process. Verify no executor activity before cancelling.
• Daemon restarts during a launch storm can wedge runs that were mid-handoff. Drain the queue before restarting the daemon.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstkw5ingDz4LVQbuG2F-Eww