agent timed out polling dagster graphql for run status, run stuck in starting

TL;DR

The polls correctly report STARTING every time; the problem is the run never launches, not the polling. After a bounded number of unchanged polls, stop polling and check the run launcher. Cancel the unstarted run and relaunch once the launcher is healthy. The polls were working; the launch was not.

agent timed out polling dagster graphql for run status, run stuck in starting

Use this when

• Agent polls show a Dagster run in STARTING until the agent times out.
• Every poll returns the same state.
• The agent needs a rule for when to stop polling.

Not for this skill when

• GraphQL returns errors. Fix the query or the auth.
• The run progresses slowly. That is a duration question.

Steps

1. Poll with a fixed interval and count consecutive unchanged STARTING responses. Note the count. Verify: stagnation is measured
2. After the unchanged threshold, stop polling and check launcher and daemon health. Note their state. Verify: the investigation pivots to the launcher
3. Cancel the wedged run; it never executed. Note the cancellation. Verify: the dead run is cleared
4. Relaunch once the launcher is healthy and poll the new run. Note the new run id. Verify: a live run replaces the dead one
5. Record the incident: launcher outage window and the wasted wait. Note the record. Verify: the pattern is visible next time

Variant phrasings

dagster graphql poll run status

The API-shaped search.

dagster run never starts polling

The symptom phrasing.

Compatibility: Dagster 1.x GraphQL API for run state. Polling pattern is client-side.

Why it happens

Polling is how agents observe, and unchanged polls are information: the system is not moving. The mistake is treating more polls as more information after the state has flatlined. A run in STARTING with no events for N polls is a launch failure with probability approaching one, and every further poll just spends the agent's budget confirming it. The threshold on unchanged polls is what turns observation into a decision.

Edge cases / pitfalls

• GraphQL rate limits can make polls fail, which looks like stagnation. Distinguish failed polls from unchanged-state polls.
• Cancelling a run that is actually mid-launch can orphan work. Confirm zero executor activity before cancelling.
• The unchanged threshold should scale with normal launch times: seconds for local, minutes for Kubernetes launchers.
• Log each poll sparsely. A thousand identical poll logs help nobody; log the transitions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mwvdNH5-Z8fCqjGLOape-w