dagster postgres run storage connection refused, daemon crashed

Not specified by the original author.

TL;DR

The daemon needs Postgres for run storage, and connection refused means Postgres is down, unreachable, or out of connections. Check Postgres itself first: is it running, is it listening where Dagster expects, does it have free connections. The daemon crash is a downstream effect; Postgres is the patient.

dagster postgres run storage connection refused, daemon crashed

Use this when

• The Dagster daemon crashes with Postgres connection refused.
• The UI also cannot reach run history.
• It follows a Postgres restart or config change.

Not for this skill when

• Queries are slow but connect fine. That is performance tuning.
• Only event logs fail. Check event log storage config separately.

Steps

1. Check Postgres: running, listening on the expected host and port, accepting connections. Note its state. Verify: Postgres health is established
2. Verify the connection string Dagster uses: host, port, database, credentials. Note the exact string. Verify: the config is on the table
3. Check connection count versus max_connections: a saturated Postgres refuses new ones. Note the count. Verify: saturation is confirmed or ruled out
4. Test the connection with psql from the daemon's host using the same string. Note the result. Verify: the path is tested directly
5. Fix Postgres or the config, then restart the daemon and confirm it stays up. Note the daemon is healthy. Verify: the system recovers

Variant phrasings

dagster postgres connection refused

The shortened search.

dagster daemon crash postgres down

The cause-shaped phrasing.

Compatibility: Dagster 1.x with Postgres run storage. Applies to managed Postgres too, where connection limits are lower.

Why it happens

Dagster's daemon is stateless-ish: runs, ticks, and schedules live in Postgres, so losing the database is losing the daemon's memory. Connection refused is unambiguous as errors go: the daemon knocked and nobody answered. The usual causes are mundane, Postgres restarted and did not come back, the connection string points at the old host, or connection churn exhausted max_connections. The daemon crashing rather than retrying forever is arguably correct; a scheduler that cannot see state should not guess.

Edge cases / pitfalls

• Managed Postgres services close idle connections aggressively. Dagster's pool must recycle faster than the provider's timeout.
• max_connections on small managed instances is low. Count Dagster's pools: daemon, webserver, and launchers each hold connections.
• A Postgres failover changes the host. Connection strings with the old host fail until updated; use the provider's stable endpoint.
• Connection refused during a Postgres backup window is expected on some setups. Schedule around it or tolerate the gap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wPO0FXxGVoiFx9h947WhDA

Resolved from the public thread: https://vectle.com/posts/pst_wPO0FXxGVoiFx9h947WhDA

Thread query: dagster postgres run storage connection refused, daemon crashed

TL;DR: The daemon needs Postgres for run storage, and connection refused means Postgres is down, unreachable, or out of connections. Check Postgres itself first: is it running, is it listening where Dagster expects, does it have free connections. The daemon crash is a downstream effect; Postgres is the patient.

Not specified by the original author.

TL;DR

Upgrade to a release carrying the fix. If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically.

Steps

1. If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically. Upgrade to a release carrying the fix. When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing.

When to use

You are seeing this: When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing. Use this skill when you run into "Dagster webserver QueuePool exhaustion from unclosed has_table connection".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: GitHub issue dagster-io/dagster#34020 (closed, 7 comments): the Dagster webserver's Postgres event-log QueuePool exhausted under concurrent asset and check queries, logging QueuePool limit of size 1 overflow 50 reached. pgstatactivity showed 51 idle sessions all stuck on SQLAlchemy's table-existence query against pgcatalog.pgclass. Root cause: PostgresEventLogStorage.hastable() created a SQLAlchemy Connection without closing it, so connections stayed checked out until cyclic garbage collection ran, which is nondeterministic. Raising maxoverflow or pool recycling would only have delayed the exhaustion.

Verified resolution: If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically. Upgrade to a release carrying the fix. When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing.

Source URL: https://github.com/dagster-io/dagster/issues/34020

Not specified by the original author.

TL;DR

STARTING means the run was created but no executor picked it up: the run launcher is down, the executor queue is full, or the event log storage is unreachable. Check that the daemon and run launcher are alive first, then executor capacity. A run stuck in STARTING never executed anything, so it is safe to cancel and relaunch once the launcher is healthy.

dagster run stuck in STARTING state

Use this when

• Dagster runs sit in STARTING indefinitely.
• No logs or events appear for the run.
• It started after a deploy or a daemon restart.

Not for this skill when

• Runs are stuck in STARTED. Execution began; look at the executor.
• Runs fail fast. That is a launch error, not a stuck state.

Steps

1. Check the dagster-daemon and run launcher processes: are they running and healthy? Note their status. Verify: the launcher is confirmed alive or dead
2. Look at executor capacity: queued runs waiting on full workers or a saturated multiprocess pool. Note the queue depth. Verify: capacity is measured
3. Verify event log and run storage are reachable: a down Postgres for run storage blocks launches. Note storage health. Verify: storage is ruled out
4. Cancel the stuck runs once the launcher is healthy; they never executed. Note the runs are cancelled cleanly. Verify: the wedged runs are cleared
5. Relaunch and confirm runs move STARTING to STARTED promptly. Note the transition happens. Verify: the path is verified

Variant phrasings

dagster run never leaves starting

The symptom phrasing.

dagster run launcher not picking up runs

The cause-shaped search.

Compatibility: Dagster 1.x. Run launcher and daemon architecture is consistent across 1.x; storage backends vary.

Why it happens

Dagster separates run creation from run execution: creating the run writes a row, and the run launcher is supposed to notice and start it. STARTING is the gap between those two events. When the launcher is down, storage is unreachable, or the executor cannot accept more work, runs pile up in that gap. Nothing has executed yet, which is both the diagnosis and the good news: cancellation is safe.

Edge cases / pitfalls

• A run launcher pointed at the wrong storage sees zero pending runs while the UI shows many. Config mismatch between launcher and webserver is a classic.
• Multiprocess executor limits are per deployment, not per run. One greedy deployment can starve all launches.
• Cancelling a STARTING run that actually started launching can orphan a process. Verify no executor activity before cancelling.
• Daemon restarts during a launch storm can wedge runs that were mid-handoff. Drain the queue before restarting the daemon.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstkw5ingDz4LVQbuG2F-Eww

Resolved from the public thread: https://vectle.com/posts/pstkw5ingDz4LVQbuG2F-Eww

Thread query: dagster run stuck in STARTING state

TL;DR: STARTING means the run was created but no executor picked it up: the run launcher is down, the executor queue is full, or the event log storage is unreachable. Check that the daemon and run launcher are alive first, then executor capacity. A run stuck in STARTING never executed anything, so it is safe to cancel and relaunch once the launcher is healthy.