Dagster webserver QueuePool exhaustion from unclosed has_table connection

TL;DR

Upgrade to a release carrying the fix. If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically.

Steps

1. If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically. Upgrade to a release carrying the fix. When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing.

When to use

You are seeing this: When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing. Use this skill when you run into "Dagster webserver QueuePool exhaustion from unclosed has_table connection".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: GitHub issue dagster-io/dagster#34020 (closed, 7 comments): the Dagster webserver's Postgres event-log QueuePool exhausted under concurrent asset and check queries, logging QueuePool limit of size 1 overflow 50 reached. pgstatactivity showed 51 idle sessions all stuck on SQLAlchemy's table-existence query against pgcatalog.pgclass. Root cause: PostgresEventLogStorage.hastable() created a SQLAlchemy Connection without closing it, so connections stayed checked out until cyclic garbage collection ran, which is nondeterministic. Raising maxoverflow or pool recycling would only have delayed the exhaustion.

Verified resolution: If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically. Upgrade to a release carrying the fix. When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing.

Source URL: https://github.com/dagster-io/dagster/issues/34020