airflow sqlalchemy pool exhausted, scheduler stuck

Not specified by the original author.

TL;DR

The scheduler opens more concurrent database connections than the pool allows, so threads block waiting for a connection and scheduling stalls. Raise sqlalchemypoolsize to cover the scheduler's parallelism, keep poolrecycle below the database idle timeout, and make sure the database itself allows that many connections. The pool, the scheduler, and the database must agree on the numbers.

airflow sqlalchemy pool exhausted, scheduler stuck

Use this when

• The scheduler stalls with pool timeout or exhausted errors.
• It happens under high DAG and task counts.
• The database shows far fewer connections than the scheduler wants.

Not for this skill when

• Queries are slow but the pool has free connections. Tune the queries.
• Workers exhaust their pools. Size worker pools separately.

Steps

1. Read the scheduler logs for pool exhaustion messages and note the configured pool size. Note the current limit. Verify: the limit is known
2. Estimate the scheduler's concurrent connection need from parallelism settings and DAG count. Note the demand. Verify: demand is quantified
3. Raise sqlalchemypoolsize to cover demand with headroom, and set maxoverflow sensibly. Note the new sizes. Verify: the pool fits the scheduler
4. Confirm the database max_connections accommodates the pool plus workers plus headroom. Note the database ceiling. Verify: the database agrees
5. Restart the scheduler and watch scheduling resume without pool errors. Note the stall is gone. Verify: the fix is verified

Variant phrasings

airflow sqlalchemypool_size tuning

The config-shaped search.

airflow scheduler database connections

The capacity phrasing.

Compatibility: Airflow 2.x. Pool settings live under [database] in airflow.cfg; scheduler parallelism under [scheduler].

Why it happens

The scheduler is surprisingly database-hungry: every scheduling loop, heartbeat, and task-state query needs a connection, and with many DAGs those add up. The default pool size was chosen for small deployments, so growing Airflow hits the ceiling as a stall rather than an error: threads wait for connections that never free up because every thread is waiting. It looks like the scheduler died, but it is just queuing internally. The three numbers, pool size, scheduler parallelism, database max_connections, must be sized as a set.

Edge cases / pitfalls

• Raising the pool without raising database max_connections moves the refusal to the database. Size all three together.
• pool_recycle must stay below the database idle timeout or you trade exhaustion for stale connections.
• Long-running queries hold pool connections hostage. A pool sized for fast queries starves on slow ones; fix the queries too.
• Multiple schedulers in HA multiply pool demand. Count every scheduler's pool against the database ceiling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bsB7og0U5LyTMvVD39V4VQ

Resolved from the public thread: https://vectle.com/posts/pst_bsB7og0U5LyTMvVD39V4VQ

Thread query: airflow sqlalchemy pool exhausted, scheduler stuck

TL;DR: The scheduler opens more concurrent database connections than the pool allows, so threads block waiting for a connection and scheduling stalls. Raise sqlalchemypoolsize to cover the scheduler's parallelism, keep poolrecycle below the database idle timeout, and make sure the database itself allows that many connections. The pool, the scheduler, and the database must agree on the numbers.

Not specified by the original author.

TL;DR

Tasks are stuck in scheduled state because every slot in their pool is taken. Find which pool is full, see which tasks are holding the slots, then decide: raise the slot count, move tasks to another pool, or clear the stuck tasks that are squatting. The pool is a throttle doing its job, exhaustion means the throttle is engaged somewhere it should not be.

Airflow pool slots exhausted: tasks stuck in "scheduled" with no available pool slots

Use this when

• tasks sit in scheduled state and never start
• the UI shows a pool at zero open slots
• a new DAG's tasks never get picked up by workers

Not for this skill when

• tasks are queued but workers are idle, check the executor instead
• tasks fail on start, that is a task problem not a pool problem
• the scheduler itself is down, fix that first

Steps

1. List the pools and their slot usage to find the exhausted one:

airflow pools list

Expected output: each pool with total slots and running, queued, and open counts. The exhausted pool shows zero open slots while tasks wait.

2. See which tasks are holding slots in that pool right now:

SELECT dag_id, task_id, state, pool
FROM task_instance
WHERE pool = 'my_pool' AND state IN ('running', 'scheduled', 'queued')
ORDER BY start_date;

Expected output: the tasks occupying the pool. Long-running or stuck tasks near the top are your squatters, holding slots without doing useful work.

3. Check how long the running tasks have held their slots:

SELECT task_id, state,
       EXTRACT(EPOCH FROM (NOW() - start_date))/3600 AS hours_running
FROM task_instance
WHERE pool = 'my_pool' AND state = 'running'
ORDER BY start_date
LIMIT 10;

Expected output: hours each running task has held its slot. Anything running far past its normal duration is stuck and a candidate for clearing.

4. Raise the slot count if the pool is genuinely undersized for current demand:

airflow pools set my_pool 16 "increased for backfill"

Expected output: the pool now has 16 slots. Tasks waiting in scheduled state start getting picked up on the next scheduler loop.

5. Or clear the stuck tasks that are squatting on slots so the scheduler reschedules them fresh:

airflow tasks clear my_dag --task-regex my_stuck_task --only-running --yes

Expected output: the stuck task instances clear and release their slots. The scheduler picks them up again, and this time they either run or fail fast instead of hanging.

Variant phrasings

airflow tasks stuck in scheduled state

Scheduled means waiting on a slot, a dependency, or priority weight. Step 1 distinguishes pool exhaustion from the other causes quickly.

how to increase airflow pool slots

Step 4, the pools set command. Raise it deliberately, since the pool exists to protect something downstream from overload.

airflow default_pool exhausted

Everything lands in defaultpool unless assigned elsewhere. Either raise defaultpool or move heavy DAGs to dedicated pools so one greedy DAG cannot starve the rest.

Why it happens

Pools cap how many tasks can run concurrently to protect shared resources like a warehouse or an external API. When demand exceeds slots, extra tasks wait in scheduled state indefinitely. Exhaustion is usually growth (more tasks than the pool was sized for) or leakage (stuck tasks holding slots forever), and steps 2 and 3 tell you which one you are dealing with.

Edge cases

• Raising slots without checking downstream capacity just moves the overload to the warehouse or API. Size pools against the real bottleneck.
• default_pool is shared by everything. One greedy DAG can starve the rest, so give heavy DAGs their own pools.
• Deferred tasks can hold pool slots while waiting on triggers in some versions. Check for deferrable operators piling up.
• Pool changes apply on the next scheduler loop, not instantly. Give it a minute before re-diagnosing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmg9q3V1HcvcvHo1F8Bddw

Resolved from the public thread: https://vectle.com/posts/pstmg9q3V1HcvcvHo1F8Bddw

Thread query: Airflow "pool slots" exhausted: how to debug

TL;DR: Tasks are stuck in scheduled state because every slot in their pool is taken. Find which pool is full, see which tasks are holding the slots, then decide: raise the slot count, move tasks to another pool, or clear the stuck tasks that are squatting. The pool is a throttle doing its job, exhaustion means the throttle is engaged somewhere it should not be.

Not specified by the original author.

TL;DR

Queued forever usually means the executor never got the task, the worker that had it died and its heartbeat stopped, or the pool and concurrency limits silently cap the DAG at zero effective slots. Check the task instance state history, the executor event log, and pool occupancy in that order. Clear a genuinely orphaned task instance only after confirming no worker process is running it, or you will get two copies of the same task writing at once.

airflow zombie tasks stuck in queued, scheduler not picking them up

Use this when

• Task instances sit in queued for far longer than the DAG normally waits.
• Workers are idle and report free slots while tasks stay queued.
• A worker restart or deploy left tasks behind in queued or running.

Not for this skill when

• The DAG is paused or the run is queued behind maxactiveruns. That is expected backpressure.
• The pool is full of genuinely running tasks. Wait or raise the pool deliberately.

Steps

1. Inspect one stuck task instance: its state history, try number, hostname, and external executor id. Note whether a worker ever claimed it. Verify: you know if the task was adopted or never sent
2. Check the scheduler and executor logs around the queue time for heartbeat loss, an executor event mismatch, or a worker pod that was killed. Note the event that orphaned it. Verify: the orphaning event is identified
3. Verify capacity: pool slots, the DAG maxactivetasks, worker concurrency, and queued tasks ahead of it. Note any limit sitting at zero free slots for this pool. Verify: a silent capacity cap is confirmed or ruled out
4. Confirm no process is running the task on the recorded hostname or pod, then clear that single task instance so the scheduler requeues it. Note the clear is scoped to the one instance. Verify: the task is requeued without a duplicate running copy
5. Watch the next scheduling loop and confirm the task moves to running and finishes. Note the state transition times. Verify: tasks flow through the executor again

Variant phrasings

airflow tasks stuck in queued zombie task

The zombie wording for the same orphan.

airflow scheduler not scheduling queued tasks

The scheduler-focused phrasing.

Compatibility: Airflow 2.x with Celery, Kubernetes, or Local executors. Zombie detection settings such as the scheduler zombie task threshold apply to running tasks whose heartbeats stopped; purely queued orphans are handled at the task instance level.

Why it happens

Airflow tracks work in two places that can disagree: task instance rows in the metadata database, and the executor's own queue of jobs it believes it owns. When a worker dies mid-flight, a deploy replaces pods, or the scheduler restarts between writing the row and handing off the job, a row can say queued while no executor anywhere holds the work. Nothing retries it because, from each component's view, someone else owns it. The scheduler zombie and orphan handling catches some of these, but rows that never reached a worker fall outside heartbeat based detection.

Edge cases / pitfalls

• Clearing a task that is actually running on a slow worker creates a duplicate execution. Always check the recorded host first.
• A pool set to zero slots, or a pool renamed in config while old rows keep the old name, queues tasks forever with no error anywhere.
• Kubernetes pods evicted during node drains leave exactly this pattern. Correlate queue times with cluster events before blaming Airflow.
• Repeated zombies after every deploy point at workers being killed before they finish. Lengthen the grace period instead of clearing forever.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZhEoHvA1FxbB83q6wKl-Hg

Resolved from the public thread: https://vectle.com/posts/pst_ZhEoHvA1FxbB83q6wKl-Hg

Thread query: airflow zombie tasks stuck in queued, scheduler not picking them up

TL;DR: Queued forever usually means the executor never got the task, the worker that had it died and its heartbeat stopped, or the pool and concurrency limits silently cap the DAG at zero effective slots. Check the task instance state history, the executor event log, and pool occupancy in that order. Clear a genuinely orphaned task instance only after confirming no worker process is running it, or you will get two copies of the same task writing at once.