how to debug connection pooling exhaustion in production

TL;DR

Pool exhaustion means demand for connections exceeds supply: too few connections, connections held too long (slow queries, missing closes), or leaks. Diagnose by watching pool metrics (active, idle, waiting threads) during the incident, then fix the actual cause: usually slow queries holding connections or code not returning them. Raising max pool size without fixing the hold time just moves the bottleneck to the database.

The query

how to debug connection pooling exhaustion in production

Use this when

• Apps log pool exhausted or connection wait timeouts
• Latency spikes correlate with traffic
• Pool sizes were set by guesswork
• After adding new features that use the database heavily

Not for when

• Slow queries themselves (tune those separately, then recheck the pool)
• Database server overload (the pool protects the DB; check both)
• First-time pool configuration

Steps

Step 1: Watch pool metrics during the incident

Look at active connections, idle connections, and threads waiting for a connection. Waiting threads with zero idle connections is exhaustion; waiting threads with idle connections available is a pool bug or misconfiguration. Expected output: the pool state classified: truly exhausted vs misbehaving.

Step 2: Find what holds connections too long

Check for slow queries and transactions holding connections open: long-running transactions, queries without timeouts, code paths that forget to close. The pool drains because checkouts never return, not because traffic is high. Expected output: the top connection-holders identified by query or code path.

Step 3: Check for leaks

If active connections grow monotonically over hours without load growth, something leaks: connections opened and never closed. Heap dumps or connection-creation stack traces find them. Leaks are the most common cause of gradual exhaustion. Expected output: leaking code path found, or leak ruled out by stable counts under steady load.

Step 4: Size the pool from data, not defaults

Set max pool size based on measured concurrent demand plus headroom, and set connection timeouts (max lifetime, idle timeout, checkout timeout) so bad states self-heal. Document the sizing math next to the config. Expected output: pool sized for the real workload with timeouts that bound the damage.

Step 5: Protect the database from the fix

Raising pool size raises database load. Check the database's connection count and per-connection overhead before increasing. Sometimes the right fix is fewer, faster queries rather than a bigger pool. Expected output: pool and database sized as a pair; neither overwhelms the other.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eUJqLZxsuUuSgtXvngu4gQ