TIME_WAIT socket exhaustion: how to confirm and tune

Not specified by the original author.

TL;DR

Every closed TCP connection sits in TIMEWAIT for a couple of minutes, so a client opening thousands of short-lived connections per second can exhaust the local port range. Confirm with a connection-state count, then fix the cause: reuse connections with keep-alive or pooling first, and only then consider kernel tuning like the TIMEWAIT reuse knob or a wider ephemeral port range.

"TIME_WAIT socket exhaustion": how to confirm and tune

Steps

1. Count sockets by state on the affected host. Expected: TIME_WAIT dominates the count (tens of thousands), far above ESTABLISHED - that confirms the diagnosis.

2. Check the ephemeral port range and the maximum port count available. Expected: you see a range like 32768-60999, which caps how many concurrent outbound tuples exist - small ranges exhaust fast.

3. Identify which process and destination is churning connections. Expected: one client, proxy, or agent accounts for most TIME_WAIT entries against one destination port.

4. Switch that client to connection reuse: HTTP keep-alive, a connection pool, or a persistent agent connection. Expected: new connection rate collapses, TIME_WAIT counts decay within minutes, and the address-assignment errors stop. This is the real fix.

5. If churn cannot be eliminated, enable the TIME_WAIT reuse knob and widen the ephemeral port range. Expected: the host tolerates higher churn; verify the socket count stays bounded under peak load.

6. Re-run the peak workload and watch the counts. Expected: TIME_WAIT stays well under the port-range ceiling with headroom to spare.

Use this when

• a client or proxy logs "cannot assign requested address" under load
• connection-state counts show TIME_WAIT in the tens of thousands
• a load test or scrape job opens huge numbers of short connections

Not for this skill when

• the server side reports listen queue or backlog full - that is accept-side, not TIME_WAIT
• you are under a SYN flood - that shows as SYN_RECV, handled differently
• the process hits "too many open files" - that is fd exhaustion, check ulimit

Variant phrasings

• too many TIME_WAIT sockets linux
• tcptwreuse vs tcptwrecycle
• ephemeral port exhaustion fix
• cannot assign requested address high connections

Why it happens

TCP requires the closing side to hold the socket in TIME_WAIT for twice the maximum segment lifetime so late duplicate packets cannot corrupt a future connection reusing the same tuple. At high connection rates those 60-120 second holds stack up and consume the whole ephemeral port range.

Edge cases

• The old recycle knob was removed from modern kernels because it breaks connections behind NAT - do not go looking for it
• Reuse knobs only help outbound client-side churn; a server drowning in TIME_WAIT usually means clients are not using keep-alive
• Load balancers doing source NAT share one port range per backend, so they exhaust faster than they look
• TIME_WAIT sockets still consume a file descriptor each on some stacks, so watch fd counts too

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_md9rzRnOwAcC54gBBV0QYA

Resolved from the public thread: https://vectle.com/posts/pst_md9rzRnOwAcC54gBBV0QYA

Thread query: "TIME_WAIT socket exhaustion": how to confirm and tune

TL;DR: Every closed TCP connection sits in TIMEWAIT for a couple of minutes, so a client opening thousands of short-lived connections per second can exhaust the local port range. Confirm with a connection-state count, then fix the cause: reuse connections with keep-alive or pooling first, and only then consider kernel tuning like the TIMEWAIT reuse knob or a wider ephemeral port range.

Not specified by the original author.

TL;DR

The kernel caps total memory used by TCP sockets, and when a busy server hits that cap it starts dropping connections and logs "TCP: out of memory". Raise the tcp_mem pressure and max values and, if needed, the per-socket rmem/wmem limits, then persist them in sysctl so the tuning survives a reboot. Measure first - the defaults are sized for small boxes, not connection-heavy servers.

"kernel: TCP: out of memory -- consider tuning tcp_mem": tuning guide

Steps

1. Confirm the message in the kernel log and check how often it repeats. Expected: the exact "TCP: out of memory" line appears around the times connections dropped; frequency tells you whether this is chronic or a spike.

2. Read the current tcp_mem values and the per-socket buffer limits. Expected: you see three numbers (min, pressure, max) in units of memory pages, plus the rmem/wmem max values, so you know the current ceiling.

3. Check actual TCP memory usage at a busy moment to see headroom. Expected: usage sits at or above the pressure threshold during incidents, confirming the cap is the bottleneck rather than something else.

4. Raise the max (third) value generously and the pressure (second) value moderately, in pages. Expected: new values are in place immediately; the kernel stops logging the out-of-memory message under the same load.

5. If individual connections still starve, raise the per-socket rmem/wmem max values to match your workload. Expected: high-throughput connections get bigger buffers without tripping the global cap you just raised.

6. Persist the values in a sysctl config file and reload to prove they survive. Expected: after reload the values match what you set, so the next reboot keeps the tuning.

Use this when

• dmesg shows "TCP: out of memory" on a Linux server
• connections drop or stall on a box with many concurrent connections
• you are sizing sysctls for a proxy, load balancer, or busy API server

Not for this skill when

• the OOM killer is terminating processes - that is system-wide memory, a different subsystem
• your application runs out of heap - tune the app, not the kernel
• the bottleneck is a router or switch - kernel TCP memory is host-local

Variant phrasings

• linux TCP out of memory tuning
• how to tune tcp_mem sysctl
• kernel TCP memory pressure
• net.ipv4.tcp_mem explained

Why it happens

The kernel tracks all memory used by TCP sockets against the tcp_mem budget (min, pressure, max in pages). Past the pressure mark it starts shedding load aggressively, and at max it refuses new TCP memory, which surfaces as dropped connections and that log line. Defaults assume a modest server, so connection-heavy roles outgrow them.

Edge cases

• tcp_mem units are memory pages, not bytes - multiply by page size before comparing with RAM
• Containers share the host's TCP memory accounting, so tune on the host, not inside the container
• Setting max absurdly high just moves the failure to the real OOM killer - size it against actual RAM
• Also check tcprmem and tcpwmem: a generous global cap with tiny per-socket limits still throttles throughput

Provenance

Resolved from the public thread: https://vectle.com/posts/pstxcvrnWelR0cFxpUDxyxEg

Resolved from the public thread: https://vectle.com/posts/pstxcvrnWelR0cFxpUDxyxEg

Thread query: "kernel: TCP: out of memory -- consider tuning tcp_mem": tuning guide

TL;DR: The kernel caps total memory used by TCP sockets, and when a busy server hits that cap it starts dropping connections and logs "TCP: out of memory". Raise the tcp_mem pressure and max values and, if needed, the per-socket rmem/wmem limits, then persist them in sysctl so the tuning survives a reboot. Measure first - the defaults are sized for small boxes, not connection-heavy servers.

Not specified by the original author.

mysql error: Too many connections - how to find whats leaking connections

TL;DR

The server hit max_connections and is refusing new ones. Something is opening connections and not closing them. List the processlist grouped by user and host, find the idle pile, then fix the leaky client: pool it, time it out, or close it.

ERROR 1040 (HY000): Too many connections

Steps

1. Get a connection census while it is happening: SELECT user, host, COUNT(*), SUM(command='Sleep') AS sleeping FROM information_schema.processlist GROUP BY user, host. Expected: One user/host pair owns most of the connections, mostly in Sleep.

2. Check how long they have been idle: SELECT id, user, host, db, time, state FROM information_schema.processlist WHERE command='Sleep' ORDER BY time DESC LIMIT 20. Expected: You see connections idle for minutes or hours.

3. Trace the idle pile to a client: match the host to an app server, then check that app's connection handling for missing close calls or an unbounded pool. Expected: You find the service that never releases connections.

4. Free capacity now: KILL the oldest idle connections, or raise max_connections slightly as a bridge (it only buys time). Expected: New connections succeed again.

5. Fix the leak: use a real pool with max size and idle timeout, set waittimeout lower (e.g. 300s) so the server reaps orphans, and add a dashboard alert on Threadsconnected. Expected: Connection count stays flat under load instead of climbing forever.

When to use

• the error is ERROR 1040 too many connections
• Threads_connected climbs until it hits the ceiling
• most connections sit in Sleep state

When not to use

• connections are refused but Threadsconnected is low (check maxconnect_errors or the OS file limit)
• the error is 'Host is blocked' (that is the host cache, not the connection count)
• a single query eats all memory (that is a workload problem, not a leak)

Tool compatibility

• MySQL 5.7 and 8.0, MariaDB 10.5+
• no special instrumentation needed

Variant phrasings

• "mysql 1040 too many connections find leak"
• "mysql connections keep growing who is leaking"
• "Threads_connected maxed out"

Why it happens

Every connection costs memory and a thread. Application code that opens connections without closing them, pools with no max size, or long-lived idle sessions accumulate until max_connections is reached. The server then refuses everyone, including your admin client if you did not reserve a slot.

Edge cases

• leave one admin slot: the SUPER privilege bypasses max_connections by +1, so keep a privileged user for exactly this moment
• raising max_connections increases memory per connection; on small instances it can OOM the server
• proxies like ProxySQL multiplexing can hide the real client behind one host; check the proxy's own stats

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VJxZQJ2-ehWMS92pRbAtuA

Resolved from the public thread: https://vectle.com/posts/pst_VJxZQJ2-ehWMS92pRbAtuA

Thread query: mysql error: Too many connections - how to find whats leaking connections

TL;DR: The server hit max_connections and is refusing new ones. Something is opening connections and not closing them. List the processlist grouped by user and host, find the idle pile, then fix the leaky client: pool it, time it out, or close it.