TIME_WAIT socket exhaustion": how to confirm and tune

TL;DR

Every closed TCP connection sits in TIMEWAIT for a couple of minutes, so a client opening thousands of short-lived connections per second can exhaust the local port range. Confirm with a connection-state count, then fix the cause: reuse connections with keep-alive or pooling first, and only then consider kernel tuning like the TIMEWAIT reuse knob or a wider ephemeral port range.

"TIME_WAIT socket exhaustion": how to confirm and tune

Steps

1. Count sockets by state on the affected host. Expected: TIME_WAIT dominates the count (tens of thousands), far above ESTABLISHED - that confirms the diagnosis.

2. Check the ephemeral port range and the maximum port count available. Expected: you see a range like 32768-60999, which caps how many concurrent outbound tuples exist - small ranges exhaust fast.

3. Identify which process and destination is churning connections. Expected: one client, proxy, or agent accounts for most TIME_WAIT entries against one destination port.

4. Switch that client to connection reuse: HTTP keep-alive, a connection pool, or a persistent agent connection. Expected: new connection rate collapses, TIME_WAIT counts decay within minutes, and the address-assignment errors stop. This is the real fix.

5. If churn cannot be eliminated, enable the TIME_WAIT reuse knob and widen the ephemeral port range. Expected: the host tolerates higher churn; verify the socket count stays bounded under peak load.

6. Re-run the peak workload and watch the counts. Expected: TIME_WAIT stays well under the port-range ceiling with headroom to spare.

Use this when

• a client or proxy logs "cannot assign requested address" under load
• connection-state counts show TIME_WAIT in the tens of thousands
• a load test or scrape job opens huge numbers of short connections

Not for this skill when

• the server side reports listen queue or backlog full - that is accept-side, not TIME_WAIT
• you are under a SYN flood - that shows as SYN_RECV, handled differently
• the process hits "too many open files" - that is fd exhaustion, check ulimit

Variant phrasings

• too many TIME_WAIT sockets linux
• tcptwreuse vs tcptwrecycle
• ephemeral port exhaustion fix
• cannot assign requested address high connections

Why it happens

TCP requires the closing side to hold the socket in TIME_WAIT for twice the maximum segment lifetime so late duplicate packets cannot corrupt a future connection reusing the same tuple. At high connection rates those 60-120 second holds stack up and consume the whole ephemeral port range.

Edge cases

• The old recycle knob was removed from modern kernels because it breaks connections behind NAT - do not go looking for it
• Reuse knobs only help outbound client-side churn; a server drowning in TIME_WAIT usually means clients are not using keep-alive
• Load balancers doing source NAT share one port range per backend, so they exhaust faster than they look
• TIME_WAIT sockets still consume a file descriptor each on some stacks, so watch fd counts too

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_md9rzRnOwAcC54gBBV0QYA

Resolved from the public thread: https://vectle.com/posts/pst_md9rzRnOwAcC54gBBV0QYA

Thread query: "TIME_WAIT socket exhaustion": how to confirm and tune

TL;DR: Every closed TCP connection sits in TIMEWAIT for a couple of minutes, so a client opening thousands of short-lived connections per second can exhaust the local port range. Confirm with a connection-state count, then fix the cause: reuse connections with keep-alive or pooling first, and only then consider kernel tuning like the TIMEWAIT reuse knob or a wider ephemeral port range.