kernel: TCP: out of memory -- consider tuning tcp_mem": tuning guide

TL;DR

The kernel caps total memory used by TCP sockets, and when a busy server hits that cap it starts dropping connections and logs "TCP: out of memory". Raise the tcp_mem pressure and max values and, if needed, the per-socket rmem/wmem limits, then persist them in sysctl so the tuning survives a reboot. Measure first - the defaults are sized for small boxes, not connection-heavy servers.

"kernel: TCP: out of memory -- consider tuning tcp_mem": tuning guide

Steps

1. Confirm the message in the kernel log and check how often it repeats. Expected: the exact "TCP: out of memory" line appears around the times connections dropped; frequency tells you whether this is chronic or a spike.

2. Read the current tcp_mem values and the per-socket buffer limits. Expected: you see three numbers (min, pressure, max) in units of memory pages, plus the rmem/wmem max values, so you know the current ceiling.

3. Check actual TCP memory usage at a busy moment to see headroom. Expected: usage sits at or above the pressure threshold during incidents, confirming the cap is the bottleneck rather than something else.

4. Raise the max (third) value generously and the pressure (second) value moderately, in pages. Expected: new values are in place immediately; the kernel stops logging the out-of-memory message under the same load.

5. If individual connections still starve, raise the per-socket rmem/wmem max values to match your workload. Expected: high-throughput connections get bigger buffers without tripping the global cap you just raised.

6. Persist the values in a sysctl config file and reload to prove they survive. Expected: after reload the values match what you set, so the next reboot keeps the tuning.

Use this when

• dmesg shows "TCP: out of memory" on a Linux server
• connections drop or stall on a box with many concurrent connections
• you are sizing sysctls for a proxy, load balancer, or busy API server

Not for this skill when

• the OOM killer is terminating processes - that is system-wide memory, a different subsystem
• your application runs out of heap - tune the app, not the kernel
• the bottleneck is a router or switch - kernel TCP memory is host-local

Variant phrasings

• linux TCP out of memory tuning
• how to tune tcp_mem sysctl
• kernel TCP memory pressure
• net.ipv4.tcp_mem explained

Why it happens

The kernel tracks all memory used by TCP sockets against the tcp_mem budget (min, pressure, max in pages). Past the pressure mark it starts shedding load aggressively, and at max it refuses new TCP memory, which surfaces as dropped connections and that log line. Defaults assume a modest server, so connection-heavy roles outgrow them.

Edge cases

• tcp_mem units are memory pages, not bytes - multiply by page size before comparing with RAM
• Containers share the host's TCP memory accounting, so tune on the host, not inside the container
• Setting max absurdly high just moves the failure to the real OOM killer - size it against actual RAM
• Also check tcprmem and tcpwmem: a generous global cap with tiny per-socket limits still throttles throughput

Provenance

Resolved from the public thread: https://vectle.com/posts/pstxcvrnWelR0cFxpUDxyxEg