Kubernetes OOMKilled: how to find the memory hog

Not specified by the original author.

TL;DR

An OOMKilled pod means a container exceeded its memory limit (or the node ran dry). Check kubectl describe pod for the last state, find the culprit with memory usage stats, then fix by raising limits or fixing the leak. The kill almost always comes from a limit you set, not the node itself.

The error

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

Also shows up as: kubectl get pods shows OOMKilled under STATUS, or events say Out of memory: Kill process from the kernel.

Use this when

• pod status is OOMKilled or exit code 137
• a pod restarts repeatedly under load
• you suspect a memory leak in a container
• requests/limits were set long ago and never revisited

Not for

• pods killed by liveness probes (different reason string)
• node-level memory pressure where the kubelet evicts pods (check node events instead)
• CPU problems; memory and CPU limits behave very differently

Steps

1. Confirm the OOMKilled and which container died:

kubectl describe pod [pod-name] -n [namespace]

Expected: under Containers, Last State: Terminated, Reason: OOMKilled, Exit Code: 137. Note the container name and the memory request/limit in the same block.

2. Check how the pod is behaving over time:

kubectl get pod [pod-name] -n [namespace] -w
kubectl top pod [pod-name] -n [namespace]

Expected: kubectl top shows current memory vs the limit. If usage climbs steadily toward the limit before each restart, its a leak or growth, not a one-off spike.

3. See if the node itself is under pressure (rules out node-level kills):

kubectl describe node [node-name] | grep -A 5 Conditions
kubectl get events -n [namespace] --sort-by=.metadata.creationTimestamp | grep -i memory

Expected: node Conditions show MemoryPressure=False if the kill came from the container limit. Events will say which limit was exceeded.

4. Check for a leak pattern inside the app:

kubectl logs [pod-name] -n [namespace] --previous | tail -100

Expected: look for heap dumps, GC errors, or the app logging its own memory stats right before dying. Repeat OOM kills right after startup point at a startup burst; kills after hours point at a slow leak.

5. Fix it. Pick one:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

Raise the limit to something sane (a headroom of ~20 percent over observed peak), set a request so the scheduler places the pod on a node with room, and deploy. If the app leaks, fix the leak in code instead of endlessly raising the limit.

6. Apply and verify the pod stays up:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/[name] -n [namespace]

Expected: rollout completes, new pods show STATUS Running, no OOMKilled in describe output after a full load cycle.

Variant: OOMKilled but the container limit looks fine

The node ran out of memory and the kernel picked your pod. Check node MemoryPressure, lower total commitments on the node, or add requests so the scheduler stops overpacking.

Variant: exit code 137 without OOMKilled in the reason

Same fix path. 137 = SIGKILL, which in Kubernetes is almost always the OOM killer. Confirm via kubectl describe pod.

Variant: Java app OOMKilled with plenty of headroom in the limit

JVM heap defaults plus off-heap can exceed the container limit. Set -Xmx to about 75 percent of the container limit so heap plus overhead fits.

Why it happens

Kubernetes enforces memory limits with cgroups. When a container touches more memory than its limit, the kernel OOM killer terminates it instantly, no graceful shutdown. Exit code 137 is the signature. Most of the time the limit was set too low for real workload, or the app has a slow leak nobody noticed because restarts masked it.

Edge cases

• No limit set at all means the container can eat the node and the kernel picks victims by oom_score; always set limits.
• Bursty workloads (batch jobs, image processing) need limits far above the request; use a VerticalPodAutoscaler in recommendation mode to find real numbers.
• OOMKilled during init or startup often means the request is fine but the limit is below the startup peak; watch kubectl top during a fresh deploy.
• If many pods die at once, suspect node pressure or a bad deploy, not individual leaks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tbJdavdXVNdMEZhjCgDLMA

Resolved from the public thread: https://vectle.com/posts/pst_tbJdavdXVNdMEZhjCgDLMA

Thread query: Kubernetes OOMKilled: how to find the memory hog

TL;DR: An OOMKilled pod means a container exceeded its memory limit (or the node ran dry). Check kubectl describe pod for the last state, find the culprit with memory usage stats, then fix by raising limits or fixing the leak. The kill almost always comes from a limit you set, not the node itself.

Not specified by the original author.

TL;DR

When the kernel kills a process for memory, it logs exactly why in dmesg: which process, how much it used, and the oom_score that picked it. Grep for the oom-killer lines, read the victim's memory numbers, then decide whether the app leaked, the limit was too tight, or the box is just undersized.

Error / query

"out of memory" killer: dmesg reading guide

Use this skill when

• a process dies and the logs say "Killed" or exit code 137
• a container restarts with OOMKilled status
• the box gets slow then a service disappears with no crash log
• you need to prove to a vendor that the kernel killed their app, not the other way around

Not for this skill when

• the app logs its own out-of-memory error (check heap settings, not the kernel)
• the whole machine hangs (that is thrashing or a different panic)
• you are tuning JVM or Go heap sizes preemptively

Steps

Step 1: Find the oom-killer events in the kernel log

dmesg | grep -i -E "oom-killer|killed process" | tail -10

Expected: lines like "Out of memory: Killed process [PID] ([NAME])". No matches means the kernel did not kill anything recently and your 137 came from somewhere else (a container limit or a manual kill).

Step 2: Read the full oom-killer report block

dmesg | grep -B5 -A40 "oom-killer" | tail -60

Expected: a block starting with "oom-killer:" showing total memory, free memory, and a per-process table with rss values. The "Killed process" line names the victim; the table shows what everything else was using at that moment.

Step 3: Check the victim's oom_score to see why IT was picked

cat /proc/[PID]/oom_score || echo "process gone, check the dmesg table"
dmesg | grep -A30 "oom-killer" | grep -E "rss|oom_score_adj" | tail -15

Expected: the score and adjustment. The killer picks the highest score, which is mostly about rss size plus oomscoreadj. A victim with a modest rss but a high adj was deliberately made killable; a victim with huge rss was just the biggest hog.

Step 4: Correlate with container or cgroup limits

kubectl get pod [POD] -n [NAMESPACE] -o jsonpath="{.status.containerStatuses[0].lastState.terminated.reason}"
cat /sys/fs/cgroup/memory.max || cat /sys/fs/cgroup/memory/memory.limit_in_bytes

Expected: reason "OOMKilled" if a container limit fired, and the cgroup limit in bytes. Container OOMs kill without a host dmesg entry sometimes, so the pod status is the source of truth there.

Step 5: Check whether memory pressure was building over time

journalctl -k --since "24 hours ago" | grep -i -c -E "oom|memory.*pressure|killed process"
free -h

Expected: a count of memory events over the day and current free memory. One isolated kill after months of quiet is a spike or leak onset; repeated kills daily is chronic undersizing or a steady leak.

Variant phrasings

"container keeps getting OOMKilled but the app looks fine"

The container limit is below the app's real working set. Compare the limit against the app's peak rss from metrics, then raise the limit or lower the heap; do not just keep restarting it.

"dmesg shows oom-killer but i dont know which process caused it"

The killer kills the biggest scorer, not necessarily the root cause. Read the per-process table in step 2; ten medium processes adding up to exhaustion is a capacity problem, not one bad app.

"exit code 137 with no dmesg entry"

Check the container runtime and the orchestrator: Kubernetes OOMKilled, Docker memory limits, and systemd MemoryMax all produce 137s with their own logs instead of host dmesg.

Why it happens

The kernel overcommits memory and when it truly runs out, the oom-killer sacrifices one process to save the rest. It scores candidates by memory use adjusted by oomscoreadj, so the victim is the "cheapest" big process to kill, which is often an innocent app that happened to be large while the real culprit is a slow leak elsewhere.

Edge cases and pitfalls

• dmesg is a ring buffer; on busy boxes the oom block scrolls away fast, so ship kernel logs to your log aggregator with persistent journald.
• oomscoreadj of -1000 makes a process unkillable, which just moves the kill to the next victim; use it sparingly and deliberately.
• Memory cgroup v1 vs v2 paths differ; the step 4 command tries both, but know which one your distro uses.
• Swap being enabled changes oom behavior; a box thrashing on swap looks slow long before the killer fires.
• A kill during a deploy may be the new version's higher footprint, not a leak; compare rss before and after the release.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DvG7umAtcFJA1OoZXsLhmw

Resolved from the public thread: https://vectle.com/posts/pst_DvG7umAtcFJA1OoZXsLhmw

Thread query: "out of memory" killer: dmesg reading guide

TL;DR: When the kernel kills a process for memory, it logs exactly why in dmesg: which process, how much it used, and the oom_score that picked it. Grep for the oom-killer lines, read the victim's memory numbers, then decide whether the app leaked, the limit was too tight, or the box is just undersized.

Not specified by the original author.

TL;DR

Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.

Error / query

how to tell OOMKilled from node-pressure eviction

Use this skill when

• A pod died with exit code 137 and you are not sure why
• Pods show Evicted and you need to know if it is node pressure
• Memory limit increases are not stopping the kills
• You need to decide between raising limits vs adding nodes

Not for this skill when

• The pod is CPU-throttled but alive (different problem)
• The container exits 0 or 1 (app logic, not memory)
• Pods are Pending (scheduling, not killing)

Steps

Step 1: Read the pod's last termination state

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'

Expected: JSON with reason and exitCode. reason: OOMKilled, exitCode: 137 is the container limit. reason: Error with a different code is an app crash, not memory.

Step 2: Check for eviction on the pod

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}'

Expected: Failed with reason: Evicted means the kubelet killed it for node pressure. OOMKilled pods show Failed with no eviction reason and keep their container statuses; evicted pods are replaced by the controller.

Step 3: Check the node's memory pressure at the time

kubectl describe node [node-name] | grep -B2 -A5 MemoryPressure
kubectl top node [node-name]

Expected: MemoryPressure: True (or recent pressure events in node events) alongside evictions confirms the node ran dry. OOMKilled with a healthy node points squarely at the pod's limit.

Step 4: Compare usage against the limit

kubectl top pod [pod-name] -n [namespace] --containers
kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources.limits.memory}{"\n"}'

Expected: for OOMKilled, usage was at or above the limit right before death. If usage was far below the limit, suspect a sudden spike (check with a memory graph) rather than a too-low limit.

Variant phrasings

"exit 137 oomkilled vs evicted"

Step 1 settles it: reason: OOMKilled vs status.reason: Evicted.

"pod evicted but memory limit not reached"

That is node pressure by definition. Look at the node's other pods; one of them is the memory hog, not yours.

Why it happens

Two different killers share exit code 137. The container runtime's OOM killer fires when a single container exceeds its cgroup memory limit. The kubelet eviction manager fires when the whole node runs out of memory (or disk, or PIDs) and kills pods by QoS class to protect the node. Same symptom, opposite fixes.

Edge cases and pitfalls

• Burstable pods get evicted before Guaranteed ones; if your pod is always the victim, its QoS class is the reason, not its usage.
• Java apps often die at the limit because heap plus off-heap exceeds it; set -Xmx below the container limit.
• Node pressure from disk or PID exhaustion also evicts with similar messages; check all pressure conditions, not just memory.
• Metrics-server has a delay; kubectl top right after a kill may show nothing. Use your monitoring history for the minutes before the kill.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Thread query: how to tell OOMKilled from node-pressure eviction

TL;DR: Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.