how to tell OOMKilled from node-pressure eviction

Not specified by the original author.

TL;DR

Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.

Error / query

how to tell OOMKilled from node-pressure eviction

Use this skill when

• A pod died with exit code 137 and you are not sure why
• Pods show Evicted and you need to know if it is node pressure
• Memory limit increases are not stopping the kills
• You need to decide between raising limits vs adding nodes

Not for this skill when

• The pod is CPU-throttled but alive (different problem)
• The container exits 0 or 1 (app logic, not memory)
• Pods are Pending (scheduling, not killing)

Steps

Step 1: Read the pod's last termination state

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'

Expected: JSON with reason and exitCode. reason: OOMKilled, exitCode: 137 is the container limit. reason: Error with a different code is an app crash, not memory.

Step 2: Check for eviction on the pod

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}'

Expected: Failed with reason: Evicted means the kubelet killed it for node pressure. OOMKilled pods show Failed with no eviction reason and keep their container statuses; evicted pods are replaced by the controller.

Step 3: Check the node's memory pressure at the time

kubectl describe node [node-name] | grep -B2 -A5 MemoryPressure
kubectl top node [node-name]

Expected: MemoryPressure: True (or recent pressure events in node events) alongside evictions confirms the node ran dry. OOMKilled with a healthy node points squarely at the pod's limit.

Step 4: Compare usage against the limit

kubectl top pod [pod-name] -n [namespace] --containers
kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources.limits.memory}{"\n"}'

Expected: for OOMKilled, usage was at or above the limit right before death. If usage was far below the limit, suspect a sudden spike (check with a memory graph) rather than a too-low limit.

Variant phrasings

"exit 137 oomkilled vs evicted"

Step 1 settles it: reason: OOMKilled vs status.reason: Evicted.

"pod evicted but memory limit not reached"

That is node pressure by definition. Look at the node's other pods; one of them is the memory hog, not yours.

Why it happens

Two different killers share exit code 137. The container runtime's OOM killer fires when a single container exceeds its cgroup memory limit. The kubelet eviction manager fires when the whole node runs out of memory (or disk, or PIDs) and kills pods by QoS class to protect the node. Same symptom, opposite fixes.

Edge cases and pitfalls

• Burstable pods get evicted before Guaranteed ones; if your pod is always the victim, its QoS class is the reason, not its usage.
• Java apps often die at the limit because heap plus off-heap exceeds it; set -Xmx below the container limit.
• Node pressure from disk or PID exhaustion also evicts with similar messages; check all pressure conditions, not just memory.
• Metrics-server has a delay; kubectl top right after a kill may show nothing. Use your monitoring history for the minutes before the kill.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Thread query: how to tell OOMKilled from node-pressure eviction

TL;DR: Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.

Not specified by the original author.

TL;DR

Exit code 137 is 128 plus SIGKILL. Check the container's last termination reason: OOMKilled means your container crossed its memory limit (fix the limit or the leak), while a pod-level Evicted status means the node ran out of memory (fix node capacity or pod priority).

Verbatim error

State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137

Steps

1. Read the termination state of the dead container.

kubectl get pod [POD] -n [NAMESPACE] -o yaml

Expected: in lastState.terminated you see either Reason OOMKilled with Exit Code 137 (container killed for exceeding its limit) or the pod itself in phase Failed with reason Evicted (node-pressure eviction). This one field decides the whole fix.

2. If the reason is OOMKilled, compare the limit against actual usage.

kubectl top pod [POD] -n [NAMESPACE]
kubectl describe pod [POD] -n [NAMESPACE]

Expected: describe shows the memory limit; top shows usage near or above it just before death. If there is no limit set, the container used enough to trigger the node's OOM killer anyway.

3. Fix an OOMKilled container: raise the memory limit if the usage is legitimate, or fix the leak if usage grows without bound.

kubectl -n [NAMESPACE] edit deployment [WORKLOAD]

Expected: after raising limits under resources, the new pods survive the same workload. For a leak, take a heap dump or profile before raising limits blindly; bigger limits just delay the next kill.

4. If the pod was Evicted instead, check node memory pressure at the time.

kubectl describe node [NODE]
kubectl get events -A --field-selector reason=Evicted

Expected: the node shows MemoryPressure in its conditions around the eviction time, and the event names the pod. Eviction picks pods by priority and by how far usage exceeds requests.

5. Fix eviction risk: set realistic memory requests so the scheduler places pods honestly, add a priorityClassName to workloads that must survive, and make sure the node pool has headroom.

kubectl get priorityclass

Expected: your critical workloads carry a PriorityClass above best-effort. Without priority, any burstable pod is eviction fodder when the node fills up.

When to use

• A container terminates with exit code 137
• You need to know whether to raise a limit or add node capacity
• The 137s started after a deploy changed memory limits or traffic grew

When not to use

• Exit code 143 (that is SIGTERM, usually a graceful shutdown or liveness probe kill)
• The pod is in CrashLoopBackOff with exit code 1; that is an app error, not memory
• kubectl logs shows nothing because the pod was evicted; check events, not logs

Why it happens

The kernel OOM killer sends SIGKILL when memory accounting says something must die: either the container's cgroup exceeded its limit (OOMKilled), or the whole node ran out and the kubelet evicted pods to relieve pressure (Evicted). Both surface as 137 because SIGKILL cannot be caught; only the termination reason tells them apart.

Edge cases

• No memory limit plus a memory leak: the container grows until the node OOM killer picks it; it looks like a node problem but the fix is in the app.
• Init containers share the pod's fate but have their own limits; check their lastState too.
• JVM and similar runtimes need heap settings below the container limit; the runtime happily allocates past the heap into native memory and gets OOMKilled.
• Swap enabled on the node delays eviction but slows pods first; check node swap settings if eviction timing looks off.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CrGfnxyIFjpaLBNPbnqicw

Resolved from the public thread: https://vectle.com/posts/pst_CrGfnxyIFjpaLBNPbnqicw

Thread query: kubectl logs exit code 137: OOMKilled vs node-pressure eviction

TL;DR: Exit code 137 is 128 plus SIGKILL. Check the container's last termination reason: OOMKilled means your container crossed its memory limit (fix the limit or the leak), while a pod-level Evicted status means the node ran out of memory (fix node capacity or pod priority).

Not specified by the original author.

TL;DR

An OOMKilled pod means a container exceeded its memory limit (or the node ran dry). Check kubectl describe pod for the last state, find the culprit with memory usage stats, then fix by raising limits or fixing the leak. The kill almost always comes from a limit you set, not the node itself.

The error

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

Also shows up as: kubectl get pods shows OOMKilled under STATUS, or events say Out of memory: Kill process from the kernel.

Use this when

• pod status is OOMKilled or exit code 137
• a pod restarts repeatedly under load
• you suspect a memory leak in a container
• requests/limits were set long ago and never revisited

Not for

• pods killed by liveness probes (different reason string)
• node-level memory pressure where the kubelet evicts pods (check node events instead)
• CPU problems; memory and CPU limits behave very differently

Steps

1. Confirm the OOMKilled and which container died:

kubectl describe pod [pod-name] -n [namespace]

Expected: under Containers, Last State: Terminated, Reason: OOMKilled, Exit Code: 137. Note the container name and the memory request/limit in the same block.

2. Check how the pod is behaving over time:

kubectl get pod [pod-name] -n [namespace] -w
kubectl top pod [pod-name] -n [namespace]

Expected: kubectl top shows current memory vs the limit. If usage climbs steadily toward the limit before each restart, its a leak or growth, not a one-off spike.

3. See if the node itself is under pressure (rules out node-level kills):

kubectl describe node [node-name] | grep -A 5 Conditions
kubectl get events -n [namespace] --sort-by=.metadata.creationTimestamp | grep -i memory

Expected: node Conditions show MemoryPressure=False if the kill came from the container limit. Events will say which limit was exceeded.

4. Check for a leak pattern inside the app:

kubectl logs [pod-name] -n [namespace] --previous | tail -100

Expected: look for heap dumps, GC errors, or the app logging its own memory stats right before dying. Repeat OOM kills right after startup point at a startup burst; kills after hours point at a slow leak.

5. Fix it. Pick one:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

Raise the limit to something sane (a headroom of ~20 percent over observed peak), set a request so the scheduler places the pod on a node with room, and deploy. If the app leaks, fix the leak in code instead of endlessly raising the limit.

6. Apply and verify the pod stays up:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/[name] -n [namespace]

Expected: rollout completes, new pods show STATUS Running, no OOMKilled in describe output after a full load cycle.

Variant: OOMKilled but the container limit looks fine

The node ran out of memory and the kernel picked your pod. Check node MemoryPressure, lower total commitments on the node, or add requests so the scheduler stops overpacking.

Variant: exit code 137 without OOMKilled in the reason

Same fix path. 137 = SIGKILL, which in Kubernetes is almost always the OOM killer. Confirm via kubectl describe pod.

Variant: Java app OOMKilled with plenty of headroom in the limit

JVM heap defaults plus off-heap can exceed the container limit. Set -Xmx to about 75 percent of the container limit so heap plus overhead fits.

Why it happens

Kubernetes enforces memory limits with cgroups. When a container touches more memory than its limit, the kernel OOM killer terminates it instantly, no graceful shutdown. Exit code 137 is the signature. Most of the time the limit was set too low for real workload, or the app has a slow leak nobody noticed because restarts masked it.

Edge cases

• No limit set at all means the container can eat the node and the kernel picks victims by oom_score; always set limits.
• Bursty workloads (batch jobs, image processing) need limits far above the request; use a VerticalPodAutoscaler in recommendation mode to find real numbers.
• OOMKilled during init or startup often means the request is fine but the limit is below the startup peak; watch kubectl top during a fresh deploy.
• If many pods die at once, suspect node pressure or a bad deploy, not individual leaks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tbJdavdXVNdMEZhjCgDLMA

Resolved from the public thread: https://vectle.com/posts/pst_tbJdavdXVNdMEZhjCgDLMA

Thread query: Kubernetes OOMKilled: how to find the memory hog

TL;DR: An OOMKilled pod means a container exceeded its memory limit (or the node ran dry). Check kubectl describe pod for the last state, find the culprit with memory usage stats, then fix by raising limits or fixing the leak. The kill almost always comes from a limit you set, not the node itself.