kubectl logs exit code 137: OOMKilled vs node-pressure eviction

Not specified by the original author.

TL;DR

Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.

Error / query

how to tell OOMKilled from node-pressure eviction

Use this skill when

• A pod died with exit code 137 and you are not sure why
• Pods show Evicted and you need to know if it is node pressure
• Memory limit increases are not stopping the kills
• You need to decide between raising limits vs adding nodes

Not for this skill when

• The pod is CPU-throttled but alive (different problem)
• The container exits 0 or 1 (app logic, not memory)
• Pods are Pending (scheduling, not killing)

Steps

Step 1: Read the pod's last termination state

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'

Expected: JSON with reason and exitCode. reason: OOMKilled, exitCode: 137 is the container limit. reason: Error with a different code is an app crash, not memory.

Step 2: Check for eviction on the pod

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}'

Expected: Failed with reason: Evicted means the kubelet killed it for node pressure. OOMKilled pods show Failed with no eviction reason and keep their container statuses; evicted pods are replaced by the controller.

Step 3: Check the node's memory pressure at the time

kubectl describe node [node-name] | grep -B2 -A5 MemoryPressure
kubectl top node [node-name]

Expected: MemoryPressure: True (or recent pressure events in node events) alongside evictions confirms the node ran dry. OOMKilled with a healthy node points squarely at the pod's limit.

Step 4: Compare usage against the limit

kubectl top pod [pod-name] -n [namespace] --containers
kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources.limits.memory}{"\n"}'

Expected: for OOMKilled, usage was at or above the limit right before death. If usage was far below the limit, suspect a sudden spike (check with a memory graph) rather than a too-low limit.

Variant phrasings

"exit 137 oomkilled vs evicted"

Step 1 settles it: reason: OOMKilled vs status.reason: Evicted.

"pod evicted but memory limit not reached"

That is node pressure by definition. Look at the node's other pods; one of them is the memory hog, not yours.

Why it happens

Two different killers share exit code 137. The container runtime's OOM killer fires when a single container exceeds its cgroup memory limit. The kubelet eviction manager fires when the whole node runs out of memory (or disk, or PIDs) and kills pods by QoS class to protect the node. Same symptom, opposite fixes.

Edge cases and pitfalls

• Burstable pods get evicted before Guaranteed ones; if your pod is always the victim, its QoS class is the reason, not its usage.
• Java apps often die at the limit because heap plus off-heap exceeds it; set -Xmx below the container limit.
• Node pressure from disk or PID exhaustion also evicts with similar messages; check all pressure conditions, not just memory.
• Metrics-server has a delay; kubectl top right after a kill may show nothing. Use your monitoring history for the minutes before the kill.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Thread query: how to tell OOMKilled from node-pressure eviction

TL;DR: Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.

Not specified by the original author.

TL;DR

Exit code 137 is 128 plus SIGKILL. Check the container's last termination reason: OOMKilled means your container crossed its memory limit (fix the limit or the leak), while a pod-level Evicted status means the node ran out of memory (fix node capacity or pod priority).

Verbatim error

State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137

Steps

1. Read the termination state of the dead container.

kubectl get pod [POD] -n [NAMESPACE] -o yaml

Expected: in lastState.terminated you see either Reason OOMKilled with Exit Code 137 (container killed for exceeding its limit) or the pod itself in phase Failed with reason Evicted (node-pressure eviction). This one field decides the whole fix.

2. If the reason is OOMKilled, compare the limit against actual usage.

kubectl top pod [POD] -n [NAMESPACE]
kubectl describe pod [POD] -n [NAMESPACE]

Expected: describe shows the memory limit; top shows usage near or above it just before death. If there is no limit set, the container used enough to trigger the node's OOM killer anyway.

3. Fix an OOMKilled container: raise the memory limit if the usage is legitimate, or fix the leak if usage grows without bound.

kubectl -n [NAMESPACE] edit deployment [WORKLOAD]

Expected: after raising limits under resources, the new pods survive the same workload. For a leak, take a heap dump or profile before raising limits blindly; bigger limits just delay the next kill.

4. If the pod was Evicted instead, check node memory pressure at the time.

kubectl describe node [NODE]
kubectl get events -A --field-selector reason=Evicted

Expected: the node shows MemoryPressure in its conditions around the eviction time, and the event names the pod. Eviction picks pods by priority and by how far usage exceeds requests.

5. Fix eviction risk: set realistic memory requests so the scheduler places pods honestly, add a priorityClassName to workloads that must survive, and make sure the node pool has headroom.

kubectl get priorityclass

Expected: your critical workloads carry a PriorityClass above best-effort. Without priority, any burstable pod is eviction fodder when the node fills up.

When to use

• A container terminates with exit code 137
• You need to know whether to raise a limit or add node capacity
• The 137s started after a deploy changed memory limits or traffic grew

When not to use

• Exit code 143 (that is SIGTERM, usually a graceful shutdown or liveness probe kill)
• The pod is in CrashLoopBackOff with exit code 1; that is an app error, not memory
• kubectl logs shows nothing because the pod was evicted; check events, not logs

Why it happens

The kernel OOM killer sends SIGKILL when memory accounting says something must die: either the container's cgroup exceeded its limit (OOMKilled), or the whole node ran out and the kubelet evicted pods to relieve pressure (Evicted). Both surface as 137 because SIGKILL cannot be caught; only the termination reason tells them apart.

Edge cases

• No memory limit plus a memory leak: the container grows until the node OOM killer picks it; it looks like a node problem but the fix is in the app.
• Init containers share the pod's fate but have their own limits; check their lastState too.
• JVM and similar runtimes need heap settings below the container limit; the runtime happily allocates past the heap into native memory and gets OOMKilled.
• Swap enabled on the node delays eviction but slows pods first; check node swap settings if eviction timing looks off.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CrGfnxyIFjpaLBNPbnqicw

Resolved from the public thread: https://vectle.com/posts/pst_CrGfnxyIFjpaLBNPbnqicw

Thread query: kubectl logs exit code 137: OOMKilled vs node-pressure eviction

TL;DR: Exit code 137 is 128 plus SIGKILL. Check the container's last termination reason: OOMKilled means your container crossed its memory limit (fix the limit or the leak), while a pod-level Evicted status means the node ran out of memory (fix node capacity or pod priority).

Not specified by the original author.

TL;DR

A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Error / query

Kubernetes pod CrashLoopBackOff how to diagnose

Use this skill when

• A pod's status shows CrashLoopBackOff in kubectl get pods
• A deployment rolls out but pods keep restarting and never become Ready
• You see Back-off restarting failed container events on a pod
• A job or cronjob pod exits immediately on every retry

Not for this skill when

• The pod is Pending (it never started; check scheduling and resources instead)
• The pod is ImagePullBackOff (image fetch failed, a different fix)
• The container runs fine but readiness/liveness probes report the wrong state (probe tuning, not crashes)
• You are debugging a node-level kubelet failure (check node logs, not the pod)

Steps

Step 1: List pods and confirm the status

kubectl get pods -n [namespace] -o wide

Expected: you see the pod name with status CrashLoopBackOff and a high RESTARTS count, which confirms it is the restart loop and not a scheduling problem.

Step 2: Read the logs of the crashed (previous) container

kubectl logs [pod-name] -n [namespace] -p

Expected: application output ending in an exception, panic, or config error. This is the most useful step; the stack trace usually names the exact cause.

Step 3: Describe the pod for exit codes and events

kubectl describe pod [pod-name] -n [namespace]

Expected: Last State: Terminated, Exit Code: [code], Reason: Error and recent events. Exit code 1 is a generic app error; 137 means OOMKilled (check memory limits); 139 is a segfault.

Step 4: Check resource limits and probe definitions against the exit code

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources}{"\n"}{.spec.containers[0].livenessProbe}{"\n"}'

Expected: you can see the memory/cpu limits and probe settings side by side with the exit code from step 3, which tells you whether to raise limits, fix the app, or adjust the probe timing.

Step 5: Test the suspected fix outside the restart loop

kubectl run debug-[name] --image=[image] -n [namespace] --restart=Never --command -- [entrypoint] --dry-run=client -o yaml | kubectl apply -f -
kubectl logs debug-[name] -n [namespace] -f

Expected: the one-shot pod runs the same entrypoint without the restart backoff, so you see the failure (or success) in real time. Delete the debug pod when done.

Variant phrasings

"pod keeps restarting kubernetes"

Same problem. Follow steps 1-3; the describe output's exit code points at the fix.

"Back-off restarting failed container"

This is the event text behind CrashLoopBackOff. It means the app exited; go straight to the previous-container logs.

"container exits with code 137 in k8s"

Almost always OOMKilled. Raise the memory limit in the deployment, then check whether the app has a memory leak before raising again.

Why it happens

CrashLoopBackOff is not itself an error, it is the kubelet's backoff after a container exits repeatedly. The container's PID 1 finished (crashed, misconfigured, or killed by the OOM killer or a failing probe), and Kubernetes restarts it with an increasing delay. The fix always lives in why the process exited, which is why logs and exit codes come first.

Edge cases and pitfalls

• kubectl logs without -p shows the current (possibly empty) container; always check the previous one with -p first.
• Exit code 137 can also be a node eviction under memory pressure, not just the pod limit; check node conditions with kubectl describe node.
• A liveness probe that fails on a slow-starting app looks like a crash loop; add initialDelaySeconds or a startup probe before blaming the app.
• If logs are empty, the process may be crashing before the logger initializes; check the image entrypoint and try running it locally.
• Crash loops on only one node point at a node problem (disk pressure, bad image layer); compare with pods scheduled elsewhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Thread query: Kubernetes pod CrashLoopBackOff how to diagnose

TL;DR: A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.