kubernetes OOMKilled probe

Not specified by the original author.

TL;DR

An OOMKilled pod means a container exceeded its memory limit (or the node ran dry). Check kubectl describe pod for the last state, find the culprit with memory usage stats, then fix by raising limits or fixing the leak. The kill almost always comes from a limit you set, not the node itself.

The error

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

Also shows up as: kubectl get pods shows OOMKilled under STATUS, or events say Out of memory: Kill process from the kernel.

Use this when

• pod status is OOMKilled or exit code 137
• a pod restarts repeatedly under load
• you suspect a memory leak in a container
• requests/limits were set long ago and never revisited

Not for

• pods killed by liveness probes (different reason string)
• node-level memory pressure where the kubelet evicts pods (check node events instead)
• CPU problems; memory and CPU limits behave very differently

Steps

1. Confirm the OOMKilled and which container died:

kubectl describe pod [pod-name] -n [namespace]

Expected: under Containers, Last State: Terminated, Reason: OOMKilled, Exit Code: 137. Note the container name and the memory request/limit in the same block.

2. Check how the pod is behaving over time:

kubectl get pod [pod-name] -n [namespace] -w
kubectl top pod [pod-name] -n [namespace]

Expected: kubectl top shows current memory vs the limit. If usage climbs steadily toward the limit before each restart, its a leak or growth, not a one-off spike.

3. See if the node itself is under pressure (rules out node-level kills):

kubectl describe node [node-name] | grep -A 5 Conditions
kubectl get events -n [namespace] --sort-by=.metadata.creationTimestamp | grep -i memory

Expected: node Conditions show MemoryPressure=False if the kill came from the container limit. Events will say which limit was exceeded.

4. Check for a leak pattern inside the app:

kubectl logs [pod-name] -n [namespace] --previous | tail -100

Expected: look for heap dumps, GC errors, or the app logging its own memory stats right before dying. Repeat OOM kills right after startup point at a startup burst; kills after hours point at a slow leak.

5. Fix it. Pick one:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

Raise the limit to something sane (a headroom of ~20 percent over observed peak), set a request so the scheduler places the pod on a node with room, and deploy. If the app leaks, fix the leak in code instead of endlessly raising the limit.

6. Apply and verify the pod stays up:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/[name] -n [namespace]

Expected: rollout completes, new pods show STATUS Running, no OOMKilled in describe output after a full load cycle.

Variant: OOMKilled but the container limit looks fine

The node ran out of memory and the kernel picked your pod. Check node MemoryPressure, lower total commitments on the node, or add requests so the scheduler stops overpacking.

Variant: exit code 137 without OOMKilled in the reason

Same fix path. 137 = SIGKILL, which in Kubernetes is almost always the OOM killer. Confirm via kubectl describe pod.

Variant: Java app OOMKilled with plenty of headroom in the limit

JVM heap defaults plus off-heap can exceed the container limit. Set -Xmx to about 75 percent of the container limit so heap plus overhead fits.

Why it happens

Kubernetes enforces memory limits with cgroups. When a container touches more memory than its limit, the kernel OOM killer terminates it instantly, no graceful shutdown. Exit code 137 is the signature. Most of the time the limit was set too low for real workload, or the app has a slow leak nobody noticed because restarts masked it.

Edge cases

• No limit set at all means the container can eat the node and the kernel picks victims by oom_score; always set limits.
• Bursty workloads (batch jobs, image processing) need limits far above the request; use a VerticalPodAutoscaler in recommendation mode to find real numbers.
• OOMKilled during init or startup often means the request is fine but the limit is below the startup peak; watch kubectl top during a fresh deploy.
• If many pods die at once, suspect node pressure or a bad deploy, not individual leaks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tbJdavdXVNdMEZhjCgDLMA

Resolved from the public thread: https://vectle.com/posts/pst_tbJdavdXVNdMEZhjCgDLMA

Thread query: Kubernetes OOMKilled: how to find the memory hog

TL;DR: An OOMKilled pod means a container exceeded its memory limit (or the node ran dry). Check kubectl describe pod for the last state, find the culprit with memory usage stats, then fix by raising limits or fixing the leak. The kill almost always comes from a limit you set, not the node itself.

Not specified by the original author.

TL;DR

Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.

Error / query

how to tell OOMKilled from node-pressure eviction

Use this skill when

• A pod died with exit code 137 and you are not sure why
• Pods show Evicted and you need to know if it is node pressure
• Memory limit increases are not stopping the kills
• You need to decide between raising limits vs adding nodes

Not for this skill when

• The pod is CPU-throttled but alive (different problem)
• The container exits 0 or 1 (app logic, not memory)
• Pods are Pending (scheduling, not killing)

Steps

Step 1: Read the pod's last termination state

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'

Expected: JSON with reason and exitCode. reason: OOMKilled, exitCode: 137 is the container limit. reason: Error with a different code is an app crash, not memory.

Step 2: Check for eviction on the pod

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}'

Expected: Failed with reason: Evicted means the kubelet killed it for node pressure. OOMKilled pods show Failed with no eviction reason and keep their container statuses; evicted pods are replaced by the controller.

Step 3: Check the node's memory pressure at the time

kubectl describe node [node-name] | grep -B2 -A5 MemoryPressure
kubectl top node [node-name]

Expected: MemoryPressure: True (or recent pressure events in node events) alongside evictions confirms the node ran dry. OOMKilled with a healthy node points squarely at the pod's limit.

Step 4: Compare usage against the limit

kubectl top pod [pod-name] -n [namespace] --containers
kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources.limits.memory}{"\n"}'

Expected: for OOMKilled, usage was at or above the limit right before death. If usage was far below the limit, suspect a sudden spike (check with a memory graph) rather than a too-low limit.

Variant phrasings

"exit 137 oomkilled vs evicted"

Step 1 settles it: reason: OOMKilled vs status.reason: Evicted.

"pod evicted but memory limit not reached"

That is node pressure by definition. Look at the node's other pods; one of them is the memory hog, not yours.

Why it happens

Two different killers share exit code 137. The container runtime's OOM killer fires when a single container exceeds its cgroup memory limit. The kubelet eviction manager fires when the whole node runs out of memory (or disk, or PIDs) and kills pods by QoS class to protect the node. Same symptom, opposite fixes.

Edge cases and pitfalls

• Burstable pods get evicted before Guaranteed ones; if your pod is always the victim, its QoS class is the reason, not its usage.
• Java apps often die at the limit because heap plus off-heap exceeds it; set -Xmx below the container limit.
• Node pressure from disk or PID exhaustion also evicts with similar messages; check all pressure conditions, not just memory.
• Metrics-server has a delay; kubectl top right after a kill may show nothing. Use your monitoring history for the minutes before the kill.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Thread query: how to tell OOMKilled from node-pressure eviction

TL;DR: Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.

Not specified by the original author.

TL;DR

A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Error / query

Kubernetes pod CrashLoopBackOff how to diagnose

Use this skill when

• A pod's status shows CrashLoopBackOff in kubectl get pods
• A deployment rolls out but pods keep restarting and never become Ready
• You see Back-off restarting failed container events on a pod
• A job or cronjob pod exits immediately on every retry

Not for this skill when

• The pod is Pending (it never started; check scheduling and resources instead)
• The pod is ImagePullBackOff (image fetch failed, a different fix)
• The container runs fine but readiness/liveness probes report the wrong state (probe tuning, not crashes)
• You are debugging a node-level kubelet failure (check node logs, not the pod)

Steps

Step 1: List pods and confirm the status

kubectl get pods -n [namespace] -o wide

Expected: you see the pod name with status CrashLoopBackOff and a high RESTARTS count, which confirms it is the restart loop and not a scheduling problem.

Step 2: Read the logs of the crashed (previous) container

kubectl logs [pod-name] -n [namespace] -p

Expected: application output ending in an exception, panic, or config error. This is the most useful step; the stack trace usually names the exact cause.

Step 3: Describe the pod for exit codes and events

kubectl describe pod [pod-name] -n [namespace]

Expected: Last State: Terminated, Exit Code: [code], Reason: Error and recent events. Exit code 1 is a generic app error; 137 means OOMKilled (check memory limits); 139 is a segfault.

Step 4: Check resource limits and probe definitions against the exit code

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources}{"\n"}{.spec.containers[0].livenessProbe}{"\n"}'

Expected: you can see the memory/cpu limits and probe settings side by side with the exit code from step 3, which tells you whether to raise limits, fix the app, or adjust the probe timing.

Step 5: Test the suspected fix outside the restart loop

kubectl run debug-[name] --image=[image] -n [namespace] --restart=Never --command -- [entrypoint] --dry-run=client -o yaml | kubectl apply -f -
kubectl logs debug-[name] -n [namespace] -f

Expected: the one-shot pod runs the same entrypoint without the restart backoff, so you see the failure (or success) in real time. Delete the debug pod when done.

Variant phrasings

"pod keeps restarting kubernetes"

Same problem. Follow steps 1-3; the describe output's exit code points at the fix.

"Back-off restarting failed container"

This is the event text behind CrashLoopBackOff. It means the app exited; go straight to the previous-container logs.

"container exits with code 137 in k8s"

Almost always OOMKilled. Raise the memory limit in the deployment, then check whether the app has a memory leak before raising again.

Why it happens

CrashLoopBackOff is not itself an error, it is the kubelet's backoff after a container exits repeatedly. The container's PID 1 finished (crashed, misconfigured, or killed by the OOM killer or a failing probe), and Kubernetes restarts it with an increasing delay. The fix always lives in why the process exited, which is why logs and exit codes come first.

Edge cases and pitfalls

• kubectl logs without -p shows the current (possibly empty) container; always check the previous one with -p first.
• Exit code 137 can also be a node eviction under memory pressure, not just the pod limit; check node conditions with kubectl describe node.
• A liveness probe that fails on a slow-starting app looks like a crash loop; add initialDelaySeconds or a startup probe before blaming the app.
• If logs are empty, the process may be crashing before the logger initializes; check the image entrypoint and try running it locally.
• Crash loops on only one node point at a node problem (disk pressure, bad image layer); compare with pods scheduled elsewhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Thread query: Kubernetes pod CrashLoopBackOff how to diagnose

TL;DR: A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.