Kubernetes pod CrashLoopBackOff how to diagnose

TL;DR

A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Error / query

Kubernetes pod CrashLoopBackOff how to diagnose

Use this skill when

• A pod's status shows CrashLoopBackOff in kubectl get pods
• A deployment rolls out but pods keep restarting and never become Ready
• You see Back-off restarting failed container events on a pod
• A job or cronjob pod exits immediately on every retry

Not for this skill when

• The pod is Pending (it never started; check scheduling and resources instead)
• The pod is ImagePullBackOff (image fetch failed, a different fix)
• The container runs fine but readiness/liveness probes report the wrong state (probe tuning, not crashes)
• You are debugging a node-level kubelet failure (check node logs, not the pod)

Steps

Step 1: List pods and confirm the status

kubectl get pods -n [namespace] -o wide

Expected: you see the pod name with status CrashLoopBackOff and a high RESTARTS count, which confirms it is the restart loop and not a scheduling problem.

Step 2: Read the logs of the crashed (previous) container

kubectl logs [pod-name] -n [namespace] -p

Expected: application output ending in an exception, panic, or config error. This is the most useful step; the stack trace usually names the exact cause.

Step 3: Describe the pod for exit codes and events

kubectl describe pod [pod-name] -n [namespace]

Expected: Last State: Terminated, Exit Code: [code], Reason: Error and recent events. Exit code 1 is a generic app error; 137 means OOMKilled (check memory limits); 139 is a segfault.

Step 4: Check resource limits and probe definitions against the exit code

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources}{"\n"}{.spec.containers[0].livenessProbe}{"\n"}'

Expected: you can see the memory/cpu limits and probe settings side by side with the exit code from step 3, which tells you whether to raise limits, fix the app, or adjust the probe timing.

Step 5: Test the suspected fix outside the restart loop

kubectl run debug-[name] --image=[image] -n [namespace] --restart=Never --command -- [entrypoint] --dry-run=client -o yaml | kubectl apply -f -
kubectl logs debug-[name] -n [namespace] -f

Expected: the one-shot pod runs the same entrypoint without the restart backoff, so you see the failure (or success) in real time. Delete the debug pod when done.

Variant phrasings

"pod keeps restarting kubernetes"

Same problem. Follow steps 1-3; the describe output's exit code points at the fix.

"Back-off restarting failed container"

This is the event text behind CrashLoopBackOff. It means the app exited; go straight to the previous-container logs.

"container exits with code 137 in k8s"

Almost always OOMKilled. Raise the memory limit in the deployment, then check whether the app has a memory leak before raising again.

Why it happens

CrashLoopBackOff is not itself an error, it is the kubelet's backoff after a container exits repeatedly. The container's PID 1 finished (crashed, misconfigured, or killed by the OOM killer or a failing probe), and Kubernetes restarts it with an increasing delay. The fix always lives in why the process exited, which is why logs and exit codes come first.

Edge cases and pitfalls

• kubectl logs without -p shows the current (possibly empty) container; always check the previous one with -p first.
• Exit code 137 can also be a node eviction under memory pressure, not just the pod limit; check node conditions with kubectl describe node.
• A liveness probe that fails on a slow-starting app looks like a crash loop; add initialDelaySeconds or a startup probe before blaming the app.
• If logs are empty, the process may be crashing before the logger initializes; check the image entrypoint and try running it locally.
• Crash loops on only one node point at a node problem (disk pressure, bad image layer); compare with pods scheduled elsewhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q