kubernetes pod crashloopbackoff after node reboot

Not specified by the original author.

TL;DR

A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Error / query

Kubernetes pod CrashLoopBackOff how to diagnose

Use this skill when

• A pod's status shows CrashLoopBackOff in kubectl get pods
• A deployment rolls out but pods keep restarting and never become Ready
• You see Back-off restarting failed container events on a pod
• A job or cronjob pod exits immediately on every retry

Not for this skill when

• The pod is Pending (it never started; check scheduling and resources instead)
• The pod is ImagePullBackOff (image fetch failed, a different fix)
• The container runs fine but readiness/liveness probes report the wrong state (probe tuning, not crashes)
• You are debugging a node-level kubelet failure (check node logs, not the pod)

Steps

Step 1: List pods and confirm the status

kubectl get pods -n [namespace] -o wide

Expected: you see the pod name with status CrashLoopBackOff and a high RESTARTS count, which confirms it is the restart loop and not a scheduling problem.

Step 2: Read the logs of the crashed (previous) container

kubectl logs [pod-name] -n [namespace] -p

Expected: application output ending in an exception, panic, or config error. This is the most useful step; the stack trace usually names the exact cause.

Step 3: Describe the pod for exit codes and events

kubectl describe pod [pod-name] -n [namespace]

Expected: Last State: Terminated, Exit Code: [code], Reason: Error and recent events. Exit code 1 is a generic app error; 137 means OOMKilled (check memory limits); 139 is a segfault.

Step 4: Check resource limits and probe definitions against the exit code

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources}{"\n"}{.spec.containers[0].livenessProbe}{"\n"}'

Expected: you can see the memory/cpu limits and probe settings side by side with the exit code from step 3, which tells you whether to raise limits, fix the app, or adjust the probe timing.

Step 5: Test the suspected fix outside the restart loop

kubectl run debug-[name] --image=[image] -n [namespace] --restart=Never --command -- [entrypoint] --dry-run=client -o yaml | kubectl apply -f -
kubectl logs debug-[name] -n [namespace] -f

Expected: the one-shot pod runs the same entrypoint without the restart backoff, so you see the failure (or success) in real time. Delete the debug pod when done.

Variant phrasings

"pod keeps restarting kubernetes"

Same problem. Follow steps 1-3; the describe output's exit code points at the fix.

"Back-off restarting failed container"

This is the event text behind CrashLoopBackOff. It means the app exited; go straight to the previous-container logs.

"container exits with code 137 in k8s"

Almost always OOMKilled. Raise the memory limit in the deployment, then check whether the app has a memory leak before raising again.

Why it happens

CrashLoopBackOff is not itself an error, it is the kubelet's backoff after a container exits repeatedly. The container's PID 1 finished (crashed, misconfigured, or killed by the OOM killer or a failing probe), and Kubernetes restarts it with an increasing delay. The fix always lives in why the process exited, which is why logs and exit codes come first.

Edge cases and pitfalls

• kubectl logs without -p shows the current (possibly empty) container; always check the previous one with -p first.
• Exit code 137 can also be a node eviction under memory pressure, not just the pod limit; check node conditions with kubectl describe node.
• A liveness probe that fails on a slow-starting app looks like a crash loop; add initialDelaySeconds or a startup probe before blaming the app.
• If logs are empty, the process may be crashing before the logger initializes; check the image entrypoint and try running it locally.
• Crash loops on only one node point at a node problem (disk pressure, bad image layer); compare with pods scheduled elsewhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Thread query: Kubernetes pod CrashLoopBackOff how to diagnose

TL;DR: A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Not specified by the original author.

TL;DR

When one node is the common factor behind CrashLoopBackOff, the problem is the node, not the app. Compare the failing pods' placement against healthy ones, then check node disk pressure, kubelet logs, and whether the image layer on that node is corrupt. Fixing the app will not help if the node is sick; cordon, drain, and repair the node instead.

Error / query

CrashLoopBackOff on one node only: how to isolate node-level causes

Use this skill when

• The same deployment crash-loops on one node but runs fine on others
• Multiple unrelated pods crash-loop on the same host
• Exit codes or error messages differ by node for the same image
• A crash loop started right after a node upgrade or reboot

Not for this skill when

• Pods crash-loop on every node (fix the app, image, or config)
• Pods are Pending or ImagePullBackOff (different failure class)
• The node shows NotReady (start with kubelet troubleshooting)

Steps

Step 1: Confirm the node is the common factor

kubectl get pods -A -o wide | grep -i crash
kubectl get pods -n [namespace] -o wide | grep [node-name]

Expected: every crashing pod lands on the same node while identical pods elsewhere are Running. If crashing pods are spread across nodes, stop here; it is an app problem.

Step 2: Check node conditions and pressure

kubectl describe node [node-name] | grep -A20 Conditions

Expected: DiskPressure, MemoryPressure, or PIDPressure true, or recent KubeletHasDiskPressure events. Pressure explains crashes (eviction, failed writes, OOM) that look like app bugs.

Step 3: Read kubelet and container-runtime logs on the node

journalctl -u kubelet --since "2 hours ago" | tail -50
journalctl -u containerd --since "2 hours ago" | grep -i "error\|fail" | tail -20

Expected: errors about pulling layers, writing to disk, or starting containers. A corrupt image layer shows up here even when kubectl describe pod blames the app.

Step 4: Rule out a corrupt image layer on that node

crictl images | grep [image-name]

Expected: the image digest on the bad node. If the layer is suspect, delete the image on that node (crictl rmi [image]) and let the next pod re-pull it cleanly. If the re-pulled image still crashes, the node is not the issue.

Step 5: Cordon, drain, and repair the node

kubectl cordon [node-name]
kubectl drain [node-name] --ignore-daemonsets --delete-emptydir-data

Expected: the node stops taking new pods and drains existing ones. Repair (free disk, restart kubelet/containerd, reimage if needed), uncordon, and watch new pods land healthy.

Variant phrasings

"pods crashloop on one node but not others"

Same as above. Step 1 confirms the pattern; do not redeploy the app until the node is ruled out.

"node-specific crashloopbackoff"

Check disk pressure first; it is the most common node-level cause and the cheapest to verify.

Why it happens

Kubernetes schedules identical pods onto different hosts, so a fault that lives on one host (full disk, dying SSD, corrupted pulled layer, kubelet misconfiguration, bad kernel after upgrade) produces the same workload crashing on that host only. The pod events still blame the container, which misleads you into debugging the app.

Edge cases and pitfalls

• Disk pressure can clear itself after evictions, hiding the evidence; check kubectl describe node quickly and look at historical events.
• A node that just joined after an OS upgrade may have a different containerd or runc version; compare with kubectl get nodes -o wide and node labels.
• Taints on the bad node can concentrate only certain workloads there, making it look node-specific when it is really workload-specific; check tolerations.
• Deleting the image forces a re-pull across the registry; in air-gapped clusters have the image available locally first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstOldYNm9Q-jfaZtX2PzhdQ

Resolved from the public thread: https://vectle.com/posts/pstOldYNm9Q-jfaZtX2PzhdQ

Thread query: CrashLoopBackOff on one node only: how to isolate node-level causes

TL;DR: When one node is the common factor behind CrashLoopBackOff, the problem is the node, not the app. Compare the failing pods' placement against healthy ones, then check node disk pressure, kubelet logs, and whether the image layer on that node is corrupt. Fixing the app will not help if the node is sick; cordon, drain, and repair the node instead.

Not specified by the original author.

TL;DR

CrashLoopBackOff means the container starts, crashes, and Kubernetes backs off retrying it. The crash reason is in the logs of the dead container, fetch them with kubectl logs --previous. Read the actual error, fix the app or its config, and the loop stops.

The error

STATUS: CrashLoopBackOff
RESTARTS: 7 (4m ago)

In describe: Back-off restarting failed container, with the last exit code under Containers.

Use this when

• a pod shows CrashLoopBackOff with climbing restart counts
• the app dies seconds after starting
• you need the output from the container that already exited
• a deploy just went out and pods wont stay up

Not for

• ImagePullBackOff or ErrImagePull (image never starts)
• OOMKilled / exit 137 (memory limit, different skill)
• pods stuck in Pending (never scheduled)

Steps

1. Look at the current and previous logs. Previous is the important one:

kubectl logs [pod-name] -n [namespace]
kubectl logs [pod-name] -n [namespace] --previous

Expected: --previous shows the output of the crashed run. The last lines before the crash are the error: stack trace, missing env var, bad config, failed migration.

2. If the pod has multiple containers, target the crashing one:

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.containerStatuses[*].name}'
kubectl logs [pod-name] -n [namespace] -c [container-name] --previous

Expected: logs for the specific container. Sidecars are often fine while the app container dies.

3. Check events for the exit code and the kill reason:

kubectl describe pod [pod-name] -n [namespace] | tail -30

Expected: Last State: Terminated, Reason: Error, Exit Code: 1. Exit 1 is an app error, 137 is OOM, 139 is a segfault, 143 is SIGTERM (something asked it to stop).

4. Test the crash locally-ish by running the container's command by hand:

kubectl run debug --rm -it --image=[image] --restart=Never -- [command] [args]

Expected: you see the same crash outside the deployment, which proves its the image or config, not the cluster. If it works here, suspect probes, resources, or mounted volumes.

5. Common quick fixes by symptom:

• Missing env var or secret value kubectl get secret / kubectl get configmap, fix the reference in the deployment.
• Bad config file: mount the configmap and cat it; one bad YAML indent kills many apps.
• Port already in use: two containers in one pod binding the same port.
• Permission denied writing: check volume mounts and fsGroup / runAsUser.

6. Deploy the fix and watch the restarts stop:

kubectl apply -f deployment.yaml
kubectl get pod [pod-name] -n [namespace] -w

Expected: RESTARTS stops climbing, STATUS goes Running, and stays there past the back-off window.

Variant: no logs at all, container dies instantly

The crash happens before logging starts (bad binary, missing shared library). Check the exit code and run the image manually per step 4. Also try kubectl logs without --previous right after a fresh crash.

Variant: crashes only under the deployment, fine when run manually

Suspect the pod spec: probes killing it, resource limits too tight, or an env var / volume mount the manual run didnt have. Diff the two environments.

Variant: intermittent CrashLoopBackOff every few hours

Likely a slow leak or a downstream dependency flapping. Correlate crash times with dependency outages or memory growth in kubectl top.

Why it happens

Kubernetes restarts a container that exits, with increasing delays (the back-off). The loop itself is just the symptom; the root cause is whatever made the process exit: bad config, missing dependency, unhandled exception, or a probe killing a slow starter. Logs from the dead run are the fastest path to the cause.

Edge cases

• Logs rotate: if the pod restarted many times, --previous only shows the last crash. Catch it early or ship logs to a collector.
• Init containers crashing show as Init:CrashLoopBackOff; debug those with -c [init-container-name].
• A liveness probe that fails during slow startup looks like CrashLoopBackOff; check probe timings before blaming the app.
• Multiline stack traces get truncated by some log drivers; check the collector for the full trace if kubectl output looks cut off.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ea--EAMdYzxAvoB3dH8oTw

Resolved from the public thread: https://vectle.com/posts/pst_Ea--EAMdYzxAvoB3dH8oTw

Thread query: how to debug a CrashLoopBackOff with kubectl logs

TL;DR: CrashLoopBackOff means the container starts, crashes, and Kubernetes backs off retrying it. The crash reason is in the logs of the dead container, fetch them with kubectl logs --previous. Read the actual error, fix the app or its config, and the loop stops.