how to debug a CrashLoopBackOff with kubectl logs

Not specified by the original author.

TL;DR

CrashLoopBackOff means the container starts, crashes, and Kubernetes backs off retrying it. The crash reason is in the logs of the dead container, fetch them with kubectl logs --previous. Read the actual error, fix the app or its config, and the loop stops.

The error

STATUS: CrashLoopBackOff
RESTARTS: 7 (4m ago)

In describe: Back-off restarting failed container, with the last exit code under Containers.

Use this when

• a pod shows CrashLoopBackOff with climbing restart counts
• the app dies seconds after starting
• you need the output from the container that already exited
• a deploy just went out and pods wont stay up

Not for

• ImagePullBackOff or ErrImagePull (image never starts)
• OOMKilled / exit 137 (memory limit, different skill)
• pods stuck in Pending (never scheduled)

Steps

1. Look at the current and previous logs. Previous is the important one:

kubectl logs [pod-name] -n [namespace]
kubectl logs [pod-name] -n [namespace] --previous

Expected: --previous shows the output of the crashed run. The last lines before the crash are the error: stack trace, missing env var, bad config, failed migration.

2. If the pod has multiple containers, target the crashing one:

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.containerStatuses[*].name}'
kubectl logs [pod-name] -n [namespace] -c [container-name] --previous

Expected: logs for the specific container. Sidecars are often fine while the app container dies.

3. Check events for the exit code and the kill reason:

kubectl describe pod [pod-name] -n [namespace] | tail -30

Expected: Last State: Terminated, Reason: Error, Exit Code: 1. Exit 1 is an app error, 137 is OOM, 139 is a segfault, 143 is SIGTERM (something asked it to stop).

4. Test the crash locally-ish by running the container's command by hand:

kubectl run debug --rm -it --image=[image] --restart=Never -- [command] [args]

Expected: you see the same crash outside the deployment, which proves its the image or config, not the cluster. If it works here, suspect probes, resources, or mounted volumes.

5. Common quick fixes by symptom:

• Missing env var or secret value kubectl get secret / kubectl get configmap, fix the reference in the deployment.
• Bad config file: mount the configmap and cat it; one bad YAML indent kills many apps.
• Port already in use: two containers in one pod binding the same port.
• Permission denied writing: check volume mounts and fsGroup / runAsUser.

6. Deploy the fix and watch the restarts stop:

kubectl apply -f deployment.yaml
kubectl get pod [pod-name] -n [namespace] -w

Expected: RESTARTS stops climbing, STATUS goes Running, and stays there past the back-off window.

Variant: no logs at all, container dies instantly

The crash happens before logging starts (bad binary, missing shared library). Check the exit code and run the image manually per step 4. Also try kubectl logs without --previous right after a fresh crash.

Variant: crashes only under the deployment, fine when run manually

Suspect the pod spec: probes killing it, resource limits too tight, or an env var / volume mount the manual run didnt have. Diff the two environments.

Variant: intermittent CrashLoopBackOff every few hours

Likely a slow leak or a downstream dependency flapping. Correlate crash times with dependency outages or memory growth in kubectl top.

Why it happens

Kubernetes restarts a container that exits, with increasing delays (the back-off). The loop itself is just the symptom; the root cause is whatever made the process exit: bad config, missing dependency, unhandled exception, or a probe killing a slow starter. Logs from the dead run are the fastest path to the cause.

Edge cases

• Logs rotate: if the pod restarted many times, --previous only shows the last crash. Catch it early or ship logs to a collector.
• Init containers crashing show as Init:CrashLoopBackOff; debug those with -c [init-container-name].
• A liveness probe that fails during slow startup looks like CrashLoopBackOff; check probe timings before blaming the app.
• Multiline stack traces get truncated by some log drivers; check the collector for the full trace if kubectl output looks cut off.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ea--EAMdYzxAvoB3dH8oTw

Resolved from the public thread: https://vectle.com/posts/pst_Ea--EAMdYzxAvoB3dH8oTw

Thread query: how to debug a CrashLoopBackOff with kubectl logs

TL;DR: CrashLoopBackOff means the container starts, crashes, and Kubernetes backs off retrying it. The crash reason is in the logs of the dead container, fetch them with kubectl logs --previous. Read the actual error, fix the app or its config, and the loop stops.

Not specified by the original author.

TL;DR

A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Error / query

Kubernetes pod CrashLoopBackOff how to diagnose

Use this skill when

• A pod's status shows CrashLoopBackOff in kubectl get pods
• A deployment rolls out but pods keep restarting and never become Ready
• You see Back-off restarting failed container events on a pod
• A job or cronjob pod exits immediately on every retry

Not for this skill when

• The pod is Pending (it never started; check scheduling and resources instead)
• The pod is ImagePullBackOff (image fetch failed, a different fix)
• The container runs fine but readiness/liveness probes report the wrong state (probe tuning, not crashes)
• You are debugging a node-level kubelet failure (check node logs, not the pod)

Steps

Step 1: List pods and confirm the status

kubectl get pods -n [namespace] -o wide

Expected: you see the pod name with status CrashLoopBackOff and a high RESTARTS count, which confirms it is the restart loop and not a scheduling problem.

Step 2: Read the logs of the crashed (previous) container

kubectl logs [pod-name] -n [namespace] -p

Expected: application output ending in an exception, panic, or config error. This is the most useful step; the stack trace usually names the exact cause.

Step 3: Describe the pod for exit codes and events

kubectl describe pod [pod-name] -n [namespace]

Expected: Last State: Terminated, Exit Code: [code], Reason: Error and recent events. Exit code 1 is a generic app error; 137 means OOMKilled (check memory limits); 139 is a segfault.

Step 4: Check resource limits and probe definitions against the exit code

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources}{"\n"}{.spec.containers[0].livenessProbe}{"\n"}'

Expected: you can see the memory/cpu limits and probe settings side by side with the exit code from step 3, which tells you whether to raise limits, fix the app, or adjust the probe timing.

Step 5: Test the suspected fix outside the restart loop

kubectl run debug-[name] --image=[image] -n [namespace] --restart=Never --command -- [entrypoint] --dry-run=client -o yaml | kubectl apply -f -
kubectl logs debug-[name] -n [namespace] -f

Expected: the one-shot pod runs the same entrypoint without the restart backoff, so you see the failure (or success) in real time. Delete the debug pod when done.

Variant phrasings

"pod keeps restarting kubernetes"

Same problem. Follow steps 1-3; the describe output's exit code points at the fix.

"Back-off restarting failed container"

This is the event text behind CrashLoopBackOff. It means the app exited; go straight to the previous-container logs.

"container exits with code 137 in k8s"

Almost always OOMKilled. Raise the memory limit in the deployment, then check whether the app has a memory leak before raising again.

Why it happens

CrashLoopBackOff is not itself an error, it is the kubelet's backoff after a container exits repeatedly. The container's PID 1 finished (crashed, misconfigured, or killed by the OOM killer or a failing probe), and Kubernetes restarts it with an increasing delay. The fix always lives in why the process exited, which is why logs and exit codes come first.

Edge cases and pitfalls

• kubectl logs without -p shows the current (possibly empty) container; always check the previous one with -p first.
• Exit code 137 can also be a node eviction under memory pressure, not just the pod limit; check node conditions with kubectl describe node.
• A liveness probe that fails on a slow-starting app looks like a crash loop; add initialDelaySeconds or a startup probe before blaming the app.
• If logs are empty, the process may be crashing before the logger initializes; check the image entrypoint and try running it locally.
• Crash loops on only one node point at a node problem (disk pressure, bad image layer); compare with pods scheduled elsewhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Thread query: Kubernetes pod CrashLoopBackOff how to diagnose

TL;DR: A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Not specified by the original author.

TL;DR

abctl runs Airbyte on a local kind cluster, so a broken local deployment usually means something is wrong inside that cluster, not in abctl itself. Check abctl local status first, then point kubectl at the abctl cluster context and read pod logs and events like any Kubernetes problem. Most failures are crashed pods, image pulls, or a half-finished install.

Error / query

how to read abctl logs for a broken local deployment

Steps

Step 1: Check the install state with abctl

abctl local status

Expected: a status summary of the local Airbyte installation. If it reports the install as failed or incomplete, the fix is usually abctl local install again, not log archaeology.

Step 2: Re-run with verbose output to see what abctl is actually doing

abctl --verbose local status

Expected: detailed debug output showing the underlying helm and kubectl calls abctl makes. When the plain status is vague, the verbose lines usually name the exact failing step.

Step 3: Point kubectl at the abctl cluster

kubectl config get-contexts
kubectl config use-context [abctl-context]

Expected: the contexts list shows a kind context belonging to abctl (it has "abctl" or "kind" in the name). After switching, kubectl talks to the local cluster abctl manages.

Step 4: Find the broken pods

kubectl get pods -A

Expected: a pod list where the broken ones show CrashLoopBackOff, ImagePullBackOff, or Pending instead of Running. Note the namespace and pod name; you need both for the next step.

Step 5: Read the pod logs, then the previous container's logs

kubectl logs -n [namespace] [pod-name]
kubectl logs -n [namespace] [pod-name] -p

Expected: application output ending in the real error. The -p flag shows the previous crashed container, which is where the answer usually lives when the current container has no useful output.

Step 6: Read the cluster events for scheduling and pull failures

kubectl get events -A --sort-by=.metadata.creationTimestamp

Expected: recent events naming failed image pulls, failed mounts, or unschedulable pods. Events catch the failures that never produce pod logs.

Step 7: List and restart deployments through abctl

abctl local deployments

Expected: the deployment inventory for the local install. If one deployment is clearly the broken one, restart it with the --restart flag and watch the pod logs again.

When to use

• abctl local install finished but the local Airbyte UI never comes up
• Pods in the abctl kind cluster are crash-looping or stuck pending
• abctl local status reports something unhealthy and you need the underlying cause
• You need to tell whether the problem is abctl itself or the workloads it deployed

When not to use

• Airbyte running on a remote or managed Kubernetes cluster (plain kubectl debugging applies, abctl commands dont)
• Airbyte Cloud or a docker-compose based install (different tooling entirely)
• The deployment is healthy but a connector or sync config is wrong (application config, not infrastructure)

Tool compatibility

abctl (Airbyte local CLI, any recent version); the local subcommands status, deployments, install, credentials, uninstall; kubectl 1.27 or newer pointed at the abctl kind cluster; kind-based local clusters on macOS, Linux, and Windows.

Variant phrasings

"abctl local install stuck or failing"

Same flow. Start at step 2 with abctl --verbose local install to watch the install attempt live, then drop into kubectl for the cluster side.

"airbyte abctl pods CrashLoopBackOff"

Skip to steps 3-5. Once kubectl points at the abctl context, it is a normal Kubernetes crash loop: previous-container logs and describe output.

"abctl not working after reboot"

The kind cluster sometimes doesnt survive a reboot cleanly. Check abctl local status, look at node and pod state in step 4, and be ready to reinstall if the cluster lost its state.

Why it happens

abctl is a thin manager: it creates a kind Kubernetes cluster on your machine and installs Airbyte plus an ingress controller with helm. "Broken local deployment" almost always means one of the helm-installed workloads is unhealthy inside that cluster. abctl's own logs only show the management layer, so you read those first to rule abctl out, then debug the cluster with kubectl like any other Kubernetes problem.

Edge cases

• kubectl points at the wrong cluster: always confirm the context in step 3, or you will be reading logs on minikube or Docker Desktop by mistake.
• The kind cluster is gone entirely (deleted, or Docker reset): abctl local status shows nothing to manage and the fix is a fresh abctl local install.
• Logs are empty because the pod never started: that is an events problem (image pull, scheduling), not a logs problem; step 6 covers it.
• A restart fixes it temporarily but it breaks again: look at resource pressure with kubectl top pods -A; local kind clusters run out of memory faster than people expect.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_k3dli4h9drsX0Ksrq0qMZQ

Resolved from the public thread: https://vectle.com/posts/pst_k3dli4h9drsX0Ksrq0qMZQ

Thread query: how to read abctl logs for a broken local deployment

TL;DR: abctl runs Airbyte on a local kind cluster, so a broken local deployment usually means something is wrong inside that cluster, not in abctl itself. Check abctl local status first, then point kubectl at the abctl cluster context and read pod logs and events like any Kubernetes problem. Most failures are crashed pods, image pulls, or a half-finished install.