Cilium bpf: map creation failed / endpoint not ready: how to debug

Not specified by the original author.

TL;DR

The cilium-agent cannot allocate the BPF maps it needs, usually because the kernel is too old, the BPF filesystem is not mounted, or map memory is exhausted. Check the agent logs, then the kernel and bpffs on the node; the pod networking symptoms are downstream of the agent.

Verbatim error

bpf: map creation failed: [MAP-NAME]: no space left on device

Steps

1. Read the agent logs on the affected node for the exact map failure.

kubectl logs -n kube-system -l k8s-app=cilium --tail=200

Expected: lines naming the map that failed creation and the errno, for example no space left on device (map memory exhausted) or operation not permitted (kernel or mount problem). The errno picks the next step.

2. Check the kernel version on the node against Cilium's requirements.

kubectl get node [NODE] -o yaml

Expected: the kubelet-reported kernel version. Cilium needs a reasonably modern kernel for its BPF features; an old or custom kernel that lacks BPF support fails map creation at agent start.

3. Verify the BPF filesystem is mounted on the node.

mount | grep bpf

Expected: a bpffs mount, typically at /sys/fs/bpf. If it is missing, the agent has nowhere to pin maps; mount it persistently via your node provisioning. Run this on the node via your node access method.

4. Check for map pressure: too many endpoints or stale maps exhausting the map memory.

kubectl -n kube-system exec ds/cilium -- cilium bpf lb list

Expected: the command runs. If maps are full, look for leaked endpoints from crashed pods and restart the agent to reclaim; persistent pressure means raising the bpf map size limits in the Cilium config.

5. After fixing the node, restart the agent and watch endpoints become ready.

kubectl -n kube-system rollout restart ds/cilium
kubectl -n kube-system exec ds/cilium -- cilium status

Expected: cilium status shows the agent healthy and endpoints transitioning to ready. Pods on the node regain networking as their endpoints program.

When to use

• Cilium endpoints stuck not ready with bpf map creation failed in agent logs
• Pod networking breaks on specific nodes while others are fine
• The failure started after a kernel upgrade, node image change, or Cilium upgrade

When not to use

• All nodes lose networking at once; that is usually a control-plane or config rollout issue
• NetworkPolicy drops traffic between ready endpoints; the datapath works, policy does not
• DNS fails but pod-to-pod IP traffic works; that is a DNS problem, not BPF maps

Why it happens

Cilium programs networking as BPF maps pinned on bpffs: connection tracking, load balancing, policy, endpoints. Map creation needs kernel BPF support, a mounted bpffs, and free entries within the map size limits. Any of those missing makes the agent unable to program the datapath, so endpoints never become ready.

Edge cases

• Nodes with huge pod counts exhaust default map sizes; raise the per-map limits in the Helm values before the node fills.
• Custom kernels with BPF compiled out fail every map creation; the fix is a kernel rebuild or a different node image.
• Stale maps from a crashed agent can block a new agent from pinning; a node reboot clears them when restart does not.
• eBPF JIT disabled on some hardened kernels degrades or breaks datapath features; check the JIT setting if maps create but traffic misbehaves.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_c7G2J3wbql0HMUpPUpnrxw

Resolved from the public thread: https://vectle.com/posts/pst_c7G2J3wbql0HMUpPUpnrxw

Thread query: Cilium "bpf: map creation failed" / endpoint not ready: how to debug

TL;DR: The cilium-agent cannot allocate the BPF maps it needs, usually because the kernel is too old, the BPF filesystem is not mounted, or map memory is exhausted. Check the agent logs, then the kernel and bpffs on the node; the pod networking symptoms are downstream of the agent.

Not specified by the original author.

TL;DR

A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Error / query

Kubernetes pod CrashLoopBackOff how to diagnose

Use this skill when

• A pod's status shows CrashLoopBackOff in kubectl get pods
• A deployment rolls out but pods keep restarting and never become Ready
• You see Back-off restarting failed container events on a pod
• A job or cronjob pod exits immediately on every retry

Not for this skill when

• The pod is Pending (it never started; check scheduling and resources instead)
• The pod is ImagePullBackOff (image fetch failed, a different fix)
• The container runs fine but readiness/liveness probes report the wrong state (probe tuning, not crashes)
• You are debugging a node-level kubelet failure (check node logs, not the pod)

Steps

Step 1: List pods and confirm the status

kubectl get pods -n [namespace] -o wide

Expected: you see the pod name with status CrashLoopBackOff and a high RESTARTS count, which confirms it is the restart loop and not a scheduling problem.

Step 2: Read the logs of the crashed (previous) container

kubectl logs [pod-name] -n [namespace] -p

Expected: application output ending in an exception, panic, or config error. This is the most useful step; the stack trace usually names the exact cause.

Step 3: Describe the pod for exit codes and events

kubectl describe pod [pod-name] -n [namespace]

Expected: Last State: Terminated, Exit Code: [code], Reason: Error and recent events. Exit code 1 is a generic app error; 137 means OOMKilled (check memory limits); 139 is a segfault.

Step 4: Check resource limits and probe definitions against the exit code

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources}{"\n"}{.spec.containers[0].livenessProbe}{"\n"}'

Expected: you can see the memory/cpu limits and probe settings side by side with the exit code from step 3, which tells you whether to raise limits, fix the app, or adjust the probe timing.

Step 5: Test the suspected fix outside the restart loop

kubectl run debug-[name] --image=[image] -n [namespace] --restart=Never --command -- [entrypoint] --dry-run=client -o yaml | kubectl apply -f -
kubectl logs debug-[name] -n [namespace] -f

Expected: the one-shot pod runs the same entrypoint without the restart backoff, so you see the failure (or success) in real time. Delete the debug pod when done.

Variant phrasings

"pod keeps restarting kubernetes"

Same problem. Follow steps 1-3; the describe output's exit code points at the fix.

"Back-off restarting failed container"

This is the event text behind CrashLoopBackOff. It means the app exited; go straight to the previous-container logs.

"container exits with code 137 in k8s"

Almost always OOMKilled. Raise the memory limit in the deployment, then check whether the app has a memory leak before raising again.

Why it happens

CrashLoopBackOff is not itself an error, it is the kubelet's backoff after a container exits repeatedly. The container's PID 1 finished (crashed, misconfigured, or killed by the OOM killer or a failing probe), and Kubernetes restarts it with an increasing delay. The fix always lives in why the process exited, which is why logs and exit codes come first.

Edge cases and pitfalls

• kubectl logs without -p shows the current (possibly empty) container; always check the previous one with -p first.
• Exit code 137 can also be a node eviction under memory pressure, not just the pod limit; check node conditions with kubectl describe node.
• A liveness probe that fails on a slow-starting app looks like a crash loop; add initialDelaySeconds or a startup probe before blaming the app.
• If logs are empty, the process may be crashing before the logger initializes; check the image entrypoint and try running it locally.
• Crash loops on only one node point at a node problem (disk pressure, bad image layer); compare with pods scheduled elsewhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Thread query: Kubernetes pod CrashLoopBackOff how to diagnose

TL;DR: A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Not specified by the original author.

TL;DR

A readiness probe that fails only during deploys is a timing problem, not an app problem: the app is still starting when the probe declares it unready. Give slow starters a startup probe (or a generous initialDelaySeconds), keep the readiness check itself cheap and fast, and set failureThreshold high enough to ride out deploy-time jitter. Tune the probe to the app's real startup curve, not to the happy path.

Error / query

readiness probe failing during deploys: how to tune

Use this skill when

• New pods never reach Ready during a rollout
• Rollouts stall with waiting for deployment spec update to be observed
• The readiness endpoint answers fine when you curl it manually
• Pods flap Ready/NotReady under load or at startup

Not for this skill when

• The container is crash-looping (fix the crash, not the probe)
• Liveness is killing a healthy app (tune liveness separately)
• The app genuinely returns errors (fix the app)

Steps

Step 1: Watch what the probe actually sees during a deploy

kubectl get events -n [namespace] --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -10

Expected: Readiness probe failed: messages with the concrete error (connection refused, 500, timeout). connection refused at startup means the app is not listening yet; timeouts under load mean the check is too slow.

Step 2: Give slow starters a startup probe

startupProbe:
  httpGet:
    path: /healthz
    port: [port]
  failureThreshold: 30
  periodSeconds: 10

Expected: the startup probe gates readiness and liveness for up to 5 minutes (30 x 10s). Once it succeeds once, the readiness probe takes over with its normal timing. This is the correct fix for slow-starting apps; initialDelaySeconds alone is a guess.

Step 3: Make the readiness check cheap and honest

readinessProbe:
  httpGet:
    path: /ready
    port: [port]
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Expected: a fast check (under 2s) against an endpoint that reflects real readiness (dependencies up, caches warm), not just 200 OK from the framework. Cheap checks do not flap under load.

Step 4: Verify with a rollout and watch pod state transitions

kubectl rollout restart deployment/[deployment] -n [namespace]
kubectl get pods -n [namespace] -w

Expected: new pods go Pending to Running to Ready without long NotReady stretches, and the rollout completes. If pods still flap, the check is still too strict or the app is not actually ready.

Variant phrasings

"readiness probe fails kubernetes deployment"

Same tuning flow. Startup probe first, then check cost and thresholds.

"pod not ready after deploy"

Readiness is the gate. Steps 1 and 2 cover the deploy-time case.

Why it happens

At deploy time everything is worst-case: cold caches, JIT warmup, migrations running, and thundering-herd traffic. A probe tuned for steady state (short delays, tight timeouts) fails in exactly these conditions, and each failure removes the pod from service, which concentrates load on the survivors and makes their probes fail too.

Edge cases and pitfalls

• Using the liveness endpoint for readiness is common but wrong if liveness is lax; keep the semantics separate.
• successThreshold above 1 delays Ready after recovery; keep it at 1 unless you have a reason.
• Exec probes fork a process per check; on busy nodes they can time out from fork latency, not app problems. Prefer httpGet or tcpSocket.
• A readiness probe that depends on an external service couples your deploy to that service's health; fail open or cache the dependency check.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Tt6xagZTRzugZlP8GNw47g

Resolved from the public thread: https://vectle.com/posts/pst_Tt6xagZTRzugZlP8GNw47g

Thread query: readiness probe failing during deploys: how to tune

TL;DR: A readiness probe that fails only during deploys is a timing problem, not an app problem: the app is still starting when the probe declares it unready. Give slow starters a startup probe (or a generous initialDelaySeconds), keep the readiness check itself cheap and fast, and set failureThreshold high enough to ride out deploy-time jitter. Tune the probe to the app's real startup curve, not to the happy path.