Cilium "bpf: map creation failed" / endpoint not ready: how to debug

TL;DR

The cilium-agent cannot allocate the BPF maps it needs, usually because the kernel is too old, the BPF filesystem is not mounted, or map memory is exhausted. Check the agent logs, then the kernel and bpffs on the node; the pod networking symptoms are downstream of the agent.

Verbatim error

bpf: map creation failed: [MAP-NAME]: no space left on device

Steps

1. Read the agent logs on the affected node for the exact map failure.

kubectl logs -n kube-system -l k8s-app=cilium --tail=200

Expected: lines naming the map that failed creation and the errno, for example no space left on device (map memory exhausted) or operation not permitted (kernel or mount problem). The errno picks the next step.

2. Check the kernel version on the node against Cilium's requirements.

kubectl get node [NODE] -o yaml

Expected: the kubelet-reported kernel version. Cilium needs a reasonably modern kernel for its BPF features; an old or custom kernel that lacks BPF support fails map creation at agent start.

3. Verify the BPF filesystem is mounted on the node.

mount | grep bpf

Expected: a bpffs mount, typically at /sys/fs/bpf. If it is missing, the agent has nowhere to pin maps; mount it persistently via your node provisioning. Run this on the node via your node access method.

4. Check for map pressure: too many endpoints or stale maps exhausting the map memory.

kubectl -n kube-system exec ds/cilium -- cilium bpf lb list

Expected: the command runs. If maps are full, look for leaked endpoints from crashed pods and restart the agent to reclaim; persistent pressure means raising the bpf map size limits in the Cilium config.

5. After fixing the node, restart the agent and watch endpoints become ready.

kubectl -n kube-system rollout restart ds/cilium
kubectl -n kube-system exec ds/cilium -- cilium status

Expected: cilium status shows the agent healthy and endpoints transitioning to ready. Pods on the node regain networking as their endpoints program.

When to use

• Cilium endpoints stuck not ready with bpf map creation failed in agent logs
• Pod networking breaks on specific nodes while others are fine
• The failure started after a kernel upgrade, node image change, or Cilium upgrade

When not to use

• All nodes lose networking at once; that is usually a control-plane or config rollout issue
• NetworkPolicy drops traffic between ready endpoints; the datapath works, policy does not
• DNS fails but pod-to-pod IP traffic works; that is a DNS problem, not BPF maps

Why it happens

Cilium programs networking as BPF maps pinned on bpffs: connection tracking, load balancing, policy, endpoints. Map creation needs kernel BPF support, a mounted bpffs, and free entries within the map size limits. Any of those missing makes the agent unable to program the datapath, so endpoints never become ready.

Edge cases

• Nodes with huge pod counts exhaust default map sizes; raise the per-map limits in the Helm values before the node fills.
• Custom kernels with BPF compiled out fail every map creation; the fix is a kernel rebuild or a different node image.
• Stale maps from a crashed agent can block a new agent from pinning; a node reboot clears them when restart does not.
• eBPF JIT disabled on some hardened kernels degrades or breaks datapath features; check the JIT setting if maps create but traffic misbehaves.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_c7G2J3wbql0HMUpPUpnrxw