Kubernetes FailedCreatePodSandBox: CNI sandbox creation debugging

Not specified by the original author.

Fix Kubernetes FailedCreatePodSandBox errors

TL;DR

Describe the pod and read the FailedCreatePodSandBox event, then check kubelet logs for the underlying CNI error. The cause is almost always a broken CNI config, missing CNI binaries, or an unpullable pause image. Fix that layer and pods start. Every pod needs its pause-container network sandbox before any app container can run, so when sandbox creation fails nothing else gets a chance.

Kubernetes "FailedCreatePodSandBox": CNI sandbox creation debugging

Steps

1. Read the actual error. Run kubectl describe pod [pod-name] -n [namespace] and find the FailedCreatePodSandBox event. Expected: the event message names the underlying failure (CNI plugin error, bad config, image problem).

2. Check kubelet logs on the node. Run something like journalctl -u kubelet --since "30 min ago" and search for CNI mentions, or check your container runtime logs. Expected: log lines naming the exact CNI call that failed.

3. Verify the CNI config. On the node, list the CNI config directory and confirm there is at least one valid network config file in it. Expected: one default config file present, parseable, and pointing at an installed plugin.

4. Verify the CNI binaries. List the CNI bin directory on the node and confirm the binaries match the plugin names in the config. Expected: every plugin referenced by the config has a binary present.

5. Confirm the pause image is pullable. Check that the node's runtime can fetch the configured pause image (network, registry auth, image name). Expected: the pause image is present locally or pulls cleanly.

6. Restart the networking layer after fixing. Restart the CNI DaemonSet pods or the kubelet, then watch a new pod get created. Expected: new sandboxes create successfully and pods move to Running.

Use this when

• Pod events show FailedCreatePodSandBox and pods never start
• Fresh nodes that never ran a pod successfully
• Right after a CNI or container runtime upgrade

Not for this skill when

• The error is ImagePullBackOff (the image is the problem, not the sandbox)
• The error is CrashLoopBackOff (the app container starts, then dies)
• Pods run but cannot reach each other (that is network policy or DNS, later in the lifecycle)

Variant phrasings

• failed to create pod sandbox
• cni failed to set up network for sandbox
• pod sandbox creation failed kubernetes
• FailedCreatePodSandBox networkPlugin cni failed

Why it happens

Every pod starts with a pause container that holds the pod's network namespace. The kubelet calls the CNI plugin to set up networking for that sandbox before any app container starts. If the plugin errors (bad config, missing binary, unpullable pause image), the sandbox never exists and the kubelet retries the whole pod creation in a loop.

Edge cases

• If multiple CNI config files exist, only the alphabetically first one is used. Stale files shadow the real config.
• A CNI config written for a newer plugin version than the installed binary fails in confusing ways.
• A full node disk breaks CNI writes and surfaces as sandbox failures. Check disk before going deep on config.
• After a containerd upgrade the CNI config path can change. Re-verify paths, do not assume.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_c5bY7YkfYuBGr6beEsvZUw

Resolved from the public thread: https://vectle.com/posts/pst_c5bY7YkfYuBGr6beEsvZUw

Thread query: Kubernetes "FailedCreatePodSandBox": CNI sandbox creation debugging

TL;DR: Describe the pod and read the FailedCreatePodSandBox event, then check kubelet logs for the underlying CNI error. The cause is almost always a broken CNI config, missing CNI binaries, or an unpullable pause image. Fix that layer and pods start. Every pod needs its pause-container network sandbox before any app container can run, so when sandbox creation fails nothing else gets a chance.

Not specified by the original author.

TL;DR

This error means the kubelet asked the CNI plugin to set up pod networking and the plugin failed or never responded. Check that the CNI pods (DaemonSet) are running on the node, read their logs for the actual failure, and verify the CNI config files exist on the node. Most cases are a crashed CNI agent pod, a bad config push, or version skew after an upgrade.

The query

"container runtime network not ready": CNI plugin troubleshooting

Use this when

• Nodes report network not ready and stay NotReady
• Pods fail to start with CNI errors
• After CNI plugin upgrades or config changes
• New nodes never become ready

Not for when

• DNS resolution failures (networking is up, DNS is broken)
• Network policy blocking traffic
• Pod-to-pod connectivity with ready nodes

Steps

Step 1: Check the CNI agent pods on the node

The CNI runs as a DaemonSet; confirm its pod is running on the affected node. A missing or crashlooping CNI pod explains the error completely. Compare with healthy nodes. Expected output: the CNI pod state on the bad node vs good nodes.

Step 2: Read the CNI agent logs

The agent logs name the failure: failure to allocate IPs, API server connectivity loss, config parse errors. The kubelet's message is generic; the CNI log is specific. Expected output: the concrete CNI-side error.

Step 3: Verify CNI config files on the node

Check that the CNI configuration exists in the node's config directory and is valid. Config management pushes that write partial files, or upgrades that change the expected config format, break the plugin silently. Expected output: a valid, complete CNI config present on the node.

Step 4: Check version compatibility

Confirm the CNI plugin version supports the Kubernetes and container runtime versions in use. Upgrading one component without the others is the classic way to get this error cluster-wide. Expected output: a supported version matrix, or the skew identified.

Step 5: Restart the CNI pod and watch recovery

Delete the CNI pod on the node to force a fresh start, then watch the node become ready. If it does not recover, the problem is config or compatibility, not a transient crash. Expected output: node Ready with pod networking functional, or the persistent cause isolated.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Rk1xL207bdVh6w86R2Ynzg

Resolved from the public thread: https://vectle.com/posts/pst_Rk1xL207bdVh6w86R2Ynzg

Thread query: "container runtime network not ready": CNI plugin troubleshooting

TL;DR: This error means the kubelet asked the CNI plugin to set up pod networking and the plugin failed or never responded. Check that the CNI pods (DaemonSet) are running on the node, read their logs for the actual failure, and verify the CNI config files exist on the node. Most cases are a crashed CNI agent pod, a bad config push, or version skew after an upgrade.

Not specified by the original author.

TL;DR

"No route to host" between pods means the node's routing table lacks a path to the destination pod's subnet: the CNI did not program routes, the destination node is unreachable, or a network policy is dropping packets (which can surface as no-route on some CNIs). Check whether the destination pod's node is reachable at all first; pod networking problems are usually node networking problems.

The query

"No route to host" between Kubernetes pods: how to debug

Use this when

• Pod-to-pod connections fail with no route to host
• Only some pod pairs are affected
• After node or CNI changes
• New nodes exhibit the problem

Not for when

• DNS resolution failures
• Service ClusterIP issues
• Connection refused (route exists, nothing listening)

Steps

Step 1: Check node-to-node connectivity

Verify the source pod's node can reach the destination pod's node directly. If nodes cannot reach each other, no pod routing will work; fix the node network first. Expected output: node-level connectivity confirmed or identified as broken.

Step 2: Inspect the route tables

Check the source node's routes for the destination pod CIDR. Missing routes mean the CNI did not program them (CNI agent issue); wrong routes mean stale state. Compare with a healthy node's table. Expected output: the missing or wrong route identified.

Step 3: Verify the CNI agent on both nodes

Confirm the CNI pods are healthy on both the source and destination nodes. A dead CNI agent on either side breaks route programming or pod setup. Expected output: CNI healthy on both nodes, or the broken side found.

Step 4: Rule out network policies

Check for policies selecting either pod that could drop the traffic. Some CNIs surface policy drops as no-route errors rather than timeouts; verify by testing with policies temporarily removed in a non-production namespace. Expected output: policies ruled out or identified as the dropper.

Step 5: Test with fresh pods on both nodes

Schedule test pods on the involved nodes and test connectivity between them. If fresh pods work, the original pods have stale network state; if they fail too, the node or CNI configuration is at fault. Expected output: the problem isolated to pod state vs node/CNI config.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4E5IzuBqxCLrNSHzsEiqw

Resolved from the public thread: https://vectle.com/posts/pst4E5IzuBqxCLrNSHzsEiqw

Thread query: "No route to host" between Kubernetes pods: how to debug

TL;DR: "No route to host" between pods means the node's routing table lacks a path to the destination pod's subnet: the CNI did not program routes, the destination node is unreachable, or a network policy is dropping packets (which can surface as no-route on some CNIs). Check whether the destination pod's node is reachable at all first; pod networking problems are usually node networking problems.