No route to host" between Kubernetes pods: how to debug

TL;DR

"No route to host" between pods means the node's routing table lacks a path to the destination pod's subnet: the CNI did not program routes, the destination node is unreachable, or a network policy is dropping packets (which can surface as no-route on some CNIs). Check whether the destination pod's node is reachable at all first; pod networking problems are usually node networking problems.

The query

"No route to host" between Kubernetes pods: how to debug

Use this when

• Pod-to-pod connections fail with no route to host
• Only some pod pairs are affected
• After node or CNI changes
• New nodes exhibit the problem

Not for when

• DNS resolution failures
• Service ClusterIP issues
• Connection refused (route exists, nothing listening)

Steps

Step 1: Check node-to-node connectivity

Verify the source pod's node can reach the destination pod's node directly. If nodes cannot reach each other, no pod routing will work; fix the node network first. Expected output: node-level connectivity confirmed or identified as broken.

Step 2: Inspect the route tables

Check the source node's routes for the destination pod CIDR. Missing routes mean the CNI did not program them (CNI agent issue); wrong routes mean stale state. Compare with a healthy node's table. Expected output: the missing or wrong route identified.

Step 3: Verify the CNI agent on both nodes

Confirm the CNI pods are healthy on both the source and destination nodes. A dead CNI agent on either side breaks route programming or pod setup. Expected output: CNI healthy on both nodes, or the broken side found.

Step 4: Rule out network policies

Check for policies selecting either pod that could drop the traffic. Some CNIs surface policy drops as no-route errors rather than timeouts; verify by testing with policies temporarily removed in a non-production namespace. Expected output: policies ruled out or identified as the dropper.

Step 5: Test with fresh pods on both nodes

Schedule test pods on the involved nodes and test connectivity between them. If fresh pods work, the original pods have stale network state; if they fail too, the node or CNI configuration is at fault. Expected output: the problem isolated to pod state vs node/CNI config.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4E5IzuBqxCLrNSHzsEiqw