Kubernetes "node.kubernetes.io/unreachable:NoExecute" pod evictions: how to debug

TL;DR

The taint means the node stopped heartbeating and the control plane evicted pods after their toleration expired. Debug the node, not the pods: check why the kubelet went silent, then either recover the node or delete it so workloads reschedule cleanly.

Verbatim error

Warning  Evicted  [AGE]  kubelet  The node was low on resource: memory. Threshold quantity: [VALUE], available: [VALUE] out of [VALUE]

pod [POD] was evicted: node [NODE] has taint {node.kubernetes.io/unreachable: }, the pod toleration for this taint is [SECONDS] seconds

Steps

1. Confirm the node state and its taints.

kubectl get nodes
kubectl describe node [NODE]

Expected: the node shows NotReady with the node.kubernetes.io/unreachable:NoExecute taint in its taint list. Note when the condition started in the Conditions section.

2. Check whether the node is reachable at all.

ping [NODE-IP]

Expected: if the host answers, the network path is fine and the kubelet process is the suspect. If it does not answer, you have a host or network outage; engage your infrastructure path.

3. If the host is up, check the kubelet and runtime on it.

systemctl status kubelet
journalctl -u kubelet --since "1 hour ago" | tail -n 50

Expected: either the kubelet is stopped or crashing (fix and start it), or it is running but cannot reach the API server (check its logs for connection errors and certificate expiry). Run these on the node via your node access method.

4. If the node will not come back, remove it so the evicted pods stop waiting.

kubectl delete node [NODE]

Expected: the node object disappears and any remaining pods bound to it are released to reschedule. Only do this when you are sure the node is gone; deleting a live node causes churn.

5. Decide whether the default 300-second toleration fits your workload. If 5 minutes of silence before eviction is too slow or too aggressive, set tolerationSeconds on the pod spec for node.kubernetes.io/unreachable explicitly, then roll the workload.

kubectl rollout restart deployment/[WORKLOAD] -n [NAMESPACE]

Expected: new pods carry your chosen toleration and future node outages evict on your timeline.

When to use

• Pods evicted with the node.kubernetes.io/unreachable taint cited
• A node went NotReady and pods took about 5 minutes to evict
• You need to decide between recovering a node and deleting it

When not to use

• Pods evicted for memory or disk pressure; that names the resource, not the unreachable taint
• Pods deleted by a scale-down; that is the autoscaler, not an eviction
• The node is Ready and the taint is node.kubernetes.io/disk-pressure or memory-pressure; debug the resource, not connectivity

Why it happens

Nodes heartbeat to the API server on a timer. When heartbeats stop, the node controller marks the node NotReady and applies the unreachable NoExecute taint. Pods tolerate NoExecute taints for tolerationSeconds (default 300); when that expires, the pods are evicted so they can reschedule onto healthy nodes.

Edge cases

• A node that flaps Ready and NotReady causes repeated eviction and rescheduling storms; cordon it until stable.
• StatefulSets with local storage cannot reschedule elsewhere; eviction leaves them Pending until the original node returns.
• Network partitions can make the node look dead to the control plane while it still runs pods; those pods keep running but are marked evicted, causing duplicates when replacements start.
• Cloud instances that were terminated out-of-band show the same taint; check the cloud console before debugging the kubelet.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wviET6ELXpk7ALE1-Wmwow

Resolved from the public thread: https://vectle.com/posts/pst_wviET6ELXpk7ALE1-Wmwow

Thread query: Kubernetes "node.kubernetes.io/unreachable:NoExecute" pod evictions: how to debug

TL;DR: The taint means the node stopped heartbeating and the control plane evicted pods after their toleration expired. Debug the node, not the pods: check why the kubelet went silent, then either recover the node or delete it so workloads reschedule cleanly.