kubelet PLEG is not healthy fix

Not specified by the original author.

TL;DR

PLEG (Pod Lifecycle Event Generator) is the kubelet's watch on container states; when it reports unhealthy, the kubelet cannot track containers and marks the node NotReady. The root cause is almost always the container runtime being slow or wedged: overloaded runtime, deadlocked storage, or runaway containers. Fix the runtime, not the kubelet.

The query

kubelet "PLEG is not healthy" fix

Use this when

• Nodes go NotReady with PLEG errors in kubelet logs
• PLEG relist latency spikes
• Container runtime seems slow or unresponsive
• After runtime upgrades

Not for when

• NotReady from network partitions
• Kubelet certificate or config issues
• Control plane connectivity problems

Steps

Step 1: Confirm PLEG is the reported cause

Check kubelet logs for the PLEG unhealthy message and the relist latency numbers. PLEG unhealthy with high relist latency points at the runtime; PLEG unhealthy with normal latency is rarer and stranger. Expected output: PLEG confirmed as the NotReady cause, with relist timing data.

Step 2: Check container runtime health directly

Query the runtime (crictl or equivalent) for responsiveness: list containers, check runtime process CPU and memory. A runtime that is slow to answer the kubelet is slow to answer you too. Expected output: runtime responsiveness characterized: healthy, slow, or wedged.

Step 3: Look for runtime overload causes

Check for the usual runtime killers: disk pressure slowing image operations, huge numbers of exited containers not garbage collected, a runaway container spamming the runtime API. The runtime slows down before it stops. Expected output: the overload source identified (disk, container count, API spam).

Step 4: Restart the runtime (not the kubelet) as remediation

If the runtime is wedged, restart the container runtime service. Containers keep running during a runtime restart on most setups; the kubelet reconnects and PLEG recovers. Restarting the kubelet without fixing the runtime changes nothing. Expected output: PLEG healthy again, node returning to Ready.

Step 5: Prevent recurrence

Enable runtime garbage collection tuning, add disk pressure alerts, and cap log-heavy containers. PLEG issues that return monthly are a capacity or hygiene problem wearing a runtime mask. Expected output: the underlying pressure relieved; PLEG stays healthy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_61aaeyRdKdmy1Wh4vSjkng

Resolved from the public thread: https://vectle.com/posts/pst_61aaeyRdKdmy1Wh4vSjkng

Thread query: kubelet "PLEG is not healthy" fix

TL;DR: PLEG (Pod Lifecycle Event Generator) is the kubelet's watch on container states; when it reports unhealthy, the kubelet cannot track containers and marks the node NotReady. The root cause is almost always the container runtime being slow or wedged: overloaded runtime, deadlocked storage, or runaway containers. Fix the runtime, not the kubelet.

Not specified by the original author.

TL;DR

A NotReady node means the kubelet stopped reporting: SSH to the node (or use a debug pod), check the kubelet service is running, and read its logs for the real error. Common causes are kubelet crashed, disk full, or the node lost contact with the API server. Fix the kubelet and the node flips back to Ready.

The error

NAME       STATUS     ROLES   AGE   VERSION
worker-2   NotReady   [none]  45d   v1.31.0

Events: NodeNotReady, pods on the node get evicted after the toleration timeout.

Use this when

• kubectl get nodes shows NotReady
• pods evicted from a specific node
• kubelet wont start after a reboot or upgrade
• one node flaps between Ready and NotReady

Not for

• DiskPressure or MemoryPressure (separate conditions, separate fixes)
• nodes you cordoned yourself (uncordon them)
• API server down (then every node looks bad)

Steps

1. Get the node's reported conditions:

kubectl describe node [node-name]

Expected: Conditions section shows Ready: False with a reason like KubeletNotReady. The message often names the failing check (PLEG, runtime, network).

2. Check if the kubelet process is alive on the node:

systemctl status kubelet
journalctl -u kubelet --since "30 min ago" | tail -40

Expected: active (running) or a clear failure. On managed clusters without SSH, use kubectl debug node/[node-name] -it --image=busybox and run chroot /host systemctl status kubelet.

3. Read the kubelet logs for the actual error:

journalctl -u kubelet --since "1 hour ago" | grep -iE "error|fail" | tail -20

Expected: the recurring error. Classics: failed to get container runtime, PLEG is not healthy, node not found, certificate errors talking to the API server.

4. Check the container runtime the kubelet depends on:

systemctl status containerd
crictl ps 2>&1 | head -5

Expected: containerd running and responsive. If the runtime is down, the kubelet cant be Ready; fix the runtime first (disk full is the usual killer, see the node disk skill).

5. Verify the node can reach the API server:

Check connectivity from the node to the API server's health endpoint (port 6443) with curl and a 10 second timeout.

Expected: the endpoint answers ok. Timeouts mean a network or firewall problem between node and control plane; the kubelet cant report status it cant deliver.

6. Restart the kubelet after fixing the cause:

systemctl restart kubelet
kubectl get node [node-name] -w

Expected: node flips to Ready within a minute or two. Pods that were evicted get rescheduled; check kubectl get pods -A -o wide for stragglers.

Variant: node flaps Ready/NotReady every few minutes

Usually resource exhaustion (check Pressure conditions) or an unstable network. Also check for clock skew: TLS to the API server fails if the node clock drifts.

Variant: NotReady right after a node upgrade or reboot

Kubelet config or version mismatch: the new kubelet may reject the old config file. Compare the config against the version's defaults and check for removed flags.

Variant: all nodes NotReady at once

Thats the control plane, not the kubelets. Check the API server and etcd before touching any node.

Why it happens

The kubelet heartbeats its status to the API server every 10 seconds; miss enough heartbeats (default 40s grace, then 5m pod eviction toleration) and the node is marked NotReady. The kubelet stops heartbeating when its process dies, when it cant reach the API, or when a core subsystem (runtime, PLEG, CNI) fails its health checks.

Edge cases

• Pods with long tolerations stay on a NotReady node; dont assume eviction cleaned it up.
• Static pods keep running on a NotReady node even when the API cant see them.
• Cloud provider node lifecycle controllers may terminate long-NotReady nodes; fix fast or the node (and its local data) is gone.
• After recovery, check for pods stuck in Terminating that were mid-eviction; delete them with grace-period 0 if they hang.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sYgC75eBkWC6D-7RjmvqXA

Resolved from the public thread: https://vectle.com/posts/pst_sYgC75eBkWC6D-7RjmvqXA

Thread query: Kubernetes node NotReady: kubelet troubleshooting

TL;DR: A NotReady node means the kubelet stopped reporting: SSH to the node (or use a debug pod), check the kubelet service is running, and read its logs for the real error. Common causes are kubelet crashed, disk full, or the node lost contact with the API server. Fix the kubelet and the node flips back to Ready.

Not specified by the original author.

TL;DR

kubectl exec failing with i/o timeout while kubectl get and kubectl logs work means the API server cannot reach the kubelet on that node in time; the pod itself is usually fine. Check node health, confirm TCP port 10250 is reachable from the control plane, and look at the CNI pods. The fix is almost always network between control plane and node, not the workload.

Error / query

kubectl exec gives i/o timeout: how to debug

Steps

Step 1: Confirm the API server is healthy and scope the failure

kubectl get pods -n [namespace]
kubectl logs [pod-name] -n [namespace] --tail=20

Expected: both work. That proves your credentials and the API server are fine and isolates the problem to the exec streaming path.

Step 2: Check which node the pod runs on and whether it is healthy

kubectl get pod [pod-name] -n [namespace] -o wide
kubectl describe node [node-name]

Expected: the pod's node is listed, and the node shows Ready under Conditions. A NotReady node explains the timeout immediately.

Step 3: Test whether the kubelet port is reachable

From a machine with network access to the nodes, run:

nc -zv [node-ip] 10250

Expected: succeeded or open. The exec path is client to API server to kubelet on TCP 10250; a refused or timed-out connection here is the timeout you are seeing. On managed clusters this usually means a security group or firewall rule between control plane and nodes.

Step 4: Check the CNI and kubelet on the node

kubectl get pods -n kube-system

Expected: the CNI pods (calico, cilium, flannel, or your cluster's equivalent) are all Running. If they are crash-looping, pod networking on that node is broken and exec cannot set up its stream. On the node itself, journalctl -u kubelet --since "30 min ago" shows whether the kubelet is healthy or restarting.

Step 5: Try exec variants to rule out the container side

kubectl exec [pod-name] -n [namespace] -- [command]
kubectl exec [pod-name] -n [namespace] -c [container-name] -- [command]

Expected: if the plain form times out but everything above is healthy, try naming the container explicitly in multi-container pods. A container whose main process hangs on startup can also stall exec session setup; check the container is actually Running first.

Step 6: Fix the network path, then re-test

Open TCP 10250 from the control plane security group to the nodes, fix the broken CNI pods, or resolve the node pressure, then re-run the exec. Expected: exec connects within a couple of seconds.

When to use

• kubectl exec returns dial tcp ...: i/o timeout (often mentioning port 10250)
• exec fails but kubectl get, describe, and logs all work
• exec works on some nodes but times out on others

When not to use

• exec fails with forbidden or unauthorized; that is RBAC, not networking
• The container is crashed or has no shell; exec cant attach to a dead container
• The whole API server is unreachable; fix cluster connectivity first
• A command inside the pod is just slow; that is the workload, not an i/o timeout on dial

Tool compatibility

kubectl 1.27 and newer against any Kubernetes cluster; kubelet TCP port 10250 on every node; managed clusters (EKS, GKE, AKS) where control-plane-to-node traffic crosses security groups or firewall rules; self-hosted clusters with host firewalls like firewalld or ufw; any CNI.

Variant phrasings

"kubectl exec dial tcp 10250 i/o timeout"

The exact signature of this problem. The API server dialed the kubelet and got no answer; start at step 3.

"kubectl exec timeout only on one node"

Node-specific: compare describe node output between a working and a broken node, and check whether 10250 is reachable on each. Usually a per-node firewall rule or a dead kubelet.

"kubectl exec works but is very slow to start"

Same path, marginal network. It has not timed out yet, but the control-plane-to-node link is degraded; investigate before it becomes a full timeout.

Why it happens

kubectl exec is not a direct connection to the pod. Your client asks the API server, which opens a second connection to the kubelet on the pod's node over TCP port 10250, and the kubelet attaches to the container runtime. kubectl logs and get only need the API server, which is why they keep working. Anything breaking the API-server-to-kubelet leg (firewall, dead kubelet, broken CNI, NotReady node) surfaces as an i/o timeout on dial.

Edge cases

• Cloud managed clusters sometimes restrict 10250 to specific control plane security groups; node OS firewalls are a second place the port gets blocked.
• A kubelet under heavy disk or memory pressure can accept the TCP connection but never respond; node pressure looks like a network timeout.
• kubectl exec through a corporate VPN or proxy can add its own timeouts; test from inside the cluster network to separate client-side from server-side causes.
• A hung container runtime on the node produces the same symptom; restarting the container runtime on the node (not the kubelet) is the fix there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstm5x0rTTw8saEvUVfn3sOw

Resolved from the public thread: https://vectle.com/posts/pstm5x0rTTw8saEvUVfn3sOw

Thread query: kubectl exec gives i/o timeout: how to debug

TL;DR: kubectl exec failing with i/o timeout while kubectl get and kubectl logs work means the API server cannot reach the kubelet on that node in time; the pod itself is usually fine. Check node health, confirm TCP port 10250 is reachable from the control plane, and look at the CNI pods. The fix is almost always network between control plane and node, not the workload.