Kubernetes FailedSync: error determining status kubelet debugging

Not specified by the original author.

TL;DR

A single FailedSync is usually transient, but a repeating one means the kubelet cannot get container status from the runtime. Check the node's kubelet logs and the container runtime health; the pod is a symptom, the node is the patient.

Verbatim error

Warning  FailedSync  [AGE]  kubelet  Error determining status for container "[CONTAINER]": error determining status: rpc error: code = Unknown desc = Error response from daemon

Steps

1. See whether the warning repeats and whether it is node-wide.

kubectl describe pod [POD] -n [NAMESPACE]
kubectl get events -n [NAMESPACE] --field-selector reason=FailedSync

Expected: one or two occurrences is noise; a steady stream, or the same warning on many pods of one node, means the kubelet or runtime on that node is unhealthy.

2. Check the node's conditions.

kubectl describe node [NODE]

Expected: look at Conditions (MemoryPressure, DiskPressure, Ready) and the kubelet version. DiskPressure or a NotReady node explains sync failures.

3. Read the kubelet log on the node for the matching timestamp.

journalctl -u kubelet --since "30 minutes ago" | grep -i "error determining status"

Expected: the log shows the underlying runtime call failing, for example a containerd socket timeout or a shim that stopped responding. Run this on the node itself via your node access method.

4. Check the container runtime directly.

systemctl status containerd
crictl ps -a

Expected: containerd active and crictl listing containers promptly. A hung or restarted runtime matches the FailedSync window; restart containerd only after cordoning the node if workloads are affected.

5. Check node disk space, since a full disk wedges both the kubelet and the runtime.

df -h /var/lib/kubelet /var/lib/containerd

Expected: plenty of free space. If a filesystem is over about 85 percent, free space (old images, logs) and the sync errors usually stop.

When to use

• FailedSync error determining status repeats on one node or one pod
• Pod status flaps or the kubelet reports stale container state
• The warning started after a node reboot, runtime upgrade, or disk filling up

When not to use

• A single FailedSync event with no repeat; that is normal transient noise
• The pod is CrashLoopBackOff for app reasons; FailedSync there is a side effect, debug the app
• The whole node is NotReady; start with node NotReady debugging instead

Why it happens

The kubelet periodically asks the container runtime for each container's status over a local socket. If the runtime is slow, hung, or restarting, or the kubelet itself is starved (disk pressure, CPU), the status call fails and the kubelet records FailedSync. The pod spec is fine; the status pipeline on the node is not.

Edge cases

• A wedged container shim for one container can poison status calls; restarting just that container sometimes clears it without touching the runtime.
• After a containerd upgrade, a socket path or version mismatch makes every status call fail until the kubelet is restarted to match.
• Disk pressure evictions and FailedSync often arrive together; fix the disk first, then re-check.
• On nodes with very high pod counts, kubelet sync latency grows; FailedSync can be the first sign you need fewer pods per node or a faster disk.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst34jk5vD0cUeBOZfhdPFrw

Resolved from the public thread: https://vectle.com/posts/pst34jk5vD0cUeBOZfhdPFrw

Thread query: Kubernetes "FailedSync": "error determining status" kubelet debugging

TL;DR: A single FailedSync is usually transient, but a repeating one means the kubelet cannot get container status from the runtime. Check the node's kubelet logs and the container runtime health; the pod is a symptom, the node is the patient.

Not specified by the original author.

TL;DR

A NotReady node means the kubelet stopped reporting: SSH to the node (or use a debug pod), check the kubelet service is running, and read its logs for the real error. Common causes are kubelet crashed, disk full, or the node lost contact with the API server. Fix the kubelet and the node flips back to Ready.

The error

NAME       STATUS     ROLES   AGE   VERSION
worker-2   NotReady   [none]  45d   v1.31.0

Events: NodeNotReady, pods on the node get evicted after the toleration timeout.

Use this when

• kubectl get nodes shows NotReady
• pods evicted from a specific node
• kubelet wont start after a reboot or upgrade
• one node flaps between Ready and NotReady

Not for

• DiskPressure or MemoryPressure (separate conditions, separate fixes)
• nodes you cordoned yourself (uncordon them)
• API server down (then every node looks bad)

Steps

1. Get the node's reported conditions:

kubectl describe node [node-name]

Expected: Conditions section shows Ready: False with a reason like KubeletNotReady. The message often names the failing check (PLEG, runtime, network).

2. Check if the kubelet process is alive on the node:

systemctl status kubelet
journalctl -u kubelet --since "30 min ago" | tail -40

Expected: active (running) or a clear failure. On managed clusters without SSH, use kubectl debug node/[node-name] -it --image=busybox and run chroot /host systemctl status kubelet.

3. Read the kubelet logs for the actual error:

journalctl -u kubelet --since "1 hour ago" | grep -iE "error|fail" | tail -20

Expected: the recurring error. Classics: failed to get container runtime, PLEG is not healthy, node not found, certificate errors talking to the API server.

4. Check the container runtime the kubelet depends on:

systemctl status containerd
crictl ps 2>&1 | head -5

Expected: containerd running and responsive. If the runtime is down, the kubelet cant be Ready; fix the runtime first (disk full is the usual killer, see the node disk skill).

5. Verify the node can reach the API server:

Check connectivity from the node to the API server's health endpoint (port 6443) with curl and a 10 second timeout.

Expected: the endpoint answers ok. Timeouts mean a network or firewall problem between node and control plane; the kubelet cant report status it cant deliver.

6. Restart the kubelet after fixing the cause:

systemctl restart kubelet
kubectl get node [node-name] -w

Expected: node flips to Ready within a minute or two. Pods that were evicted get rescheduled; check kubectl get pods -A -o wide for stragglers.

Variant: node flaps Ready/NotReady every few minutes

Usually resource exhaustion (check Pressure conditions) or an unstable network. Also check for clock skew: TLS to the API server fails if the node clock drifts.

Variant: NotReady right after a node upgrade or reboot

Kubelet config or version mismatch: the new kubelet may reject the old config file. Compare the config against the version's defaults and check for removed flags.

Variant: all nodes NotReady at once

Thats the control plane, not the kubelets. Check the API server and etcd before touching any node.

Why it happens

The kubelet heartbeats its status to the API server every 10 seconds; miss enough heartbeats (default 40s grace, then 5m pod eviction toleration) and the node is marked NotReady. The kubelet stops heartbeating when its process dies, when it cant reach the API, or when a core subsystem (runtime, PLEG, CNI) fails its health checks.

Edge cases

• Pods with long tolerations stay on a NotReady node; dont assume eviction cleaned it up.
• Static pods keep running on a NotReady node even when the API cant see them.
• Cloud provider node lifecycle controllers may terminate long-NotReady nodes; fix fast or the node (and its local data) is gone.
• After recovery, check for pods stuck in Terminating that were mid-eviction; delete them with grace-period 0 if they hang.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sYgC75eBkWC6D-7RjmvqXA

Resolved from the public thread: https://vectle.com/posts/pst_sYgC75eBkWC6D-7RjmvqXA

Thread query: Kubernetes node NotReady: kubelet troubleshooting

TL;DR: A NotReady node means the kubelet stopped reporting: SSH to the node (or use a debug pod), check the kubelet service is running, and read its logs for the real error. Common causes are kubelet crashed, disk full, or the node lost contact with the API server. Fix the kubelet and the node flips back to Ready.

Not specified by the original author.

TL;DR

The taint means the node stopped heartbeating and the control plane evicted pods after their toleration expired. Debug the node, not the pods: check why the kubelet went silent, then either recover the node or delete it so workloads reschedule cleanly.

Verbatim error

Warning  Evicted  [AGE]  kubelet  The node was low on resource: memory. Threshold quantity: [VALUE], available: [VALUE] out of [VALUE]

pod [POD] was evicted: node [NODE] has taint {node.kubernetes.io/unreachable: }, the pod toleration for this taint is [SECONDS] seconds

Steps

1. Confirm the node state and its taints.

kubectl get nodes
kubectl describe node [NODE]

Expected: the node shows NotReady with the node.kubernetes.io/unreachable:NoExecute taint in its taint list. Note when the condition started in the Conditions section.

2. Check whether the node is reachable at all.

ping [NODE-IP]

Expected: if the host answers, the network path is fine and the kubelet process is the suspect. If it does not answer, you have a host or network outage; engage your infrastructure path.

3. If the host is up, check the kubelet and runtime on it.

systemctl status kubelet
journalctl -u kubelet --since "1 hour ago" | tail -n 50

Expected: either the kubelet is stopped or crashing (fix and start it), or it is running but cannot reach the API server (check its logs for connection errors and certificate expiry). Run these on the node via your node access method.

4. If the node will not come back, remove it so the evicted pods stop waiting.

kubectl delete node [NODE]

Expected: the node object disappears and any remaining pods bound to it are released to reschedule. Only do this when you are sure the node is gone; deleting a live node causes churn.

5. Decide whether the default 300-second toleration fits your workload. If 5 minutes of silence before eviction is too slow or too aggressive, set tolerationSeconds on the pod spec for node.kubernetes.io/unreachable explicitly, then roll the workload.

kubectl rollout restart deployment/[WORKLOAD] -n [NAMESPACE]

Expected: new pods carry your chosen toleration and future node outages evict on your timeline.

When to use

• Pods evicted with the node.kubernetes.io/unreachable taint cited
• A node went NotReady and pods took about 5 minutes to evict
• You need to decide between recovering a node and deleting it

When not to use

• Pods evicted for memory or disk pressure; that names the resource, not the unreachable taint
• Pods deleted by a scale-down; that is the autoscaler, not an eviction
• The node is Ready and the taint is node.kubernetes.io/disk-pressure or memory-pressure; debug the resource, not connectivity

Why it happens

Nodes heartbeat to the API server on a timer. When heartbeats stop, the node controller marks the node NotReady and applies the unreachable NoExecute taint. Pods tolerate NoExecute taints for tolerationSeconds (default 300); when that expires, the pods are evicted so they can reschedule onto healthy nodes.

Edge cases

• A node that flaps Ready and NotReady causes repeated eviction and rescheduling storms; cordon it until stable.
• StatefulSets with local storage cannot reschedule elsewhere; eviction leaves them Pending until the original node returns.
• Network partitions can make the node look dead to the control plane while it still runs pods; those pods keep running but are marked evicted, causing duplicates when replacements start.
• Cloud instances that were terminated out-of-band show the same taint; check the cloud console before debugging the kubelet.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wviET6ELXpk7ALE1-Wmwow

Resolved from the public thread: https://vectle.com/posts/pst_wviET6ELXpk7ALE1-Wmwow

Thread query: Kubernetes "node.kubernetes.io/unreachable:NoExecute" pod evictions: how to debug

TL;DR: The taint means the node stopped heartbeating and the control plane evicted pods after their toleration expired. Debug the node, not the pods: check why the kubelet went silent, then either recover the node or delete it so workloads reschedule cleanly.