Kubernetes "FailedSync": "error determining status" kubelet debugging

TL;DR

A single FailedSync is usually transient, but a repeating one means the kubelet cannot get container status from the runtime. Check the node's kubelet logs and the container runtime health; the pod is a symptom, the node is the patient.

Verbatim error

Warning  FailedSync  [AGE]  kubelet  Error determining status for container "[CONTAINER]": error determining status: rpc error: code = Unknown desc = Error response from daemon

Steps

1. See whether the warning repeats and whether it is node-wide.

kubectl describe pod [POD] -n [NAMESPACE]
kubectl get events -n [NAMESPACE] --field-selector reason=FailedSync

Expected: one or two occurrences is noise; a steady stream, or the same warning on many pods of one node, means the kubelet or runtime on that node is unhealthy.

2. Check the node's conditions.

kubectl describe node [NODE]

Expected: look at Conditions (MemoryPressure, DiskPressure, Ready) and the kubelet version. DiskPressure or a NotReady node explains sync failures.

3. Read the kubelet log on the node for the matching timestamp.

journalctl -u kubelet --since "30 minutes ago" | grep -i "error determining status"

Expected: the log shows the underlying runtime call failing, for example a containerd socket timeout or a shim that stopped responding. Run this on the node itself via your node access method.

4. Check the container runtime directly.

systemctl status containerd
crictl ps -a

Expected: containerd active and crictl listing containers promptly. A hung or restarted runtime matches the FailedSync window; restart containerd only after cordoning the node if workloads are affected.

5. Check node disk space, since a full disk wedges both the kubelet and the runtime.

df -h /var/lib/kubelet /var/lib/containerd

Expected: plenty of free space. If a filesystem is over about 85 percent, free space (old images, logs) and the sync errors usually stop.

When to use

• FailedSync error determining status repeats on one node or one pod
• Pod status flaps or the kubelet reports stale container state
• The warning started after a node reboot, runtime upgrade, or disk filling up

When not to use

• A single FailedSync event with no repeat; that is normal transient noise
• The pod is CrashLoopBackOff for app reasons; FailedSync there is a side effect, debug the app
• The whole node is NotReady; start with node NotReady debugging instead

Why it happens

The kubelet periodically asks the container runtime for each container's status over a local socket. If the runtime is slow, hung, or restarting, or the kubelet itself is starved (disk pressure, CPU), the status call fails and the kubelet records FailedSync. The pod spec is fine; the status pipeline on the node is not.

Edge cases

• A wedged container shim for one container can poison status calls; restarting just that container sometimes clears it without touching the runtime.
• After a containerd upgrade, a socket path or version mismatch makes every status call fail until the kubelet is restarted to match.
• Disk pressure evictions and FailedSync often arrive together; fix the disk first, then re-check.
• On nodes with very high pod counts, kubelet sync latency grows; FailedSync can be the first sign you need fewer pods per node or a faster disk.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst34jk5vD0cUeBOZfhdPFrw