Kubernetes "FailedMount": "unable to attach or mount volumes" debugging

TL;DR

Describe the pod; the FailedMount event names the volume and the reason. Most cases are an unbound PVC, a broken StorageClass provisioner, or a missing CSI driver secret. Fix the volume layer and the pod follows.

Verbatim error

Warning  FailedMount  [AGE]  kubelet  Unable to attach or mount volumes: unmounted volumes=[[VOLUME-NAME]], unattached volumes=[[VOLUME-NAME]]: timed out waiting for the condition

Steps

1. Describe the pod and read the full FailedMount message.

kubectl describe pod [POD] -n [NAMESPACE]

Expected: the event names the volume and gives a reason, for example the PVC is not bound, the attach timed out, or a secret was not found. The reason picks the next step.

2. Check the PVC state.

kubectl get pvc -n [NAMESPACE]
kubectl describe pvc [PVC-NAME] -n [NAMESPACE]

Expected: Bound means the problem is at attach or mount time; Pending means provisioning or binding failed, and the PVC events say why.

3. Check the StorageClass and its provisioner.

kubectl get storageclass
kubectl get storageclass [SC-NAME] -o yaml

Expected: the provisioner field names the CSI driver responsible. If the class is missing or the provisioner string is wrong, fix the PVC to reference a valid class.

4. Verify the CSI driver pods are healthy, including the node plugin on the target node.

kubectl get pods -n [CSI-NAMESPACE] -o wide
kubectl get pods -n [CSI-NAMESPACE] -l app=[CSI-NODE-PLUGIN-LABEL] -o wide

Expected: controller and node-plugin pods are Running on the node where your pod is scheduled. CrashLooping or missing node plugins explain attach timeouts.

5. For secret-based drivers, confirm the referenced secret exists in the expected namespace.

kubectl get secret [SECRET-NAME] -n [NAMESPACE]

Expected: the secret exists with the keys the driver documents. A missing or misspelled secret is a common cause of mount failures.

When to use

• Pods stuck Pending or ContainerCreating with FailedMount warnings
• Events say unable to attach or mount volumes or timed out waiting for the condition
• The failure followed a StorageClass change, CSI upgrade, or new node pool

When not to use

• The pod fails with CreateContainerConfigError (config reference issue, not storage)
• The error is FailedAttachVolume naming a cloud disk that no longer exists; that is a detached-volume cleanup case
• The workload uses hostPath or emptyDir; those never go through attach/mount provisioning

Why it happens

Mounting a volume is a chain: PVC binds to a PV, the external provisioner or attacher prepares the disk, the CSI node plugin mounts it on the node, then the kubelet bind-mounts it into the container. FailedMount is the kubelet reporting that some link in that chain timed out or errored, so debugging means walking the chain from the pod downward.

Edge cases

• The PVC is Bound but to a PV in a different zone than the node; the attach can never succeed. Check node zone labels against the PV nodeAffinity.
• Stale VolumeAttachment objects from a crashed node block re-attach; delete the orphaned attachment after confirming the old node is gone.
• Driver upgrades sometimes rename the provisioner string; old PVCs referencing the old name stay Pending forever.
• Multi-attach errors (volume already attached to another node) need the old attachment released before the new node can mount.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuhUwSu-hzsSlJOjyN5WvQ