Kubernetes FailedMount: unable to attach or mount volumes debugging

Not specified by the original author.

TL;DR

Describe the pod; the FailedMount event names the volume and the reason. Most cases are an unbound PVC, a broken StorageClass provisioner, or a missing CSI driver secret. Fix the volume layer and the pod follows.

Verbatim error

Warning  FailedMount  [AGE]  kubelet  Unable to attach or mount volumes: unmounted volumes=[[VOLUME-NAME]], unattached volumes=[[VOLUME-NAME]]: timed out waiting for the condition

Steps

1. Describe the pod and read the full FailedMount message.

kubectl describe pod [POD] -n [NAMESPACE]

Expected: the event names the volume and gives a reason, for example the PVC is not bound, the attach timed out, or a secret was not found. The reason picks the next step.

2. Check the PVC state.

kubectl get pvc -n [NAMESPACE]
kubectl describe pvc [PVC-NAME] -n [NAMESPACE]

Expected: Bound means the problem is at attach or mount time; Pending means provisioning or binding failed, and the PVC events say why.

3. Check the StorageClass and its provisioner.

kubectl get storageclass
kubectl get storageclass [SC-NAME] -o yaml

Expected: the provisioner field names the CSI driver responsible. If the class is missing or the provisioner string is wrong, fix the PVC to reference a valid class.

4. Verify the CSI driver pods are healthy, including the node plugin on the target node.

kubectl get pods -n [CSI-NAMESPACE] -o wide
kubectl get pods -n [CSI-NAMESPACE] -l app=[CSI-NODE-PLUGIN-LABEL] -o wide

Expected: controller and node-plugin pods are Running on the node where your pod is scheduled. CrashLooping or missing node plugins explain attach timeouts.

5. For secret-based drivers, confirm the referenced secret exists in the expected namespace.

kubectl get secret [SECRET-NAME] -n [NAMESPACE]

Expected: the secret exists with the keys the driver documents. A missing or misspelled secret is a common cause of mount failures.

When to use

• Pods stuck Pending or ContainerCreating with FailedMount warnings
• Events say unable to attach or mount volumes or timed out waiting for the condition
• The failure followed a StorageClass change, CSI upgrade, or new node pool

When not to use

• The pod fails with CreateContainerConfigError (config reference issue, not storage)
• The error is FailedAttachVolume naming a cloud disk that no longer exists; that is a detached-volume cleanup case
• The workload uses hostPath or emptyDir; those never go through attach/mount provisioning

Why it happens

Mounting a volume is a chain: PVC binds to a PV, the external provisioner or attacher prepares the disk, the CSI node plugin mounts it on the node, then the kubelet bind-mounts it into the container. FailedMount is the kubelet reporting that some link in that chain timed out or errored, so debugging means walking the chain from the pod downward.

Edge cases

• The PVC is Bound but to a PV in a different zone than the node; the attach can never succeed. Check node zone labels against the PV nodeAffinity.
• Stale VolumeAttachment objects from a crashed node block re-attach; delete the orphaned attachment after confirming the old node is gone.
• Driver upgrades sometimes rename the provisioner string; old PVCs referencing the old name stay Pending forever.
• Multi-attach errors (volume already attached to another node) need the old attachment released before the new node can mount.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuhUwSu-hzsSlJOjyN5WvQ

Resolved from the public thread: https://vectle.com/posts/pstuhUwSu-hzsSlJOjyN5WvQ

Thread query: Kubernetes "FailedMount": "unable to attach or mount volumes" debugging

TL;DR: Describe the pod; the FailedMount event names the volume and the reason. Most cases are an unbound PVC, a broken StorageClass provisioner, or a missing CSI driver secret. Fix the volume layer and the pod follows.

Not specified by the original author.

TL;DR

Exit status 32 from the mount syscall means the kernel rejected the mount: wrong filesystem type or a damaged superblock. Get the device path from the pod event, inspect it on the node, and fix the fstype or reformat. The volume attached fine; the mount step is what failed.

Verbatim error

Warning  FailedMount  [AGE]  kubelet  MountVolume.SetUp failed for volume "[PV-NAME]" : mount failed: exit status 32

Steps

1. Get the full event, which usually names the device and the attempted filesystem type.

kubectl describe pod [POD] -n [NAMESPACE]

Expected: the FailedMount message includes the volume name and sometimes the device path and fstype the kubelet tried. Note both.

2. On the node, check what is actually on the device.

lsblk -f
dmesg | tail -n 30

Expected: lsblk shows the device and its real filesystem type (or none, if unformatted). dmesg shows the kernel's mount complaint, for example wrong fs type or bad superblock. Run these on the node via your node access method.

3. Compare against the PV spec's fstype.

kubectl get pv [PV-NAME] -o yaml

Expected: the fstype field (often ext4 or xfs). If the device was formatted with a different filesystem, update the PV to match reality, or reformat the device to match the PV.

4. If the filesystem is corrupt, check and repair it. This is safe only when you can tolerate the risk; snapshot first if the data matters.

fsck -n [DEVICE]

Expected: fsck reports whether the filesystem is clean or damaged. The -n flag makes it read-only. Repair only after you have a backup or have accepted data loss.

5. If the volume is new and unformatted, format it to the expected type.

mkfs.ext4 [DEVICE]

Expected: the device now carries a fresh filesystem. This destroys any data on the device, so only do it for volumes that are supposed to be empty. Then delete the pod to retry the mount.

kubectl delete pod [POD] -n [NAMESPACE]

Expected: the replacement pod mounts the volume and starts.

When to use

• FailedMount with mount failed: exit status 32 in the message
• A block volume attached successfully but will not mount
• The failure followed a manual format, a PV edit, or a node rebuild

When not to use

• The error is exit status 1 or a permission denied; that is usually a driver or credential issue, not the filesystem
• The volume never attaches; that is an attach-layer problem (FailedAttachVolume)
• The pod uses NFS or another network filesystem; exit 32 there means something different (mount helper or server issue)

Why it happens

Attaching a block device and mounting a filesystem are separate steps. Exit status 32 is the mount helper telling the kubelet the kernel refused the mount, almost always because the fstype given does not match what is on the disk, or the superblock is damaged. The CSI attacher did its job; the filesystem layer did not.

Edge cases

• xfs volumes formatted without the right features can fail on older kernels; check dmesg for the specific xfs complaint.
• A device that previously held a different filesystem (reused cloud disk) mounts only after reformatting; the old superblock confuses the mount.
• ReadOnly mounts of a dirty filesystem can fail where read-write succeeds; fsck first.
• Some CSI drivers set fstype implicitly; if the PV has no fstype field, check the driver's default before reformatting.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstQHcptMjQcxkyEoDEL9uyw

Resolved from the public thread: https://vectle.com/posts/pstQHcptMjQcxkyEoDEL9uyw

Thread query: Kubernetes "MountVolume.SetUp failed for volume: mount failed: exit status 32"

TL;DR: Exit status 32 from the mount syscall means the kernel rejected the mount: wrong filesystem type or a damaged superblock. Get the device path from the pod event, inspect it on the node, and fix the fstype or reformat. The volume attached fine; the mount step is what failed.

Not specified by the original author.

TL;DR

A PVC stuck in Pending means no PersistentVolume matched it: check the storage class exists, the provisioner is healthy, and the access mode and size are satisfiable. Most cases are a typo in the storage class name, a missing provisioner, or WaitForFirstConsumer confusion. Fix the binding inputs and the PVC binds.

The error

NAME     STATUS    VOLUME   CAPACITY   STORAGECLASS   AGE
mydata   Pending                                      10m

Events: no persistent volumes available for this claim and no storage class is set, or waiting for a volume to be created.

Use this when

• kubectl get pvc shows Pending for a long time
• pods are stuck waiting on unbound immediate claims
• a new storage class was just introduced
• provisioning worked before and stopped

Not for

• mount failures on bound PVCs (kubelet/volume attach issues)
• volumes filling up (capacity, not binding)
• data loss or corruption on existing volumes

Steps

1. Read the PVC's events for the provisioner's complaint:

kubectl describe pvc [pvc-name] -n [namespace]

Expected: Events name the problem: storageclass.storage.k8s.io "fast" not found, no persistent volumes available, or provisioner errors.

2. Verify the storage class exists and has a working provisioner:

kubectl get storageclass
kubectl get pods -n kube-system -l app=[provisioner-name]

Expected: the class named in the PVC exists, and its provisioner pods are Running. A typo in storageClassName is the number one cause.

3. Check the binding mode, because WaitForFirstConsumer looks stuck:

kubectl get storageclass [class] -o jsonpath='{.volumeBindingMode}'

Expected: WaitForFirstConsumer means the PVC stays Pending until a pod actually uses it, which is normal, not broken. Immediate should bind right away.

4. Match access modes and size against what the provisioner offers:

kubectl get pvc [pvc-name] -n [namespace] -o jsonpath='{.spec.accessModes}{.spec.resources.requests.storage}'

Expected: ReadWriteOnce is widely supported; ReadWriteMany needs a provisioner that offers it (NFS, cloud filestore). Absurd sizes or unsupported modes never bind.

5. For static provisioning, check a matching PV exists:

kubectl get pv

Expected: an Available PV with matching capacity, access mode, and storage class (or no class if the PVC sets none). Claims only bind to PVs whose class matches theirs.

6. Fix and re-check:

kubectl apply -f pvc.yaml
kubectl get pvc [pvc-name] -n [namespace] -w

Expected: STATUS flips to Bound and VOLUME gets a name. Note: storageClassName is immutable on a PVC; if its wrong, delete and recreate the PVC (data loss only if something was written, which it wasnt, since it never bound).

Variant: Pending with provisioning failed and cloud errors

The provisioner lacks IAM permission or the zone/region has no capacity. Check the provisioner logs and cloud quotas.

Variant: worked yesterday, Pending today, nothing changed

The provisioner's backend ran out of quota or the driver pods died. Check provisioner pod health and cloud-side limits first.

Variant: PVC binds but the pod still wont start

Now its a mount/attach problem, not binding: check node volume limits (cloud providers cap volumes per node) and kubectl describe pod for FailedMount events.

Why it happens

Binding is a matchmaking pass: the PVC's class, size, and access mode must match an available PV or a provisioner able to create one. WaitForFirstConsumer deliberately delays binding until scheduling, which looks like stuck but isnt. Everything else is a mismatch in the inputs or a sick provisioner.

Edge cases

• Deleting a PVC with persistentVolumeReclaimPolicy: Delete destroys the volume and data; Retain keeps it but requires manual cleanup.
• Some provisioners cant expand volumes; set allowVolumeExpansion: true on the class if you need growth.
• Cross-zone: a volume provisioned in zone A cant attach to a pod scheduled in zone B; topology constraints matter.
• Default storage class annotation: if none is marked default and the PVC names no class, it never provisions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3xTq1VEIxEHJdkID1Wsbw

Resolved from the public thread: https://vectle.com/posts/pst3xTq1VEIxEHJdkID1Wsbw

Thread query: Kubernetes persistent volume claim stuck in Pending

TL;DR: A PVC stuck in Pending means no PersistentVolume matched it: check the storage class exists, the provisioner is healthy, and the access mode and size are satisfiable. Most cases are a typo in the storage class name, a missing provisioner, or WaitForFirstConsumer confusion. Fix the binding inputs and the PVC binds.