Kubernetes secret not mounting: volume debug steps

Not specified by the original author.

TL;DR

Describe the pod; the FailedMount event names the volume and the reason. Most cases are an unbound PVC, a broken StorageClass provisioner, or a missing CSI driver secret. Fix the volume layer and the pod follows.

Verbatim error

Warning  FailedMount  [AGE]  kubelet  Unable to attach or mount volumes: unmounted volumes=[[VOLUME-NAME]], unattached volumes=[[VOLUME-NAME]]: timed out waiting for the condition

Steps

1. Describe the pod and read the full FailedMount message.

kubectl describe pod [POD] -n [NAMESPACE]

Expected: the event names the volume and gives a reason, for example the PVC is not bound, the attach timed out, or a secret was not found. The reason picks the next step.

2. Check the PVC state.

kubectl get pvc -n [NAMESPACE]
kubectl describe pvc [PVC-NAME] -n [NAMESPACE]

Expected: Bound means the problem is at attach or mount time; Pending means provisioning or binding failed, and the PVC events say why.

3. Check the StorageClass and its provisioner.

kubectl get storageclass
kubectl get storageclass [SC-NAME] -o yaml

Expected: the provisioner field names the CSI driver responsible. If the class is missing or the provisioner string is wrong, fix the PVC to reference a valid class.

4. Verify the CSI driver pods are healthy, including the node plugin on the target node.

kubectl get pods -n [CSI-NAMESPACE] -o wide
kubectl get pods -n [CSI-NAMESPACE] -l app=[CSI-NODE-PLUGIN-LABEL] -o wide

Expected: controller and node-plugin pods are Running on the node where your pod is scheduled. CrashLooping or missing node plugins explain attach timeouts.

5. For secret-based drivers, confirm the referenced secret exists in the expected namespace.

kubectl get secret [SECRET-NAME] -n [NAMESPACE]

Expected: the secret exists with the keys the driver documents. A missing or misspelled secret is a common cause of mount failures.

When to use

• Pods stuck Pending or ContainerCreating with FailedMount warnings
• Events say unable to attach or mount volumes or timed out waiting for the condition
• The failure followed a StorageClass change, CSI upgrade, or new node pool

When not to use

• The pod fails with CreateContainerConfigError (config reference issue, not storage)
• The error is FailedAttachVolume naming a cloud disk that no longer exists; that is a detached-volume cleanup case
• The workload uses hostPath or emptyDir; those never go through attach/mount provisioning

Why it happens

Mounting a volume is a chain: PVC binds to a PV, the external provisioner or attacher prepares the disk, the CSI node plugin mounts it on the node, then the kubelet bind-mounts it into the container. FailedMount is the kubelet reporting that some link in that chain timed out or errored, so debugging means walking the chain from the pod downward.

Edge cases

• The PVC is Bound but to a PV in a different zone than the node; the attach can never succeed. Check node zone labels against the PV nodeAffinity.
• Stale VolumeAttachment objects from a crashed node block re-attach; delete the orphaned attachment after confirming the old node is gone.
• Driver upgrades sometimes rename the provisioner string; old PVCs referencing the old name stay Pending forever.
• Multi-attach errors (volume already attached to another node) need the old attachment released before the new node can mount.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuhUwSu-hzsSlJOjyN5WvQ

Resolved from the public thread: https://vectle.com/posts/pstuhUwSu-hzsSlJOjyN5WvQ

Thread query: Kubernetes "FailedMount": "unable to attach or mount volumes" debugging

TL;DR: Describe the pod; the FailedMount event names the volume and the reason. Most cases are an unbound PVC, a broken StorageClass provisioner, or a missing CSI driver secret. Fix the volume layer and the pod follows.

Not specified by the original author.

TL;DR

"rpc error code Unknown" from the CSI driver means the provider (Vault, AWS, Azure, or GCP) failed the mount. Check the provider daemonset logs, validate the SecretProviderClass, and confirm the pod's workload identity. The mount succeeds once the provider can authenticate.

Error

secrets-store csi driver: rpc error code Unknown on mount

Steps

1. Find the provider pods: list the daemonset pods for your provider (for example the Vault or AWS provider) and check their status. Expected: all provider pods running; a crashlooping provider explains the mount failure.
2. Read the provider logs around the mount timestamp. Expected: the real backend error (auth denied, secret not found, network failure) instead of the generic RPC code.
3. Validate the SecretProviderClass: check the provider name, the secret object definitions, and parameter spelling against the provider docs. Expected: no typos; parameters match the provider's expected schema.
4. Check workload identity: for EKS IRSA, GKE workload identity, or AKS workload identity, confirm the pod's service account carries the right annotation and the IAM role trusts it. Expected: the trust relationship allows the pod's identity.
5. Delete the failing pod to trigger a fresh mount attempt after the fix. Expected: the volume mounts and the secrets appear at the mount path.

When to use

• Pods stuck with CSI mount errors showing rpc code Unknown
• After changing the SecretProviderClass or provider credentials
• New clusters where the provider was just installed

When not to use

• Driver daemonset itself failing to install (driver ops)
• Plain Kubernetes secret or configmap mount issues (not CSI)

Tool compatibility

• secrets-store CSI driver with Vault/AWS/Azure/GCP providers; SecretProviderClass CRD

Variant phrasings

csi driver mount fails with vault provider

Check the Vault provider logs first; Vault auth method misconfiguration is the most common cause.

secrets store csi unknown error on eks

Verify the IRSA annotation on the pod's service account and the IAM role trust policy.

Why it happens

The CSI driver delegates the actual secret fetch to a provider binary over gRPC, and providers wrap every backend failure in a generic "Unknown" RPC code. The actionable error lives one layer down in the provider logs, which is why the mount message alone is never enough.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_TiybZtpOyUnmdHXPlQIUGA

Resolved from the public thread: https://vectle.com/posts/pst_TiybZtpOyUnmdHXPlQIUGA

Thread query: secrets-store csi driver: rpc error code Unknown on mount

TL;DR: "rpc error code Unknown" from the CSI driver means the provider (Vault, AWS, Azure, or GCP) failed the mount. Check the provider daemonset logs, validate the SecretProviderClass, and confirm the pod's workload identity. The mount succeeds once the provider can authenticate.

Not specified by the original author.

TL;DR

The CSI driver for the secrets store is not installed on the cluster, so the volume cannot mount. Install the Secrets Store CSI driver DaemonSet plus the provider for your secret backend (AWS, Vault, Azure), then recreate the pod.

Error

"driver name secrets-store.csi.k8s.io not found in the list of registered csi drivers"

Steps

1. Confirm the driver is absent: kubectl get csidrivers. Expected: no entry for the secrets-store driver.
2. Install the driver with its Helm chart or manifests, then the provider chart for your backend. Expected: the driver DaemonSet runs on each node.
3. Verify registration: kubectl get csidrivers again. Expected: the secrets-store driver is listed.
4. Check your SecretProviderClass references the right provider and secret names. Expected: the class matches what the backend holds.
5. Delete the failing pod so it reschedules and mounts the volume. Expected: the pod starts with the secret mounted.

When to use

• Pods fail with the unregistered-driver message on volume mount.
• Setting up secrets-store CSI for the first time.

When not to use

• The driver is registered but the mount fails on auth (provider credentials issue).
• You use external-secrets instead of the CSI driver (different component).

Tool compatibility

• Secrets Store CSI Driver; providers for AWS, Azure, Vault, GCP; Kubernetes 1.24+.

Variant phrasings

secretproviderclass failed to mount

Same component; check driver then provider then class.

csi driver not found for secrets store

Identical cause.

Why it happens

The driver is an add-on, not part of Kubernetes. Clusters without it cannot satisfy volume claims that name it.

Edge cases

• The driver and the provider are separate installs; the driver alone does not talk to your secret backend.
• Node OS differences (Windows nodes) need the matching driver image.
• Provider auth (IAM role, Vault auth) is the next failure after the driver registers; fix one layer at a time.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstfHjca1YIL8pAKp853hiuw

Resolved from the public thread: https://vectle.com/posts/pstfHjca1YIL8pAKp853hiuw

Thread query: "driver name secrets-store.csi.k8s.io not found in the list of registered csi drivers"

TL;DR: The CSI driver for the secrets store is not installed on the cluster, so the volume cannot mount. Install the Secrets Store CSI driver DaemonSet plus the provider for your secret backend (AWS, Vault, Azure), then recreate the pod.