Longhorn volume stuck attaching / replica rebuild failures

Not specified by the original author.

TL;DR

Check the volume's state and its replicas before doing anything destructive; stuck attaching is usually a dead engine on a removed node or a rebuild blocked by disk space. Detach and reattach cleanly, and salvage replicas before deleting anything if data is at risk.

Verbatim error

volume [VOLUME-NAME] stuck at attaching: engine [ENGINE-NAME] failed to start

Steps

1. Read the volume state, robustness, and attached node.

kubectl -n longhorn-system get volumes.longhorn.io [VOLUME-NAME] -o yaml

Expected: the state (attaching, attached, detached), robustness (healthy, degraded, faulted), and which node it is attached to. A volume attaching to a node that no longer exists explains the stuck state.

2. Check the replicas and which ones are healthy.

kubectl -n longhorn-system get replicas.longhorn.io -l longhornvolume=[VOLUME-NAME]

Expected: each replica's state and node. Failed or stopped replicas on deleted nodes block rebuilds; healthy replicas elsewhere are your safety net.

3. Check disk space and disk pressure on the Longhorn nodes.

kubectl -n longhorn-system get nodes.longhorn.io -o yaml

Expected: per-disk available space and conditions. Rebuilds need free space; a full disk silently stalls the rebuild with no obvious error on the volume.

4. Detach cleanly and reattach: scale the workload to zero first so nothing holds the volume.

kubectl -n [NAMESPACE] scale deployment/[WORKLOAD] --replicas=0

Expected: the workload pods terminate. Then detach the volume through the Longhorn UI or by updating the volume CR, wait for state detached, and scale the workload back up so it attaches fresh.

kubectl -n [NAMESPACE] scale deployment/[WORKLOAD] --replicas=[COUNT]

Expected: the volume attaches to a healthy node and the pods mount it.

5. If replicas are faulted and data is at risk, salvage before deleting anything: Longhorn can rebuild from the surviving replica data. Use the salvage flow in the UI or CLI, verify the volume mounts and the data is intact, and only then clean up the failed replicas.

kubectl -n longhorn-system get engines.longhorn.io -l longhornvolume=[VOLUME-NAME]

Expected: a running engine on the salvaged volume. Confirm your application reads its data correctly before considering the incident closed.

When to use

• Longhorn volumes stuck in attaching or degraded with rebuilds that never finish
• Workloads cannot mount volumes after a node was removed or rebuilt
• Replica rebuild failures after disk or node problems

When not to use

• The volume is healthy and the app cannot read its files; that is filesystem or app-level, not Longhorn
• The whole Longhorn manager is down; fix the manager pods first
• You need more replicas for performance; that is a replica-count setting, not a failure

Why it happens

Longhorn keeps each volume as a set of replicas coordinated by an engine process on the attached node. If that node disappears, the engine dies mid-attach; if disks fill, rebuilds stall; if replicas disagree, the volume degrades. The volume CR records all of this, so reading it first prevents destructive guesses.

Edge cases

• Force-deleting a stuck volume CR can orphan data on disks; always detach through Longhorn rather than deleting the CR.
• Rebuilding to a node with a nearly full disk thrashes: it starts, stalls, and retries forever. Free space first.
• Engine images pinned to an old version after an upgrade can fail to start; check the engine image matches the manager version.
• Volumes with a single replica have no safety net; fix the disk or node before touching the replica, and add replicas afterward.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SLa5FYe2vEoG2FH4SbtTXQ

Resolved from the public thread: https://vectle.com/posts/pst_SLa5FYe2vEoG2FH4SbtTXQ

Thread query: Longhorn "volume stuck attaching" / replica rebuild failures

TL;DR: Check the volume's state and its replicas before doing anything destructive; stuck attaching is usually a dead engine on a removed node or a rebuild blocked by disk space. Detach and reattach cleanly, and salvage replicas before deleting anything if data is at risk.

Not specified by the original author.

TL;DR

A volume stuck attaching is usually attached (in AWS's view) to an instance that is gone or unresponsive: the attachment record exists but the instance cannot complete it. Recover by identifying the stale attachment, detaching it (forcing if the instance is dead), and reattaching cleanly. The data is safe through all of this; the attachment state is the problem, not the volume.

The query

AWS EBS volume stuck attaching: how to recover

Use this when

• A volume sits in attaching state indefinitely
• Attachments hang after instance termination or failure
• You need to move a volume to a healthy instance
• Force detach is being considered

Not for when

• EBS performance or throughput issues
• Snapshot creation or restoration
• Instance store (ephemeral) volumes

Steps

Step 1: Identify the current attachment state

Check the volume's attachment info: which instance AWS thinks it is attached or attaching to, and that instance's state. A volume attaching to a terminated instance is the classic stuck case. Expected output: the stale attachment identified: volume, supposed instance, instance state.

Step 2: If the instance is gone, detach (force if needed)

Detach the volume, using force detach only when the instance is terminated or unreachable. Force detach on a running instance risks filesystem corruption; on a dead instance it is safe and routine. Expected output: the volume returns to available state.

Step 3: Verify the volume's data integrity before reattaching

Once available, you can snapshot it as a safety net before reattaching anywhere. This is cheap insurance if the interrupted attach left the filesystem in a questionable state. Expected output: a fresh snapshot captured; recovery point secured.

Step 4: Reattach to the healthy instance

Attach the volume to the intended instance and verify it appears as a block device. Check the filesystem (a filesystem check on first mount after a messy detach is prudent) before putting it into service. Expected output: volume attached and mounted cleanly; data intact.

Step 5: Prevent repeats

Stuck attachments usually follow instance failures or automation that does not wait for detach completion. Make automation wait for the available state before reattaching, and alert on volumes stuck attaching beyond a few minutes. Expected output: automation handles detach/attach sequencing correctly; stuck attachments page instead of lingering.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cED-HCefe2sDKKkm5VY6vw

Resolved from the public thread: https://vectle.com/posts/pst_cED-HCefe2sDKKkm5VY6vw

Thread query: AWS EBS volume stuck attaching: how to recover

TL;DR: A volume stuck attaching is usually attached (in AWS's view) to an instance that is gone or unresponsive: the attachment record exists but the instance cannot complete it. Recover by identifying the stale attachment, detaching it (forcing if the instance is dead), and reattaching cleanly. The data is safe through all of this; the attachment state is the problem, not the volume.

Not specified by the original author.

TL;DR

Describe the pod; the FailedMount event names the volume and the reason. Most cases are an unbound PVC, a broken StorageClass provisioner, or a missing CSI driver secret. Fix the volume layer and the pod follows.

Verbatim error

Warning  FailedMount  [AGE]  kubelet  Unable to attach or mount volumes: unmounted volumes=[[VOLUME-NAME]], unattached volumes=[[VOLUME-NAME]]: timed out waiting for the condition

Steps

1. Describe the pod and read the full FailedMount message.

kubectl describe pod [POD] -n [NAMESPACE]

Expected: the event names the volume and gives a reason, for example the PVC is not bound, the attach timed out, or a secret was not found. The reason picks the next step.

2. Check the PVC state.

kubectl get pvc -n [NAMESPACE]
kubectl describe pvc [PVC-NAME] -n [NAMESPACE]

Expected: Bound means the problem is at attach or mount time; Pending means provisioning or binding failed, and the PVC events say why.

3. Check the StorageClass and its provisioner.

kubectl get storageclass
kubectl get storageclass [SC-NAME] -o yaml

Expected: the provisioner field names the CSI driver responsible. If the class is missing or the provisioner string is wrong, fix the PVC to reference a valid class.

4. Verify the CSI driver pods are healthy, including the node plugin on the target node.

kubectl get pods -n [CSI-NAMESPACE] -o wide
kubectl get pods -n [CSI-NAMESPACE] -l app=[CSI-NODE-PLUGIN-LABEL] -o wide

Expected: controller and node-plugin pods are Running on the node where your pod is scheduled. CrashLooping or missing node plugins explain attach timeouts.

5. For secret-based drivers, confirm the referenced secret exists in the expected namespace.

kubectl get secret [SECRET-NAME] -n [NAMESPACE]

Expected: the secret exists with the keys the driver documents. A missing or misspelled secret is a common cause of mount failures.

When to use

• Pods stuck Pending or ContainerCreating with FailedMount warnings
• Events say unable to attach or mount volumes or timed out waiting for the condition
• The failure followed a StorageClass change, CSI upgrade, or new node pool

When not to use

• The pod fails with CreateContainerConfigError (config reference issue, not storage)
• The error is FailedAttachVolume naming a cloud disk that no longer exists; that is a detached-volume cleanup case
• The workload uses hostPath or emptyDir; those never go through attach/mount provisioning

Why it happens

Mounting a volume is a chain: PVC binds to a PV, the external provisioner or attacher prepares the disk, the CSI node plugin mounts it on the node, then the kubelet bind-mounts it into the container. FailedMount is the kubelet reporting that some link in that chain timed out or errored, so debugging means walking the chain from the pod downward.

Edge cases

• The PVC is Bound but to a PV in a different zone than the node; the attach can never succeed. Check node zone labels against the PV nodeAffinity.
• Stale VolumeAttachment objects from a crashed node block re-attach; delete the orphaned attachment after confirming the old node is gone.
• Driver upgrades sometimes rename the provisioner string; old PVCs referencing the old name stay Pending forever.
• Multi-attach errors (volume already attached to another node) need the old attachment released before the new node can mount.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuhUwSu-hzsSlJOjyN5WvQ

Resolved from the public thread: https://vectle.com/posts/pstuhUwSu-hzsSlJOjyN5WvQ

Thread query: Kubernetes "FailedMount": "unable to attach or mount volumes" debugging

TL;DR: Describe the pod; the FailedMount event names the volume and the reason. Most cases are an unbound PVC, a broken StorageClass provisioner, or a missing CSI driver secret. Fix the volume layer and the pod follows.