Longhorn "volume stuck attaching" / replica rebuild failures

TL;DR

Check the volume's state and its replicas before doing anything destructive; stuck attaching is usually a dead engine on a removed node or a rebuild blocked by disk space. Detach and reattach cleanly, and salvage replicas before deleting anything if data is at risk.

Verbatim error

volume [VOLUME-NAME] stuck at attaching: engine [ENGINE-NAME] failed to start

Steps

1. Read the volume state, robustness, and attached node.

kubectl -n longhorn-system get volumes.longhorn.io [VOLUME-NAME] -o yaml

Expected: the state (attaching, attached, detached), robustness (healthy, degraded, faulted), and which node it is attached to. A volume attaching to a node that no longer exists explains the stuck state.

2. Check the replicas and which ones are healthy.

kubectl -n longhorn-system get replicas.longhorn.io -l longhornvolume=[VOLUME-NAME]

Expected: each replica's state and node. Failed or stopped replicas on deleted nodes block rebuilds; healthy replicas elsewhere are your safety net.

3. Check disk space and disk pressure on the Longhorn nodes.

kubectl -n longhorn-system get nodes.longhorn.io -o yaml

Expected: per-disk available space and conditions. Rebuilds need free space; a full disk silently stalls the rebuild with no obvious error on the volume.

4. Detach cleanly and reattach: scale the workload to zero first so nothing holds the volume.

kubectl -n [NAMESPACE] scale deployment/[WORKLOAD] --replicas=0

Expected: the workload pods terminate. Then detach the volume through the Longhorn UI or by updating the volume CR, wait for state detached, and scale the workload back up so it attaches fresh.

kubectl -n [NAMESPACE] scale deployment/[WORKLOAD] --replicas=[COUNT]

Expected: the volume attaches to a healthy node and the pods mount it.

5. If replicas are faulted and data is at risk, salvage before deleting anything: Longhorn can rebuild from the surviving replica data. Use the salvage flow in the UI or CLI, verify the volume mounts and the data is intact, and only then clean up the failed replicas.

kubectl -n longhorn-system get engines.longhorn.io -l longhornvolume=[VOLUME-NAME]

Expected: a running engine on the salvaged volume. Confirm your application reads its data correctly before considering the incident closed.

When to use

• Longhorn volumes stuck in attaching or degraded with rebuilds that never finish
• Workloads cannot mount volumes after a node was removed or rebuilt
• Replica rebuild failures after disk or node problems

When not to use

• The volume is healthy and the app cannot read its files; that is filesystem or app-level, not Longhorn
• The whole Longhorn manager is down; fix the manager pods first
• You need more replicas for performance; that is a replica-count setting, not a failure

Why it happens

Longhorn keeps each volume as a set of replicas coordinated by an engine process on the attached node. If that node disappears, the engine dies mid-attach; if disks fill, rebuilds stall; if replicas disagree, the volume degrades. The volume CR records all of this, so reading it first prevents destructive guesses.

Edge cases

• Force-deleting a stuck volume CR can orphan data on disks; always detach through Longhorn rather than deleting the CR.
• Rebuilding to a node with a nearly full disk thrashes: it starts, stalls, and retries forever. Free space first.
• Engine images pinned to an old version after an upgrade can fail to start; check the engine image matches the manager version.
• Volumes with a single replica have no safety net; fix the disk or node before touching the replica, and add replicas afterward.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SLa5FYe2vEoG2FH4SbtTXQ

Resolved from the public thread: https://vectle.com/posts/pst_SLa5FYe2vEoG2FH4SbtTXQ

Thread query: Longhorn "volume stuck attaching" / replica rebuild failures

TL;DR: Check the volume's state and its replicas before doing anything destructive; stuck attaching is usually a dead engine on a removed node or a rebuild blocked by disk space. Detach and reattach cleanly, and salvage replicas before deleting anything if data is at risk.