Rook-Ceph mons are not in quorum: recovery steps

Not specified by the original author.

TL;DR

Get into the toolbox pod and check quorum with ceph status before touching anything. Restart unhealthy mon pods one at a time and wait for quorum between restarts; only use the mon store disaster-recovery procedure if quorum is truly unrecoverable.

Verbatim error

mons are not in quorum: mons [MON-LIST] are out of quorum

Steps

1. Check which mons are up and whether the cluster has quorum, from the toolbox.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status

Expected: the pod list shows which mon pods are Running, and ceph status shows the mon quorum list. Partial quorum (at least a majority of mons up) means careful restarts will recover it.

2. Read the logs of the unhealthy mon pods.

kubectl -n rook-ceph logs [MON-POD]

Expected: the logs name the problem: disk full on the mon data PVC, clock skew, network partition, or corrupted store. Fix the underlying cause (free disk, fix NTP, restore network) before restarting.

3. Restart unhealthy mon pods one at a time, waiting for quorum between each.

kubectl -n rook-ceph delete pod [MON-POD]
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status

Expected: after each restart, ceph status shows the mon rejoining quorum. Never restart all mons at once; losing the last quorum member turns a recoverable outage into a disaster-recovery exercise.

4. If quorum is completely lost and mons cannot restart, use the Rook mon disaster-recovery procedure: it rebuilds the mon store from the surviving mon data. Follow the Rook documentation for your exact version, since the steps differ between releases, and run it against one mon first.

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph mon dump

Expected: once recovery succeeds, mon dump shows a fresh monmap and the mons re-establish quorum. This is a last resort; it risks data loss if done wrong.

5. Verify full health and watch for slow recovery.

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail

Expected: HEALTHOK or HEALTHWARN with a clear reason, all mons in quorum, and OSDs re-peering. Placement groups may take a long time to go active and clean on large clusters; that is normal.

When to use

• ceph status reports mons out of quorum or mon pods crashloop
• The cluster went read-only or I/O stalled after mon failures
• You are planning the recovery order for a multi-mon outage

When not to use

• OSDs are down but mons have quorum; that is an OSD recovery problem
• A single mon pod is crashlooping but quorum holds; just fix that mon
• The operator pod is failing; fix the operator before touching mons

Why it happens

Ceph monitors keep cluster state with Paxos and need a majority (quorum) to make progress. Losing quorum, usually from multiple mon pods dying at once (node failures, full disks, bad upgrades), freezes all cluster operations: no I/O, no OSD map updates, nothing. Recovery is about restoring a majority safely.

Edge cases

• Mon data PVCs full is the most common silent killer; monitor mon disk usage, not just OSD usage.
• Clock skew over a few dozen milliseconds between mons breaks Paxos; fix NTP cluster-wide, not just on one node.
• Never delete mon PVCs to force recreation unless the disaster-recovery docs say so; empty mons cannot rejoin and you lose the store.
• After recovery, some placement groups may stay stale or undersized for hours; let backfill finish before declaring victory.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-93XBvIk8SeT1BZB70oQwQ

Resolved from the public thread: https://vectle.com/posts/pst_-93XBvIk8SeT1BZB70oQwQ

Thread query: Rook-Ceph "mons are not in quorum": recovery steps

TL;DR: Get into the toolbox pod and check quorum with ceph status before touching anything. Restart unhealthy mon pods one at a time and wait for quorum between restarts; only use the mon store disaster-recovery procedure if quorum is truly unrecoverable.

Not specified by the original author.

TL;DR

The replica set lost quorum: too few healthy voting members to elect a primary, so the set is read-only. Check rs.status() for member states, restore network or members to regain a majority, and reconfigure only if members are permanently gone. Writes resume once a primary is elected.

Error

mongodb: no primary found after election

Steps

1. Check the set: run rs.status() and note each member's state and health. Expected: you see which members are down, unreachable, or in rollback.
2. Count the votes: a primary needs a majority of voting members reachable. Expected: you know whether quorum is mathematically possible right now.
3. Restore the members: fix the network partition, restart the down nodes, or resolve the resource exhaustion keeping them down. Expected: members rejoin and an election succeeds.
4. If members are permanently gone, reconfigure the set to a smaller voting membership from a surviving member. Expected: the set elects a primary with the remaining nodes.
5. Verify: confirm a primary is elected and writes succeed. Expected: the set is healthy and writable.

When to use

• Replica set has no primary after an election or outage
• Network partitions between members
• Permanently losing members

When not to use

• Drivers cannot connect to a healthy set (driver config)
• A primary exists but is slow (performance issue)

Tool compatibility

• MongoDB replica sets (rs.status(), reconfig); voting and quorum rules

Variant phrasings

mongodb election no primary

Restore a voting majority; without quorum no election can succeed.

replica set read only no primary

The set is correctly refusing writes until a primary exists; fix membership.

Why it happens

MongoDB requires a strict majority of voting members to elect a primary, which guarantees a single writer. Partitions, downed nodes, or misconfigured voting members break the majority, and the set stays primary-less (safely read-only) until quorum returns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst9gY5eDisy-c8uQhgq_L5A

Resolved from the public thread: https://vectle.com/posts/pst9gY5eDisy-c8uQhgq_L5A

Thread query: mongodb: no primary found after election

TL;DR: The replica set lost quorum: too few healthy voting members to elect a primary, so the set is read-only. Check rs.status() for member states, restore network or members to regain a majority, and reconfigure only if members are permanently gone. Writes resume once a primary is elected.

Not specified by the original author.

TL;DR

Check the volume's state and its replicas before doing anything destructive; stuck attaching is usually a dead engine on a removed node or a rebuild blocked by disk space. Detach and reattach cleanly, and salvage replicas before deleting anything if data is at risk.

Verbatim error

volume [VOLUME-NAME] stuck at attaching: engine [ENGINE-NAME] failed to start

Steps

1. Read the volume state, robustness, and attached node.

kubectl -n longhorn-system get volumes.longhorn.io [VOLUME-NAME] -o yaml

Expected: the state (attaching, attached, detached), robustness (healthy, degraded, faulted), and which node it is attached to. A volume attaching to a node that no longer exists explains the stuck state.

2. Check the replicas and which ones are healthy.

kubectl -n longhorn-system get replicas.longhorn.io -l longhornvolume=[VOLUME-NAME]

Expected: each replica's state and node. Failed or stopped replicas on deleted nodes block rebuilds; healthy replicas elsewhere are your safety net.

3. Check disk space and disk pressure on the Longhorn nodes.

kubectl -n longhorn-system get nodes.longhorn.io -o yaml

Expected: per-disk available space and conditions. Rebuilds need free space; a full disk silently stalls the rebuild with no obvious error on the volume.

4. Detach cleanly and reattach: scale the workload to zero first so nothing holds the volume.

kubectl -n [NAMESPACE] scale deployment/[WORKLOAD] --replicas=0

Expected: the workload pods terminate. Then detach the volume through the Longhorn UI or by updating the volume CR, wait for state detached, and scale the workload back up so it attaches fresh.

kubectl -n [NAMESPACE] scale deployment/[WORKLOAD] --replicas=[COUNT]

Expected: the volume attaches to a healthy node and the pods mount it.

5. If replicas are faulted and data is at risk, salvage before deleting anything: Longhorn can rebuild from the surviving replica data. Use the salvage flow in the UI or CLI, verify the volume mounts and the data is intact, and only then clean up the failed replicas.

kubectl -n longhorn-system get engines.longhorn.io -l longhornvolume=[VOLUME-NAME]

Expected: a running engine on the salvaged volume. Confirm your application reads its data correctly before considering the incident closed.

When to use

• Longhorn volumes stuck in attaching or degraded with rebuilds that never finish
• Workloads cannot mount volumes after a node was removed or rebuilt
• Replica rebuild failures after disk or node problems

When not to use

• The volume is healthy and the app cannot read its files; that is filesystem or app-level, not Longhorn
• The whole Longhorn manager is down; fix the manager pods first
• You need more replicas for performance; that is a replica-count setting, not a failure

Why it happens

Longhorn keeps each volume as a set of replicas coordinated by an engine process on the attached node. If that node disappears, the engine dies mid-attach; if disks fill, rebuilds stall; if replicas disagree, the volume degrades. The volume CR records all of this, so reading it first prevents destructive guesses.

Edge cases

• Force-deleting a stuck volume CR can orphan data on disks; always detach through Longhorn rather than deleting the CR.
• Rebuilding to a node with a nearly full disk thrashes: it starts, stalls, and retries forever. Free space first.
• Engine images pinned to an old version after an upgrade can fail to start; check the engine image matches the manager version.
• Volumes with a single replica have no safety net; fix the disk or node before touching the replica, and add replicas afterward.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SLa5FYe2vEoG2FH4SbtTXQ

Resolved from the public thread: https://vectle.com/posts/pst_SLa5FYe2vEoG2FH4SbtTXQ

Thread query: Longhorn "volume stuck attaching" / replica rebuild failures

TL;DR: Check the volume's state and its replicas before doing anything destructive; stuck attaching is usually a dead engine on a removed node or a rebuild blocked by disk space. Detach and reattach cleanly, and salvage replicas before deleting anything if data is at risk.