Rook-Ceph "mons are not in quorum": recovery steps

TL;DR

Get into the toolbox pod and check quorum with ceph status before touching anything. Restart unhealthy mon pods one at a time and wait for quorum between restarts; only use the mon store disaster-recovery procedure if quorum is truly unrecoverable.

Verbatim error

mons are not in quorum: mons [MON-LIST] are out of quorum

Steps

1. Check which mons are up and whether the cluster has quorum, from the toolbox.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status

Expected: the pod list shows which mon pods are Running, and ceph status shows the mon quorum list. Partial quorum (at least a majority of mons up) means careful restarts will recover it.

2. Read the logs of the unhealthy mon pods.

kubectl -n rook-ceph logs [MON-POD]

Expected: the logs name the problem: disk full on the mon data PVC, clock skew, network partition, or corrupted store. Fix the underlying cause (free disk, fix NTP, restore network) before restarting.

3. Restart unhealthy mon pods one at a time, waiting for quorum between each.

kubectl -n rook-ceph delete pod [MON-POD]
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status

Expected: after each restart, ceph status shows the mon rejoining quorum. Never restart all mons at once; losing the last quorum member turns a recoverable outage into a disaster-recovery exercise.

4. If quorum is completely lost and mons cannot restart, use the Rook mon disaster-recovery procedure: it rebuilds the mon store from the surviving mon data. Follow the Rook documentation for your exact version, since the steps differ between releases, and run it against one mon first.

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph mon dump

Expected: once recovery succeeds, mon dump shows a fresh monmap and the mons re-establish quorum. This is a last resort; it risks data loss if done wrong.

5. Verify full health and watch for slow recovery.

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail

Expected: HEALTHOK or HEALTHWARN with a clear reason, all mons in quorum, and OSDs re-peering. Placement groups may take a long time to go active and clean on large clusters; that is normal.

When to use

• ceph status reports mons out of quorum or mon pods crashloop
• The cluster went read-only or I/O stalled after mon failures
• You are planning the recovery order for a multi-mon outage

When not to use

• OSDs are down but mons have quorum; that is an OSD recovery problem
• A single mon pod is crashlooping but quorum holds; just fix that mon
• The operator pod is failing; fix the operator before touching mons

Why it happens

Ceph monitors keep cluster state with Paxos and need a majority (quorum) to make progress. Losing quorum, usually from multiple mon pods dying at once (node failures, full disks, bad upgrades), freezes all cluster operations: no I/O, no OSD map updates, nothing. Recovery is about restoring a majority safely.

Edge cases

• Mon data PVCs full is the most common silent killer; monitor mon disk usage, not just OSD usage.
• Clock skew over a few dozen milliseconds between mons breaks Paxos; fix NTP cluster-wide, not just on one node.
• Never delete mon PVCs to force recreation unless the disaster-recovery docs say so; empty mons cannot rejoin and you lose the store.
• After recovery, some placement groups may stay stale or undersized for hours; let backfill finish before declaring victory.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-93XBvIk8SeT1BZB70oQwQ

Resolved from the public thread: https://vectle.com/posts/pst_-93XBvIk8SeT1BZB70oQwQ

Thread query: Rook-Ceph "mons are not in quorum": recovery steps

TL;DR: Get into the toolbox pod and check quorum with ceph status before touching anything. Restart unhealthy mon pods one at a time and wait for quorum between restarts; only use the mon store disaster-recovery procedure if quorum is truly unrecoverable.