consul: no cluster leader after restart

Not specified by the original author.

TL;DR

No leader after restart means the servers cannot form quorum: bootstrap_expect mismatches the actual server count, or a restarted server rejoined with stale peer state. Align the expected count, clear stale data on the failed node if needed, and let the servers elect.

Error

consul: no cluster leader after restart

Steps

1. Count the servers: check how many server agents are actually running versus bootstrap_expect. Expected: the numbers match; a mismatch prevents election.
2. Check the logs on each server for raft election messages and peer complaints. Expected: the logs name the specific problem (stale peer, no quorum, address conflict).
3. If a server restarted with stale state pointing at dead peers, stop it, clear its data directory, and rejoin it to the cluster. Expected: it joins as a fresh voter.
4. Fix bootstrap_expect to the true server count and restart the servers consistently. Expected: the servers agree on quorum size.
5. Verify leadership: query the leader endpoint until a leader is elected. Expected: a stable leader and healthy raft peers.

When to use

• Consul has no leader after restarts or upgrades
• "Failed to get leader" errors from clients
• Changing the server count

When not to use

• Client agents cannot reach a healthy cluster (networking)
• Single-server dev mode (no election expected)

Tool compatibility

• Consul server agents, raft; bootstrap_expect, data directory, join operations

Variant phrasings

consul failed to elect leader

Check bootstrap_expect against the live server count first.

consul stale peer after restart

Clear the failed node's data dir and rejoin; stale peer lists block elections.

Why it happens

Raft needs a quorum of voters agreeing on membership. A restarted node with an outdated peer list, or a bootstrap_expect that no longer matches reality, leaves the servers unable to agree, so no leader is elected and the cluster is read-only at best.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__41hiyQoid6qrIj87MYB1g

Resolved from the public thread: https://vectle.com/posts/pst__41hiyQoid6qrIj87MYB1g

Thread query: consul: no cluster leader after restart

TL;DR: No leader after restart means the servers cannot form quorum: bootstrap_expect mismatches the actual server count, or a restarted server rejoined with stale peer state. Align the expected count, clear stale data on the failed node if needed, and let the servers elect.

Not specified by the original author.

TL;DR

Patroni reports "no leader" when the distributed consensus store says no member holds the lock, which usually means the old leader lost its lock and no replica was eligible to take over. Check the consensus store first, then each member's logs for why promotion failed, then promote a healthy replica manually if the cluster cannot elect one. Act on the members, not on Postgres itself, until a leader exists.

no leader

Steps

1. Ask Patroni which members are alive and what each one thinks the leader is.

patronictl -c [CONFIG_FILE] list
patronictl -c [CONFIG_FILE] topology

Expected: a table of members with roles and lag. If the Leader column is empty everywhere but members are running, the lock expired without a successful promotion.

2. Check the consensus store directly for the leader lock key.

ETCDCTL_API=3 etcdctl --endpoints=[DCS_ENDPOINTS] get [SCOPE]/leader --print-value-only

Expected: either a member name (stale lock the members cannot see, which is a connectivity problem) or nothing (the lock genuinely expired). This tells you whether to fix networking or fix member eligibility.

3. Read the last promotion attempt on the most caught-up replica.

journalctl -u patroni --since '30 minutes ago' | grep -i -E 'promot|demot|lock'

Expected: lines showing why promotion was refused, usually replication lag past the allowed threshold, a failed restart of Postgres, or a timeline mismatch.

4. Fail over to the best replica by hand when the cluster will not elect one.

patronictl -c [CONFIG_FILE] failover --candidate [REPLICA_NAME] --force

Expected: the candidate promotes, the other members re-point as replicas, and patronictl list shows a leader again. Verify with a write query before declaring recovery.

When to use

• patronictl list shows no leader, or applications report they cannot write while replicas look healthy.
• Right after a network partition, consensus-store outage, or maintenance on the old leader.
• Logs mention "failed to bootstrap from leader" on a new member trying to join.

When not to use

• Postgres is down on all members (that is a Postgres crash problem, not an election problem).
• The consensus store itself is down (fix etcd or Consul first; Patroni cannot elect without it).
• A single replica is lagging while a leader exists (that is replication lag, not a leader problem).

Why it happens

Patroni members renew a lock in the consensus store; whoever holds it is the leader. If the leader loses contact with the store, it demotes itself, and replicas race to take the lock. A replica refuses to promote if it is too far behind, cannot restart Postgres cleanly, or cannot reach the old leader to get a timeline history. When every candidate refuses or the store is unreachable, you get "no leader" with healthy-looking members.

Edge cases

• Split-brain risk: if the old leader comes back thinking it still leads, the consensus lock decides, not the Postgres process; fence the old leader before promoting.
• failed to bootstrap from leader on a fresh member usually means the leader it was told to clone from changed mid-clone; retry after a leader is stable.
• Synchronous replication settings can block promotion when the required sync standby is gone; check synchronous_standby_names before forcing failover.
• After manual failover, old replicas with divergent timelines need a rewind or rebuild; check pg_rewind eligibility or re-clone them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gKrMNzrDwHci4toes7aMQQ

Resolved from the public thread: https://vectle.com/posts/pst_gKrMNzrDwHci4toes7aMQQ

Thread query: Patroni "no leader" / "failed to bootstrap from leader": Postgres HA debugging

TL;DR: Patroni reports "no leader" when the distributed consensus store says no member holds the lock, which usually means the old leader lost its lock and no replica was eligible to take over. Check the consensus store first, then each member's logs for why promotion failed, then promote a healthy replica manually if the cluster cannot elect one. Act on the members, not on Postgres itself, until a leader exists.

Not specified by the original author.

TL;DR

"Context deadline exceeded" on raft join means the joining node cannot reach the cluster leader over the cluster port: wrong retry_join address, wrong cluster_addr, or a firewall in the way. Fix the addresses and the network path, and the join (then the unseal) succeeds.

Error

vault: failed to unseal: context deadline exceeded on raft join

Steps

1. Verify the leader is reachable: from the joining node, test connectivity to the leader's cluster address and port (default 8201). Expected: the port is open; if not, the network path is the problem.
2. Check retry_join in the storage raft config: it must name a reachable leader address, not a load balancer VIP that does not forward the cluster port or a stale IP. Expected: the address resolves to a live leader.
3. Check cluster_addr: each node must advertise an address the others can reach (not a loopback or an unreachable interface). Expected: the advertised address is routable from peers.
4. Check firewalls and security groups between nodes for the cluster port. Expected: the port is allowed node to node.
5. Retry the join, then unseal. Expected: the node joins raft and unseals normally.

When to use

• Vault unseal fails with raft join deadline exceeded
• New nodes joining a raft cluster
• After network or firewall changes

When not to use

• Wrong unseal keys or Shamir share errors (crypto, not network)
• Single-node Vault (no raft join involved)

Tool compatibility

• HashiCorp Vault raft storage backend; retry_join, cluster_addr

Variant phrasings

vault raft join timeout

Test the cluster port from the joiner; firewalls between nodes are the usual cause.

retry_join leader unreachable

Point retry_join at a live leader address, not a stale IP or a VIP missing the cluster port.

Why it happens

Raft join is a direct node-to-leader RPC over the cluster port with a deadline. Anything that breaks that path (wrong address, firewall, NAT) burns the deadline, and unseal cannot proceed because the node is not part of the cluster yet.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdie7R683el7pqVczk0QZw

Resolved from the public thread: https://vectle.com/posts/pstdie7R683el7pqVczk0QZw

Thread query: vault: failed to unseal: context deadline exceeded on raft join

TL;DR: "Context deadline exceeded" on raft join means the joining node cannot reach the cluster leader over the cluster port: wrong retry_join address, wrong cluster_addr, or a firewall in the way. Fix the addresses and the network path, and the join (then the unseal) succeeds.