ElastiCache/Redis "CLUSTERDOWN": cluster state debugging

TL;DR

CLUSTERDOWN means the Redis cluster cannot serve the hash slots your command needs: slots are uncovered because nodes failed, or the cluster is mid-failover and has not reassigned them. Find which slots are uncovered with cluster slots, then fail over or restart the missing primaries so every slot has an owner. Reads and writes stay broken for exactly the slots with no owner until you fix that.

CLUSTERDOWN

Steps

1. Check the cluster state from several nodes and list slot ownership.

redis-cli -h [NODE_HOST] -p [PORT] cluster info
redis-cli -h [NODE_HOST] -p [PORT] cluster slots

Expected: cluster_state:ok when healthy. In the failing state you see cluster_state:fail and slot ranges with no owning node, or nodes flagged fail in the output.

2. Identify which nodes are down and whether replicas are ready to take over.

redis-cli -h [NODE_HOST] -p [PORT] cluster nodes

Expected: one line per node with flags. Look for fail on primaries and matching replicas flagged slave with a healthy link state; those replicas are your failover candidates.

3. Force a replica to take over for a failed primary.

redis-cli -h [REPLICA_HOST] -p [PORT] cluster failover TAKEOVER

Expected: the replica becomes master for its slots. Re-run cluster info and confirm cluster_state:ok and full slot coverage.

4. If a failed primary comes back with stale data, let the cluster settle before trusting it.

redis-cli -h [NODE_HOST] -p [PORT] cluster info

Expected: state stays ok for several checks in a row. Watch for flapping between ok and fail, which means the node is still unhealthy and should stay out of rotation.

When to use

• Clients get CLUSTERDOWN on reads or writes that worked before, after node failures or restarts.
• Right after a deploy, AZ outage, or a rolling restart of cluster nodes.
• Slot coverage looks wrong in cluster slots even though most nodes are up.

When not to use

• Single-node Redis or ElastiCache non-cluster mode (there are no slots there; this error cannot occur).
• The error is MOVED or ASK (those are normal redirections, not a cluster outage).
• All nodes are up and slots are covered (then the client is misconfigured or pointing at the wrong port).

Why it happens

A Redis cluster splits the keyspace into 16384 hash slots, and each slot must be owned by a reachable primary. When a primary dies and its replica cannot promote (replica down, replication lag too high, or manual failover blocked), those slots have no owner and the cluster refuses commands touching them. ElastiCache surfaces the same state; the fix is identical because the underlying cluster protocol is the same.

Edge cases

• If more than half the primaries are down with no replicas, the cluster cannot reach quorum for failover; you restore from backup instead of forcing promotion.
• TAKEOVER can lose writes that never replicated; check how much data loss is acceptable before using it over a normal FAILOVER.
• A node that was down briefly may rejoin as a replica of its own former replica; that is normal, do not try to "fix" the role swap.
• Clients that cache the slot map aggressively keep erroring after recovery; restart or reconnect them to pick up the new map.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstM0SybABpTZaDypqPoUYWw

Resolved from the public thread: https://vectle.com/posts/pstM0SybABpTZaDypqPoUYWw

Thread query: ElastiCache/Redis "CLUSTERDOWN": cluster state debugging

TL;DR: CLUSTERDOWN means the Redis cluster cannot serve the hash slots your command needs: slots are uncovered because nodes failed, or the cluster is mid-failover and has not reassigned them. Find which slots are uncovered with cluster slots, then fail over or restart the missing primaries so every slot has an owner. Reads and writes stay broken for exactly the slots with no owner until you fix that.