opensearch 'shard' unassigned after a node restart: how to tell slow recovery from stuck

TL;DR

Ask the cluster why the shard is unassigned. GET cluster/allocation/explain names the exact decider blocking it. If the shard is relocating or initializing, it is slow recovery, so watch cat/recovery for progress. If a decider (disk threshold, max retries) is blocking, it is stuck until you fix that condition.

{
  "status": "yellow",
  "unassigned_shards": 3,
  "initializing_shards": 0,
  "relocating_shards": 0
}

Steps

1. Check overall health: GET cluster/health. Note the unassignedshards count and whether it shrinks over several minutes. Expected: you have the current unassigned count as a baseline.
2. Ask why: GET _cluster/allocation/explain for one unassigned shard. The response names the node and the decider (disk threshold, shard limit, delayed allocation). Expected: a specific reason, not a guess.
3. If it is recovering, watch progress: GET cat/recovery and look at the bytespercent column climbing. Expected: a climbing percent means slow recovery; wait, do not intervene.
4. If a decider blocks it, fix the condition. Disk watermark breached: free disk or raise the watermark. Max retries exceeded: POST cluster/reroute with retryfailed set. Delayed allocation: wait out the delay or shorten it. Expected: after fixing the condition, the shard starts initializing.
5. Re-check health until status returns to green and unassigned_shards hits zero. Expected: green status, zero unassigned shards.

Use this when

• OpenSearch or Elasticsearch stays yellow after a node restart
• Shards show as unassigned and the count is not dropping
• A node came back but its shards did not reassign
• You cannot tell whether to wait or intervene

Not for this skill when

• Status is red with missing primary shards (that is data-loss risk; restore from snapshot, do not just reroute)
• Shards are unassigned on a fresh cluster (check discovery and node count first)
• Searches are slow on a green cluster (different problem)

Variant phrasings

• opensearch unassigned shards after restart
• elasticsearch shards stuck unassigned
• how to tell if shard recovery is stuck
• opensearch yellow status unassigned shards

Why it happens

After a restart the cluster must reassign shards to nodes, copying segment files over the network. That is genuinely slow for large shards. It only becomes stuck when an allocation decider, disk space, shard counts per node, or retry limits, vetoes the assignment; then the shard waits forever instead of progressing.

Edge cases

• The flood-stage watermark marks indexes read-only; clear the block after freeing disk or writes stay rejected.
• Forcing allocation can build an empty shard if the data is gone; use retry_failed, never force-allocate a primary you cannot afford to lose.
• Delayed allocation is intentional after a transient node drop; overriding it causes needless copying.
• One bad disk can pin many shards; check per-node disk, not just cluster totals.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8S4cyI6z5SY5kJ4BE1jXgg