opensearch: unassigned shards after node restart
Fixes OpenSearch unassigned shards after a node restart using the allocation explain API, waiting out delayed allocation, and rerouting when safe. Not for disk-watermark blocks.
TL;DR
After a restart, shards stay unassigned while the cluster waits (delayed allocation) or because allocation is genuinely blocked. Check _cluster/allocation/explain for the reason: wait it out, fix the blocker, or reroute. Shards assign once the decider is satisfied.
Error
opensearch: unassigned shards after node restartSteps
- Check cluster health and the pending tasks: confirm the restart finished and the cluster is otherwise green or yellow. Expected: the scope of the problem is just the unassigned shards.
- Ask the decider: call the allocation explain API for an unassigned shard. Expected: the exact reason (delayed allocation timer, disk decider, awareness, no valid node).
- If it is delayed allocation, wait: the timer exists to let the node rejoin without shuffling data. Expected: shards reassign when the timer expires.
- If a decider blocks (disk, awareness, allocation disabled), fix the underlying condition. Expected: the decider permits allocation.
- As a last resort, use the reroute API to manually assign, understanding the data-safety tradeoff. Expected: shards assigned, cluster green.
When to use
- Yellow or red cluster with unassigned shards after restarts
- Planned node maintenance
- Understanding allocation decisions
When not to use
- Read-only blocks from disk watermarks (different fix)
- Brand-new indices with no replicas configured (expected state)
Tool compatibility
- OpenSearch
_cluster/allocation/explain, reroute API; allocation deciders
Variant phrasings
shards unassigned delayed allocation
Wait for the timer; it avoids pointless data movement when the node is coming back.
allocation explain no valid node
Fix the decider's complaint (disk, zone awareness) rather than forcing assignment.
Why it happens
OpenSearch is cautious about moving shard copies: delayed allocation avoids re-replicating data for a node that is briefly gone, and deciders veto placements that would violate disk, awareness, or filtering rules. Unassigned is the safe default until a placement is genuinely valid.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstoHWnp1x7XlMDDezZdfwZA