opensearch: cluster_block_exception index read-only after disk watermark

TL;DR

OpenSearch flips indices to read-only when disk hits the flood-stage watermark, to protect the cluster. Free disk space (delete old indices or add capacity), then clear the index.blocks.read_only_allow_delete setting. Writes resume once the block is lifted.

Error

opensearch: cluster_block_exception index read-only after disk watermark

Steps

1. Confirm disk pressure: check disk usage on the data nodes. Expected: usage at or above the flood-stage watermark.
2. Free space: delete old indices, force-merge, or add disk capacity. Expected: usage drops below the watermark with margin.
3. Clear the block: update the index settings to set index.blocks.read_only_allow_delete to false (all indices, or the affected ones). Expected: the setting clears.
4. Verify writes: index a test document. Expected: writes succeed without the block exception.
5. Prevent recurrence: set up index lifecycle management and disk alerting well below the flood stage. Expected: the watermark never triggers unnoticed again.

When to use

• Writes fail with cluster_block_exception and read-only indices
• After disk filled on data nodes
• Setting up retention to prevent recurrence

When not to use

• Unassigned shards (allocation issue)
• The block was set manually (just clear the setting)

Tool compatibility

• OpenSearch index settings (index.blocks.read_only_allow_delete); disk watermarks, ILM

Variant phrasings

opensearch disk watermark read only

Free disk first; clearing the block without headroom just re-triggers it.

flood stage watermark opensearch

Delete old data or add capacity, then lift the block.

Why it happens

The flood-stage watermark is a safety brake: rather than letting disks fill completely (which corrupts the cluster), OpenSearch makes indices read-only. The block persists even after disk frees up, so it must be cleared explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iIRFPkyx87oXzFK9O1MtAw