Elastic "index read-only / allow delete (api)": disk watermark fix

Elastic "index read-only / allow delete (api)": disk watermark fix

TL;DR: Elasticsearch hit its flood-stage disk watermark (95% full by default) and locked indices to read-only to protect itself. Free disk space first (old indices, snapshots, stray logs on the volume), then clear the block with one settings update. Clearing the block without freeing space just re-triggers it minutes later.

{"type":"cluster_block_exception","reason":"index [logs-2026.10.01] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]"}

Steps

1. Confirm the cause. Check disk allocation per node; look for a node at or above the flood-stage watermark.

Expected: You see which node is full and how far over it is.

2. Free space. Delete indices past retention, clean up old snapshots, and check for non-Elasticsearch disk hogs sharing the volume (container logs are the classic surprise).

Expected: Disk usage drops back below the high watermark.

3. Clear the block explicitly. The block persists even after space is freed until you lift it:

PUT /_all/_settings
{"index.blocks.read_only_allow_delete": null}

Expected: The update is acknowledged and writes resume.

4. Verify writes. Index a test document or watch the indexing rate recover.

Expected: No more cluster_block_exception on writes.

5. Prevent the encore. Set up index lifecycle retention, alert on disk at 80%, and only raise the watermark levels if you genuinely have the headroom.

Expected: This does not happen again next month.

Use this when

• Writes fail with index read-only / allow delete (api) and a cluster_block_exception

Not for this skill when

• The block is plain read_only without allow_delete. That was set manually; unset it directly
• The error is about a closed index. Open it instead
• Writes are blocked after license expiry. That is licensing, not disk

Variant phrasings

FORBIDDEN/12/index read-only / allow delete (api)

The short form of the block reason, seen in client error logs.

Why it happens

The flood-stage watermark is a self-defense tripwire. When a node runs out of disk, Elasticsearch stops accepting writes to avoid corruption, and it keeps the block in place until an operator explicitly clears it, even if space has since been freed.

Edge cases

• The block is per index; /_all/_settings clears it everywhere at once
• A node below the watermark can still hold blocked indices from an earlier incident; they need the manual clear too
• Never raise the flood stage near 100% on small disks; the node needs working room to merge and relocate shards

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgf5qE6ZHI1hHms9isW8aw