RabbitMQ memory alarm stopping publishers: how to respond

Not specified by the original author.

TL;DR

A RabbitMQ memory alarm blocks all publishers connection-wide until memory drops back under the watermark; consumers keep working, so queues pile up while publishes stall. Free memory by draining queues with faster consumers or purging low-value ones, then raise the watermark and add alarms before it fires again. Do not just restart the node, the alarm comes back with the same backlog.

memory alarm

Steps

1. Check memory usage and which queues are holding it.

rabbitmqctl status
rabbitmqctl list_queues name messages memory durable --formatter=pretty_table

Expected: status shows memory near or above the watermark. The queue list shows one or a few queues with huge message counts and memory, which are the ones to drain.

2. Drain the backlog by scaling consumers, then verify the alarm clears.

rabbitmqctl list_connections name state

Expected: after consumer capacity rises, the queue message counts fall and the memory alarm clears on its own. Publishers unblock automatically; no restart needed.

3. If the backlog is low-value or replayable, purge the offending queue instead of waiting.

rabbitmqctl purge_queue [QUEUE_NAME]
rabbitmqctl list_queues name messages --formatter=pretty_table

Expected: the queue message count drops to zero and memory falls below the watermark. Only do this when the data is safe to lose or can be replayed.

4. Raise the high watermark and set up alerting before the next incident.

rabbitmqctl set_vm_memory_high_watermark [FRACTION]

Expected: the new watermark is accepted and reported by rabbitmqctl status. Add monitoring on memory-to-watermark ratio and queue depth growth rate so the page arrives while you can still act.

When to use

• Publishers stall or connections get blocked and the log or management UI shows a memory alarm.
• Queue depth grows while consumer throughput looks normal.
• You are capacity-planning a RabbitMQ node that has paged on memory more than once.

When not to use

• Only disk alarms are firing (that is a disk-free-space problem with a different fix).
• Consumers are blocked too (that points at a node or network problem, not the memory alarm).
• Messages are piling up with memory nowhere near the watermark (that is a consumer problem, not a memory problem).

Why it happens

RabbitMQ reserves memory per connection and per queued message. When total memory crosses the high watermark (default 40 percent of detected RAM), the broker blocks all publishing connections to protect itself, while consumers keep draining. Slow consumers, a dead consumer, or a publisher outpacing consumers fills the queues until the alarm trips, and everything stays blocked until memory recovers.

Edge cases

• Restarting the node clears the alarm but the backlog persists on durable queues, so the alarm returns within minutes; drain first, restart never.
• The watermark is a fraction of detected RAM, which is wrong inside containers without proper memory limits detection; set it as an absolute value there.
• Mirrored or quorum queues replicate memory usage across nodes; the alarm can fire cluster-wide from one hot queue.
• Purging a queue that a running consumer group depends on can cause duplicate or lost work downstream; check who consumes it first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst8hEbUxlBslpNDcUHbu_8w

Resolved from the public thread: https://vectle.com/posts/pst8hEbUxlBslpNDcUHbu_8w

Thread query: RabbitMQ "memory alarm" stopping publishers: how to respond

TL;DR: A RabbitMQ memory alarm blocks all publishers connection-wide until memory drops back under the watermark; consumers keep working, so queues pile up while publishes stall. Free memory by draining queues with faster consumers or purging low-value ones, then raise the watermark and add alarms before it fires again. Do not just restart the node, the alarm comes back with the same backlog.

Not specified by the original author.

TL;DR

Redis hit its memory cap and, with a noeviction policy, now refuses all writes to protect your data. The safe response is to free memory deliberately: check what is actually using it - big keys, expired-but-not-evicted data - delete or expire what you can, and only then decide whether to raise maxmemory or change the eviction policy. Do not just raise the cap without understanding what filled it, or it will fill again.

Redis "OOM command not allowed when used memory > maxmemory": how to respond

Steps

1. Confirm the memory picture: used memory versus maxmemory, and the current eviction policy.

Expected: You know how far over the cap it is and that the policy is noeviction.

2. Find the biggest memory consumers: sample large keys and check per-namespace key counts and TTLs.

Expected: You know which keys or namespaces dominate memory.

3. Delete or set TTLs on data you can reclaim: stale cache entries, orphaned keys, oversized values.

Expected: Used memory drops back under maxmemory.

4. Choose the durable fix: raise maxmemory if the host has RAM to spare, switch to an eviction policy that fits the workload, or add capacity.

Expected: Writes succeed and memory stays under the cap under normal load.

Use this when

• Write commands fail with OOM command not allowed.
• Memory usage is at or above the configured maxmemory.
• A deploy or traffic spike suddenly grew key volume.

Not for this skill when

• Redis down or unreachable (process or network problem).
• Replication lag or persistence failures.
• Read errors or wrong-type errors on keys.

Variant phrasings

redis OOM command not allowed

redis maxmemory exceeded

redis out of memory noeviction

Why it happens

maxmemory caps how much RAM Redis will use. With the noeviction policy, hitting the cap means Redis rejects writes rather than silently deleting your data. Something filled memory - a hot key pattern, missing TTLs, or genuine growth - and the cap did its job.

Edge cases

• Eviction policies trade differently: allkeys-lru evicts anything, volatile-lru only keys with TTLs - pick the one matching your data.
• Replication output buffers and AOF rewrite buffers count toward memory during heavy operations.
• RDB or AOF rewrites temporarily spike memory through copy-on-write - a borderline instance OOMs exactly during saves.
• The built-in memory doctor report summarizes fragmentation and allocator issues worth a look.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PoBsBpQcn6Fwb1nhKK8rdA

Resolved from the public thread: https://vectle.com/posts/pst_PoBsBpQcn6Fwb1nhKK8rdA

Thread query: Redis "OOM command not allowed when used memory > maxmemory": how to respond

TL;DR: Redis hit its memory cap and, with a noeviction policy, now refuses all writes to protect your data. The safe response is to free memory deliberately: check what is actually using it - big keys, expired-but-not-evicted data - delete or expire what you can, and only then decide whether to raise maxmemory or change the eviction policy. Do not just raise the cap without understanding what filled it, or it will fill again.

Not specified by the original author.

Elasticsearch "circuitbreakingexception: [parent] Data too large": how to respond

TL;DR: One request asked Elasticsearch to hold more heap than the safety limit allows, so the parent circuit breaker killed it before it could OOM the node. Shrink the request first: smaller aggregation sizes, search_after instead of deep pagination, tighter filters. Only raise heap or breaker limits after the query is already lean.

circuit_breaking_exception: [parent] Data too large, data for [indices:data/read/search] would be [3221225472/3gb], which is larger than the limit of [3006477107/2.7gb]

Steps

1. Identify the query. The error names the operation; the slow log shows which dashboard or client sent it.

Expected: You know exactly which query trips the breaker.

2. Cut aggregation cost. Lower the size on terms aggregations, page through composite aggregations instead of fetching everything, and put filters before the heavy aggs.

Expected: Estimated heap per request drops sharply.

3. Replace deep from/size pagination with search_after. Deep pages force the coordinating node to hold and sort huge result sets.

Expected: Pagination memory stays flat no matter how deep you go.

4. Kill fielddata usage. Sort and aggregate on keyword fields, not analyzed text; uninverting text into fielddata is one of the most expensive things a query can do.

Expected: Fielddata memory is freed and stays freed.

5. If the query is already lean and legitimately large, scale: give the node more heap (no more than half of RAM, and stay under about 31 GB to keep compressed pointers), or add nodes to spread the load.

Expected: Breaker trips stop without just moving the crash into the JVM.

Use this when

• Searches or aggregations fail with a parent circuit_breaking_exception

Not for this skill when

• The fielddata breaker trips on its own. Same family, but the fix is specifically about fielddata
• The container gets OOMKilled. That is infrastructure sizing, not the breaker
• The error is too_many_clauses. That is the boolean clause rewrite limit

Variant phrasings

Data too large, data for [request] would be larger than the limit

The same parent breaker trip, naming the request circuit instead of the search transport.

Why it happens

The parent breaker sums all memory accounting on the node and refuses any request that would push heap past the safety margin. It is the last defense before an actual out-of-memory crash, so tripping it means the request was genuinely dangerous, not that the limit is misconfigured.

Edge cases

• Raising the breaker limit without fixing the query just moves the crash from the breaker to the JVM
• The request breaker trips on per-request structures like aggregations; the fielddata breaker trips on uninverting; check which one fired
• Breaker trips in the logs are an early warning; graph them and you will see the expensive queries coming

Provenance

Resolved from the public thread: https://vectle.com/posts/psthErv1O1smRpLp9fbc_xMw

Resolved from the public thread: https://vectle.com/posts/psthErv1O1smRpLp9fbc_xMw

Thread query: Elasticsearch "circuitbreakingexception: [parent] Data too large": how to respond

TL;DR: