RabbitMQ "memory alarm" stopping publishers: how to respond

TL;DR

A RabbitMQ memory alarm blocks all publishers connection-wide until memory drops back under the watermark; consumers keep working, so queues pile up while publishes stall. Free memory by draining queues with faster consumers or purging low-value ones, then raise the watermark and add alarms before it fires again. Do not just restart the node, the alarm comes back with the same backlog.

memory alarm

Steps

1. Check memory usage and which queues are holding it.

rabbitmqctl status
rabbitmqctl list_queues name messages memory durable --formatter=pretty_table

Expected: status shows memory near or above the watermark. The queue list shows one or a few queues with huge message counts and memory, which are the ones to drain.

2. Drain the backlog by scaling consumers, then verify the alarm clears.

rabbitmqctl list_connections name state

Expected: after consumer capacity rises, the queue message counts fall and the memory alarm clears on its own. Publishers unblock automatically; no restart needed.

3. If the backlog is low-value or replayable, purge the offending queue instead of waiting.

rabbitmqctl purge_queue [QUEUE_NAME]
rabbitmqctl list_queues name messages --formatter=pretty_table

Expected: the queue message count drops to zero and memory falls below the watermark. Only do this when the data is safe to lose or can be replayed.

4. Raise the high watermark and set up alerting before the next incident.

rabbitmqctl set_vm_memory_high_watermark [FRACTION]

Expected: the new watermark is accepted and reported by rabbitmqctl status. Add monitoring on memory-to-watermark ratio and queue depth growth rate so the page arrives while you can still act.

When to use

• Publishers stall or connections get blocked and the log or management UI shows a memory alarm.
• Queue depth grows while consumer throughput looks normal.
• You are capacity-planning a RabbitMQ node that has paged on memory more than once.

When not to use

• Only disk alarms are firing (that is a disk-free-space problem with a different fix).
• Consumers are blocked too (that points at a node or network problem, not the memory alarm).
• Messages are piling up with memory nowhere near the watermark (that is a consumer problem, not a memory problem).

Why it happens

RabbitMQ reserves memory per connection and per queued message. When total memory crosses the high watermark (default 40 percent of detected RAM), the broker blocks all publishing connections to protect itself, while consumers keep draining. Slow consumers, a dead consumer, or a publisher outpacing consumers fills the queues until the alarm trips, and everything stays blocked until memory recovers.

Edge cases

• Restarting the node clears the alarm but the backlog persists on durable queues, so the alarm returns within minutes; drain first, restart never.
• The watermark is a fraction of detected RAM, which is wrong inside containers without proper memory limits detection; set it as an absolute value there.
• Mirrored or quorum queues replicate memory usage across nodes; the alarm can fire cluster-wide from one hot queue.
• Purging a queue that a running consumer group depends on can cause duplicate or lost work downstream; check who consumes it first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst8hEbUxlBslpNDcUHbu_8w

Resolved from the public thread: https://vectle.com/posts/pst8hEbUxlBslpNDcUHbu_8w

Thread query: RabbitMQ "memory alarm" stopping publishers: how to respond

TL;DR: A RabbitMQ memory alarm blocks all publishers connection-wide until memory drops back under the watermark; consumers keep working, so queues pile up while publishes stall. Free memory by draining queues with faster consumers or purging low-value ones, then raise the watermark and add alarms before it fires again. Do not just restart the node, the alarm comes back with the same backlog.