rabbitmq error: queue message count growing - how to find the bottleneck consumer

rabbitmq error: queue message count growing - how to find the bottleneck consumer

TL;DR

A growing queue means publishes outpace acks. Check consumer utilization in the management UI: a consumer at 100% utilization that still cannot keep up needs more prefetch or more consumers; a consumer at low utilization with a growing queue is blocked downstream.

rabbitmq: queue message count growing (ready messages climbing in management UI)

Steps

1. Open the queue in the management UI and read the consumer utilisation field. Expected: You see a percentage: near 100% means the consumer is the bottleneck, low means it is blocked.

2. If utilization is high, raise prefetch (basic.qos) so the consumer holds more unacked messages, or add consumer instances. Expected: Ack rate rises and the queue drains.

3. If utilization is low, the consumer is stuck downstream: check its logs for slow dependency calls or errors. Expected: You find the downstream call eating the time.

4. Check for poison messages: a message that always fails and requeues pins one consumer while the queue grows. Look at redelivered counts. Expected: The poison message is identified and dead-lettered.

5. Set queue length limits or TTLs as a backstop so a future stall pages instead of growing forever. Expected: The queue has a ceiling and alerts fire before it matters.

When to use

• a RabbitMQ queue's message count grows steadily
• you need to tell a slow consumer from a blocked one
• consumer utilisation is visible in the management UI

When not to use

• the queue grows in bursts then drains (normal burstiness)
• there are no consumers at all (bind one first)
• the publisher is the problem (a runaway publisher fills any queue)

Tool compatibility

• RabbitMQ 3.x
• management plugin

Variant phrasings

• "rabbitmq queue message count growing"
• "rabbitmq find bottleneck consumer"
• "rabbitmq consumer utilisation low queue growing"

Why it happens

Queue depth is the integral of (publish rate minus ack rate). Consumers report utilization: the fraction of time they could take a message and did. High utilization with growth means the consumer is simply too slow; low utilization with growth means it is waiting on something else and the queue is the symptom.

Edge cases

• prefetch too high on a slow consumer just moves the backlog into unacked; tune it against real processing time
• a single poison message with infinite requeue looks identical to a slow consumer until you check redelivered
• mirrored/classic queue sync can stall consumers during node events; check cluster status too

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_woLwxVJZQyqqBJw142qbMA