kafka error: consumer lag growing on the group - how to find the slow consumer

kafka error: consumer lag growing on the group - how to find the slow consumer

TL;DR

Lag grows when consumption is slower than production. Find which partitions lag (kafka-consumer-groups --describe), then find which consumer instance owns them and why it is slow: processing time per record, downstream calls, or too few instances for the partition count.

kafka consumer lag growing on the group (LAG column climbing in consumer-group describe)

Steps

1. Describe the group: kafka-consumer-groups --bootstrap-server [host] --describe --group [group]. Expected: The output shows per-partition CURRENT-OFFSET, LOG-END-OFFSET, and LAG.

2. Identify the lagging partitions and which consumer instance owns them (CONSUMER-ID column). Expected: One or two instances own all the lagging partitions.

3. Check that instance's processing: per-record processing time, downstream latency, and error rates. A slow downstream call per record is the usual cause. Expected: You see the bottleneck in the consumer's own metrics.

4. Check the partition-to-instance ratio: if partitions far outnumber instances, add instances up to the partition count (more instances than partitions adds nothing). Expected: Lag drains after scaling.

5. If processing is inherently slow, reduce per-record work: batch downstream calls, add caching, or move heavy work to a follow-up queue. Expected: Per-record time drops and the consumer keeps up.

When to use

• consumer group lag grows over time
• you need to name the slow consumer instance
• lag is concentrated on some partitions

When not to use

• lag spikes then drains (that is a burst, not a slow consumer)
• all partitions lag equally and production spiked (the producer outran a healthy consumer)
• the consumer is not running at all (lag grows because nothing consumes)

Tool compatibility

• Kafka 2.x/3.x
• kafka-consumer-groups.sh

Variant phrasings

• "kafka consumer lag growing find slow consumer"
• "kafka consumer group lag increasing"
• "kafka per partition lag describe"

Why it happens

Lag is the distance between the newest produced offset and the consumer's committed offset. It grows whenever the consumer's sustained throughput is below the produce rate. Per-partition lag pinpoints where, the consumer-id mapping pinpoints who, and the instance's own metrics explain why.

Edge cases

• rebalances pause consumption and spike lag; distinguish rebalance storms from genuine slowness
• a poison-pill record that always fails and retries looks like a slow consumer; check the error rate per partition
• lag on _consumeroffsets itself can make the describe output stale; that is meta-lag, not consumer lag

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgz-bZJJhIlOuO9GOGURFg