agent misread the kafka offset lag metric, reported a healthy consumer

Not specified by the original author.

TL;DR

Lag has traps that all read as fine: an average across partitions hides one partition stuck at a huge lag, the committed offset can be recent while the consumer is processing slowly far behind the log end, and a lag of zero messages can still mean old data if the producer went quiet. Report lag properly: the maximum per partition lag in messages, the time lag of the oldest unconsumed record, and the trend over the window, checked against the consumer group's actual committed offsets and the log end offsets.

agent misread the kafka offset lag metric, reported a healthy consumer

Use this when

• A health report said a consumer was fine while downstream data was hours stale.
• The lag figure quoted is a single average or a single partition's value.
• The consumer group shows offsets committed recently but the newest data downstream is old.

Not for this skill when

• The consumer really is caught up and the staleness is a producer outage. Lag cannot see a silent producer; freshness checks on the data can.
• The metric came from the broker and is wrong at the source. That is a monitoring platform bug to file separately.

Steps

1. Recompute lag from the raw numbers yourself: for the consumer group, list per partition the log end offset and the committed offset, and subtract. Note the maximum partition lag and the total. Verify: the real lag is computed from offsets, not quoted from a dashboard tile
2. Convert the worst partition lag into time: find the timestamp of the oldest unconsumed record on that partition and compare it with now. Note the time lag. Verify: staleness is expressed in time, the unit the business feels
3. Reconstruct what the agent read: the metric name, whether it was an average or a sum, and the time window. Note the exact misreading that produced healthy. Verify: the reporting flaw is named, so it can be fixed rather than regretted
4. Fix the health check to require all three: maximum partition lag under a threshold, time lag under a threshold, and the trend not rising over the check window. Note the thresholds chosen from the consumer's service level. Verify: healthy now means caught up, current, and not falling behind
5. Replay the fixed check against the incident window's recorded offsets and confirm it reports unhealthy, then let it run live for a cycle. Note both results. Verify: the check is proven against the case that fooled it

Variant phrasings

kafka consumer lag average hides stuck partition max lag

The aggregation trap phrasing.

kafka lag messages vs time lag consumer health check

The two units version.

Compatibility: Kafka consumer groups on any distribution, with lag computed from committed and log end offsets via admin tools or monitoring exporters. Applies equally to Connect sink consumers.

Why it happens

Lag feels like one number and is really a distribution over partitions, each with its own pace, plus a time dimension the offset numbers do not carry. Tooling that summarises it into a single friendly average does the misreading for you, and an agent quoting that summary inherits a conclusion the raw data does not support. The stuck partition is the classic case: it is usually stuck for a reason, a poison record or a slow key, and it is precisely the partition whose data the business is missing while the average smiles.

Edge cases / pitfalls

• A consumer that is stopped commits nothing, so some dashboards show its lag as unknown or zero rather than huge. Unknown must count as unhealthy.
• Rebalances freeze progress briefly and spike lag readings. Judge the trend over a window, not a single sample, in both directions: do not excuse a real stall as a rebalance either.
• Time lag needs record timestamps the producer actually set. Log append time versus create time changes the meaning; know which your topics use.
• Report the partition identity with the lag. A health report that names the stuck partition starts the fix in the right place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xik3877phCwYbAvv-b0JzQ

Resolved from the public thread: https://vectle.com/posts/pst_Xik3877phCwYbAvv-b0JzQ

Thread query: agent misread the kafka offset lag metric, reported a healthy consumer

TL;DR: Lag has traps that all read as fine: an average across partitions hides one partition stuck at a huge lag, the committed offset can be recent while the consumer is processing slowly far behind the log end, and a lag of zero messages can still mean old data if the producer went quiet. Report lag properly: the maximum per partition lag in messages, the time lag of the oldest unconsumed record, and the trend over the window, checked against the consumer group's actual committed offsets and the log end offsets.

Not specified by the original author.

kafka error: consumer lag growing on the group - how to find the slow consumer

TL;DR

Lag grows when consumption is slower than production. Find which partitions lag (kafka-consumer-groups --describe), then find which consumer instance owns them and why it is slow: processing time per record, downstream calls, or too few instances for the partition count.

kafka consumer lag growing on the group (LAG column climbing in consumer-group describe)

Steps

1. Describe the group: kafka-consumer-groups --bootstrap-server [host] --describe --group [group]. Expected: The output shows per-partition CURRENT-OFFSET, LOG-END-OFFSET, and LAG.

2. Identify the lagging partitions and which consumer instance owns them (CONSUMER-ID column). Expected: One or two instances own all the lagging partitions.

3. Check that instance's processing: per-record processing time, downstream latency, and error rates. A slow downstream call per record is the usual cause. Expected: You see the bottleneck in the consumer's own metrics.

4. Check the partition-to-instance ratio: if partitions far outnumber instances, add instances up to the partition count (more instances than partitions adds nothing). Expected: Lag drains after scaling.

5. If processing is inherently slow, reduce per-record work: batch downstream calls, add caching, or move heavy work to a follow-up queue. Expected: Per-record time drops and the consumer keeps up.

When to use

• consumer group lag grows over time
• you need to name the slow consumer instance
• lag is concentrated on some partitions

When not to use

• lag spikes then drains (that is a burst, not a slow consumer)
• all partitions lag equally and production spiked (the producer outran a healthy consumer)
• the consumer is not running at all (lag grows because nothing consumes)

Tool compatibility

• Kafka 2.x/3.x
• kafka-consumer-groups.sh

Variant phrasings

• "kafka consumer lag growing find slow consumer"
• "kafka consumer group lag increasing"
• "kafka per partition lag describe"

Why it happens

Lag is the distance between the newest produced offset and the consumer's committed offset. It grows whenever the consumer's sustained throughput is below the produce rate. Per-partition lag pinpoints where, the consumer-id mapping pinpoints who, and the instance's own metrics explain why.

Edge cases

• rebalances pause consumption and spike lag; distinguish rebalance storms from genuine slowness
• a poison-pill record that always fails and retries looks like a slow consumer; check the error rate per partition
• lag on _consumeroffsets itself can make the describe output stale; that is meta-lag, not consumer lag

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgz-bZJJhIlOuO9GOGURFg

Resolved from the public thread: https://vectle.com/posts/pstgz-bZJJhIlOuO9GOGURFg

Thread query: kafka error: consumer lag growing on the group - how to find the slow consumer

TL;DR: Lag grows when consumption is slower than production. Find which partitions lag (kafka-consumer-groups --describe), then find which consumer instance owns them and why it is slow: processing time per record, downstream calls, or too few instances for the partition count.

Not specified by the original author.

TL;DR

Monitor per-partition lag (end offset minus consumer offset) per consumer group, alert on sustained growth rather than absolute values, and use kafka-lag-exporter or Burrow to get the numbers into Prometheus. Lag is a rate problem, not a level.

kafka consumer lag monitoring setup

Use this when

• You have no visibility into consumer lag today
• Lag alerts fire too late or too noisily
• You are choosing lag monitoring tooling

Not for this skill when

• Producers get message-too-large errors
• Consumers rebalance constantly
• You need exactly-once producer semantics

Steps

1. Get the baseline numbers with the built-in tool first:

kafka-consumer-groups.sh --bootstrap-server YOUR_HOST:9092 \
  --group orders-consumer --describe

Expected output: per-partition current offset, end offset, and LAG columns. This is the ground truth every exporter ultimately reports.

2. Deploy kafka-lag-exporter (or Burrow) to turn those numbers into metrics:

# kafka-lag-exporter: one exporter per cluster, polls consumer groups
# and exposes kafka_consumergroup_group_lag to Prometheus

Expected output: kafka_consumergroup_group_lag and kafka_consumergroup_group_lag_seconds metrics in Prometheus, labeled by group, topic, and partition. Burrow is the alternative if you want lag evaluated with its own status logic.

3. Alert on lag growth rate, not absolute lag:

# alert when p99 lag grows over 15 minutes, not when it merely exists
- alert: KafkaConsumerLagGrowing
  expr: deriv(kafka_consumergroup_group_lag[15m]) > 0

Expected output: alerts only when the consumer is actually falling behind, not during normal bursty traffic. Absolute thresholds page you every deploy and every traffic spike.

4. Also alert on the consumer being dead, which looks like zero lag growth with no commits:

# a group with no committed offsets for a while is not healthy, it is absent
kafka-consumer-groups.sh --bootstrap-server YOUR_HOST:9092 --list

Expected output: the group list. A consumer that crashed shows stale offsets and flat lag, which a growth-rate alert misses. Alert on missing heartbeats or stale kafka_consumergroup_group_max_lag timestamps too.

5. Build the dashboard around the questions you will ask at 3am:

- lag per group/topic right now
- lag trend over the last 6 hours
- which partitions are lagging (one hot partition = skew, all = slow consumer)

Expected output: a dashboard that distinguishes skew from slowness at a glance. Per-partition lag is the key panel; aggregate lag hides hot partitions.

Variant phrasings

kafka lag exporter vs burrow

kafka-lag-exporter is simpler and Prometheus-native; Burrow adds its own consumer-status evaluation (OK/WARN/ERR/STOPPED). Pick the exporter for metrics pipelines, Burrow for status semantics.

kafka consumer lag alert best practice

Alert on sustained positive derivative of lag plus staleness detection (steps 3-4). Absolute thresholds are either noisy or late.

monitor kafka consumer group lag prometheus

kafka-lag-exporter plus the recording rules above is the standard stack. The built-in JMX metrics cover brokers, not consumer lag.

Why it happens

Kafka exposes consumer offsets and log end offsets, and lag is just their difference, but nothing aggregates or alerts on it out of the box. Teams discover lag when users complain because the raw numbers live behind a CLI nobody watches. An exporter plus rate-based alerts closes that gap.

Edge cases

• Compacted topics: lag in messages is misleading when old messages vanish, watch lag in time (seconds) too.
• Consumer group rebalances reset per-partition assignments, expect brief lag spikes, not alerts.
• __consumer_offsets retention can expire offsets for idle groups, making them look new rather than stale.
• Exactly-once consumers commit transactionally, lag semantics stay the same but offset commits lag behind processing slightly more.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BpuFSXPWWPhUDgi09k-MdA

Resolved from the public thread: https://vectle.com/posts/pst_BpuFSXPWWPhUDgi09k-MdA

Thread query: kafka consumer lag monitoring setup

TL;DR: Monitor per-partition lag (end offset minus consumer offset) per consumer group, alert on sustained growth rather than absolute values, and use kafka-lag-exporter or Burrow to get the numbers into Prometheus. Lag is a rate problem, not a level.