agent misread the kafka offset lag metric, reported a healthy consumer

TL;DR

Lag has traps that all read as fine: an average across partitions hides one partition stuck at a huge lag, the committed offset can be recent while the consumer is processing slowly far behind the log end, and a lag of zero messages can still mean old data if the producer went quiet. Report lag properly: the maximum per partition lag in messages, the time lag of the oldest unconsumed record, and the trend over the window, checked against the consumer group's actual committed offsets and the log end offsets.

agent misread the kafka offset lag metric, reported a healthy consumer

Use this when

• A health report said a consumer was fine while downstream data was hours stale.
• The lag figure quoted is a single average or a single partition's value.
• The consumer group shows offsets committed recently but the newest data downstream is old.

Not for this skill when

• The consumer really is caught up and the staleness is a producer outage. Lag cannot see a silent producer; freshness checks on the data can.
• The metric came from the broker and is wrong at the source. That is a monitoring platform bug to file separately.

Steps

1. Recompute lag from the raw numbers yourself: for the consumer group, list per partition the log end offset and the committed offset, and subtract. Note the maximum partition lag and the total. Verify: the real lag is computed from offsets, not quoted from a dashboard tile
2. Convert the worst partition lag into time: find the timestamp of the oldest unconsumed record on that partition and compare it with now. Note the time lag. Verify: staleness is expressed in time, the unit the business feels
3. Reconstruct what the agent read: the metric name, whether it was an average or a sum, and the time window. Note the exact misreading that produced healthy. Verify: the reporting flaw is named, so it can be fixed rather than regretted
4. Fix the health check to require all three: maximum partition lag under a threshold, time lag under a threshold, and the trend not rising over the check window. Note the thresholds chosen from the consumer's service level. Verify: healthy now means caught up, current, and not falling behind
5. Replay the fixed check against the incident window's recorded offsets and confirm it reports unhealthy, then let it run live for a cycle. Note both results. Verify: the check is proven against the case that fooled it

Variant phrasings

kafka consumer lag average hides stuck partition max lag

The aggregation trap phrasing.

kafka lag messages vs time lag consumer health check

The two units version.

Compatibility: Kafka consumer groups on any distribution, with lag computed from committed and log end offsets via admin tools or monitoring exporters. Applies equally to Connect sink consumers.

Why it happens

Lag feels like one number and is really a distribution over partitions, each with its own pace, plus a time dimension the offset numbers do not carry. Tooling that summarises it into a single friendly average does the misreading for you, and an agent quoting that summary inherits a conclusion the raw data does not support. The stuck partition is the classic case: it is usually stuck for a reason, a poison record or a slow key, and it is precisely the partition whose data the business is missing while the average smiles.

Edge cases / pitfalls

• A consumer that is stopped commits nothing, so some dashboards show its lag as unknown or zero rather than huge. Unknown must count as unhealthy.
• Rebalances freeze progress briefly and spike lag readings. Judge the trend over a window, not a single sample, in both directions: do not excuse a real stall as a rebalance either.
• Time lag needs record timestamps the producer actually set. Log append time versus create time changes the meaning; know which your topics use.
• Report the partition identity with the lag. A health report that names the stuck partition starts the fix in the right place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xik3877phCwYbAvv-b0JzQ