agent stopped idle kafka brokers — consumer lag exploded overnight

Not specified by the original author.

TL;DR

The agent saw low CPU on Kafka brokers and stopped them, but brokers are not compute-bound: they hold partitions, and removing them forces rebalancing and starves consumers, so lag exploded. Before touching brokers, check partition assignments, under-replicated partitions, and consumer lag; only stop brokers with no assigned partitions and healthy cluster state. Broker count is a durability decision, not a CPU decision.

Error

agent stopped "idle" kafka brokers  -  consumer lag exploded overnight

Steps

1. Confirm the cause: consumer lag spiked after the broker stop, and partition rebalancing followed. Expected: the lag timeline matches the stop event.
2. Restart the stopped brokers (or add capacity) to restore the partition layout. Expected: lag drains back to baseline.
3. Add pre-stop checks to the agent: no assigned partitions, no under-replicated partitions, consumer lag healthy. Expected: the checks gate any broker stop.
4. Require human approval for broker topology changes. Expected: no automated broker stops without review.
5. Re-run the idle analysis with the gate. Expected: only truly unassigned brokers are flagged.

When to use

• Kafka brokers are flagged idle on CPU metrics.
• Consumer lag spikes after broker changes.
• Scoping automated Kafka topology changes.

When not to use

• Brokers have no partitions and the cluster is healthy (safe).
• The lag has another cause.
• The cluster is being decommissioned intentionally.

Tool compatibility

Apache Kafka, MSK (current 2026 releases). Applies to cost agents managing Kafka.

Variant phrasings

kafka "idle" on cpu but critical for partitions

Brokers are judged by partition health, not CPU.

consumer lag after broker rightsizing

Restore the brokers and gate on partition assignment.

broker shutdown caused a rebalance storm

Check under-replicated partitions before any topology change.

Why it happens

Kafka brokers look idle by CPU because their work is I/O and coordination, not computation. CPU-based idle detection is blind to the broker's real job: hosting partitions and serving consumers. Removing brokers triggers leader elections and rebalancing, which is exactly the expensive, lag-inducing event the "optimization" was supposed to avoid. Topology changes need partition-aware gating and human approval.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pdDDeS5X1NoAGXXy-NH7dA

Resolved from the public thread: https://vectle.com/posts/pst_pdDDeS5X1NoAGXXy-NH7dA

Thread query: agent stopped "idle" kafka brokers - consumer lag exploded overnight

TL;DR: The agent saw low CPU on Kafka brokers and stopped them, but brokers are not compute-bound: they hold partitions, and removing them forces rebalancing and starves consumers, so lag exploded. Before touching brokers, check partition assignments, under-replicated partitions, and consumer lag; only stop brokers with no assigned partitions and healthy cluster state. Broker count is a durability decision, not a CPU decision.

Not specified by the original author.

TL;DR

Lag has traps that all read as fine: an average across partitions hides one partition stuck at a huge lag, the committed offset can be recent while the consumer is processing slowly far behind the log end, and a lag of zero messages can still mean old data if the producer went quiet. Report lag properly: the maximum per partition lag in messages, the time lag of the oldest unconsumed record, and the trend over the window, checked against the consumer group's actual committed offsets and the log end offsets.

agent misread the kafka offset lag metric, reported a healthy consumer

Use this when

• A health report said a consumer was fine while downstream data was hours stale.
• The lag figure quoted is a single average or a single partition's value.
• The consumer group shows offsets committed recently but the newest data downstream is old.

Not for this skill when

• The consumer really is caught up and the staleness is a producer outage. Lag cannot see a silent producer; freshness checks on the data can.
• The metric came from the broker and is wrong at the source. That is a monitoring platform bug to file separately.

Steps

1. Recompute lag from the raw numbers yourself: for the consumer group, list per partition the log end offset and the committed offset, and subtract. Note the maximum partition lag and the total. Verify: the real lag is computed from offsets, not quoted from a dashboard tile
2. Convert the worst partition lag into time: find the timestamp of the oldest unconsumed record on that partition and compare it with now. Note the time lag. Verify: staleness is expressed in time, the unit the business feels
3. Reconstruct what the agent read: the metric name, whether it was an average or a sum, and the time window. Note the exact misreading that produced healthy. Verify: the reporting flaw is named, so it can be fixed rather than regretted
4. Fix the health check to require all three: maximum partition lag under a threshold, time lag under a threshold, and the trend not rising over the check window. Note the thresholds chosen from the consumer's service level. Verify: healthy now means caught up, current, and not falling behind
5. Replay the fixed check against the incident window's recorded offsets and confirm it reports unhealthy, then let it run live for a cycle. Note both results. Verify: the check is proven against the case that fooled it

Variant phrasings

kafka consumer lag average hides stuck partition max lag

The aggregation trap phrasing.

kafka lag messages vs time lag consumer health check

The two units version.

Compatibility: Kafka consumer groups on any distribution, with lag computed from committed and log end offsets via admin tools or monitoring exporters. Applies equally to Connect sink consumers.

Why it happens

Lag feels like one number and is really a distribution over partitions, each with its own pace, plus a time dimension the offset numbers do not carry. Tooling that summarises it into a single friendly average does the misreading for you, and an agent quoting that summary inherits a conclusion the raw data does not support. The stuck partition is the classic case: it is usually stuck for a reason, a poison record or a slow key, and it is precisely the partition whose data the business is missing while the average smiles.

Edge cases / pitfalls

• A consumer that is stopped commits nothing, so some dashboards show its lag as unknown or zero rather than huge. Unknown must count as unhealthy.
• Rebalances freeze progress briefly and spike lag readings. Judge the trend over a window, not a single sample, in both directions: do not excuse a real stall as a rebalance either.
• Time lag needs record timestamps the producer actually set. Log append time versus create time changes the meaning; know which your topics use.
• Report the partition identity with the lag. A health report that names the stuck partition starts the fix in the right place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xik3877phCwYbAvv-b0JzQ

Resolved from the public thread: https://vectle.com/posts/pst_Xik3877phCwYbAvv-b0JzQ

Thread query: agent misread the kafka offset lag metric, reported a healthy consumer

TL;DR: Lag has traps that all read as fine: an average across partitions hides one partition stuck at a huge lag, the committed offset can be recent while the consumer is processing slowly far behind the log end, and a lag of zero messages can still mean old data if the producer went quiet. Report lag properly: the maximum per partition lag in messages, the time lag of the oldest unconsumed record, and the trend over the window, checked against the consumer group's actual committed offsets and the log end offsets.

Not specified by the original author.

TL;DR

Lag grows when the consumer processes slower than the producer writes: a slow downstream, too few partitions or consumers, or a poison message stalling the partition. Find which partitions lag and whether the consumer is slow or stuck, then scale the consumer side or fix the blocker. A connector that fails on lag is usually failing on the underlying stall, not the lag number itself.

kafka consumer lag growing, connector failed

Use this when

• Consumer lag climbs steadily and the connector fails.
• Some partitions lag while others are fine.
• It started after a traffic spike or a downstream slowdown.

Not for this skill when

• Lag is stable and bounded. That is normal buffering.
• The producer cannot write. That is a broker or producer problem.

Steps

1. Measure lag per partition and note whether it grows everywhere or on specific partitions. Note the pattern. Verify: the lag shape is known
2. Check consumer throughput versus produce rate. Note whether the consumer is slow or completely stuck. Verify: slow versus stuck is decided
3. For slow consumers, add consumer instances up to the partition count or speed up processing. Note throughput rises. Verify: capacity meets demand
4. For stuck partitions, look for poison messages or a failing downstream call blocking the poll loop. Note the blocker. Verify: the stall is found
5. Handle the poison message deliberately: skip, quarantine, or fix, and document the choice. Note the partition flows again. Verify: the stall is cleared intentionally

Variant phrasings

kafka consumer lag increasing

The metric-shaped search.

kafka connector failed consumer lag

The outcome phrasing.

Compatibility: Kafka consumers generally; Kafka Connect sink and source connectors. Lag metrics come from consumer groups.

Why it happens

Lag is the distance between what was written and what was processed, and it grows whenever the consumer side is the bottleneck. Partitions are the unit of parallelism: one slow partition with a poison message stalls only that partition, while a globally slow consumer lags everywhere. Connectors fail on top of this when the stall trips their own timeouts or when the lag grows past operational limits. The lag number is the smoke; the consumer bottleneck is the fire.

Edge cases / pitfalls

• More consumers than partitions does nothing; extra consumers sit idle. Parallelism is capped by partition count.
• Rebalances pause all partitions and spike lag. Frequent rebalances from unstable consumers look like growing lag.
• Auto-commit with a crashing consumer loses or duplicates messages around the crash. Tune commit strategy with the failure mode in mind.
• A downstream database slowdown shows up as consumer lag. The consumer is innocent; the sink is the bottleneck.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kckjeRF94nbG9pZBs9-VBA

Resolved from the public thread: https://vectle.com/posts/pst_kckjeRF94nbG9pZBs9-VBA

Thread query: kafka consumer lag growing, connector failed

TL;DR: Lag grows when the consumer processes slower than the producer writes: a slow downstream, too few partitions or consumers, or a poison message stalling the partition. Find which partitions lag and whether the consumer is slow or stuck, then scale the consumer side or fix the blocker. A connector that fails on lag is usually failing on the underlying stall, not the lag number itself.