agent stopped "idle" kafka brokers - consumer lag exploded overnight

TL;DR

The agent saw low CPU on Kafka brokers and stopped them, but brokers are not compute-bound: they hold partitions, and removing them forces rebalancing and starves consumers, so lag exploded. Before touching brokers, check partition assignments, under-replicated partitions, and consumer lag; only stop brokers with no assigned partitions and healthy cluster state. Broker count is a durability decision, not a CPU decision.

Error

agent stopped "idle" kafka brokers  -  consumer lag exploded overnight

Steps

1. Confirm the cause: consumer lag spiked after the broker stop, and partition rebalancing followed. Expected: the lag timeline matches the stop event.
2. Restart the stopped brokers (or add capacity) to restore the partition layout. Expected: lag drains back to baseline.
3. Add pre-stop checks to the agent: no assigned partitions, no under-replicated partitions, consumer lag healthy. Expected: the checks gate any broker stop.
4. Require human approval for broker topology changes. Expected: no automated broker stops without review.
5. Re-run the idle analysis with the gate. Expected: only truly unassigned brokers are flagged.

When to use

• Kafka brokers are flagged idle on CPU metrics.
• Consumer lag spikes after broker changes.
• Scoping automated Kafka topology changes.

When not to use

• Brokers have no partitions and the cluster is healthy (safe).
• The lag has another cause.
• The cluster is being decommissioned intentionally.

Tool compatibility

Apache Kafka, MSK (current 2026 releases). Applies to cost agents managing Kafka.

Variant phrasings

kafka "idle" on cpu but critical for partitions

Brokers are judged by partition health, not CPU.

consumer lag after broker rightsizing

Restore the brokers and gate on partition assignment.

broker shutdown caused a rebalance storm

Check under-replicated partitions before any topology change.

Why it happens

Kafka brokers look idle by CPU because their work is I/O and coordination, not computation. CPU-based idle detection is blind to the broker's real job: hosting partitions and serving consumers. Removing brokers triggers leader elections and rebalancing, which is exactly the expensive, lag-inducing event the "optimization" was supposed to avoid. Topology changes need partition-aware gating and human approval.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pdDDeS5X1NoAGXXy-NH7dA