## TL;DR

The agent saw low CPU on Kafka brokers and stopped them, but brokers are not compute-bound: they hold partitions, and removing them forces rebalancing and starves consumers, so lag exploded. Before touching brokers, check partition assignments, under-replicated partitions, and consumer lag; only stop brokers with no assigned partitions and healthy cluster state. Broker count is a durability decision, not a CPU decision.

## Error

```text
agent stopped "idle" kafka brokers  -  consumer lag exploded overnight
```

## Steps

1. Confirm the cause: consumer lag spiked after the broker stop, and partition rebalancing followed. Expected: the lag timeline matches the stop event.
2. Restart the stopped brokers (or add capacity) to restore the partition layout. Expected: lag drains back to baseline.
3. Add pre-stop checks to the agent: no assigned partitions, no under-replicated partitions, consumer lag healthy. Expected: the checks gate any broker stop.
4. Require human approval for broker topology changes. Expected: no automated broker stops without review.
5. Re-run the idle analysis with the gate. Expected: only truly unassigned brokers are flagged.

## When to use

- Kafka brokers are flagged idle on CPU metrics.
- Consumer lag spikes after broker changes.
- Scoping automated Kafka topology changes.

## When not to use

- Brokers have no partitions and the cluster is healthy (safe).
- The lag has another cause.
- The cluster is being decommissioned intentionally.

## Tool compatibility

Apache Kafka, MSK (current 2026 releases). Applies to cost agents managing Kafka.

### Variant phrasings

### kafka "idle" on cpu but critical for partitions

Brokers are judged by partition health, not CPU.

### consumer lag after broker rightsizing

Restore the brokers and gate on partition assignment.

### broker shutdown caused a rebalance storm

Check under-replicated partitions before any topology change.

## Why it happens

Kafka brokers look idle by CPU because their work is I/O and coordination, not computation. CPU-based idle detection is blind to the broker's real job: hosting partitions and serving consumers. Removing brokers triggers leader elections and rebalancing, which is exactly the expensive, lag-inducing event the "optimization" was supposed to avoid. Topology changes need partition-aware gating and human approval.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pdDDeS5X1NoAGXXy-NH7dA