kafka consumer rebalance storms fix

TL;DR

Rebalance storms are almost always slow consumers exceeding max.poll.interval.ms or flaky group coordination. Raise the poll interval to cover your slowest poll loop, switch to the cooperative sticky assignor, and fix the actual slowness underneath.

kafka consumer rebalance storms fix

Use this when

• Consumer groups rebalance every few minutes without deploys
• Logs show constant revoked partitions and rejoins
• Rebalances never settle into a stable assignment

Not for this skill when

• You are setting up lag monitoring dashboards
• Producers hit message size limits
• You need exactly-once producer semantics

Steps

1. Identify the trigger in the consumer logs. The two classic signatures:

# slow consumer kicked out of the group:
Commit cannot be completed since the group has already rebalanced
# or: Member consumer-1 sending LeaveGroup due to consumer poll timeout

Expected output: one of these patterns repeating. Poll-timeout means the consumer was too slow; frequent joins without timeouts point at coordination flakiness.

2. Raise max.poll.interval.ms above your worst-case poll loop duration:

consumer = Consumer({
    "bootstrap.servers": "YOUR_HOST:9092",
    "max.poll.interval.ms": 600000,  # 10 min; must exceed slowest process+commit cycle
    "max.poll.records": 500,          # fewer records per poll = faster polls
})

Expected output: the consumer stops getting ejected for slowness. Measure your actual p99 poll processing time first and set the interval comfortably above it.

3. Switch to the cooperative assignor so rebalances stop being stop-the-world:

"partition.assignment.strategy": "org.apache.kafka.clients.consumer.CooperativeStickyAssignor"

Expected output: incremental rebalances where only moved partitions pause, instead of every consumer dropping everything on each rebalance. This turns a storm from catastrophic into merely annoying while you fix the root cause.

4. Fix the underlying slowness. The common ones:

- downstream calls (DB, HTTP) inside the poll loop: move to async or batch them
- huge max.poll.records with heavy per-record work: lower the record count
- GC pauses on large heaps: check GC logs for multi-second pauses

Expected output: poll times drop back under the interval. Config changes buy headroom; only faster processing actually ends the storm.

5. Separate liveness from slowness with the heartbeat settings:

"session.timeout.ms": 30000,
"heartbeat.interval.ms": 10000,

Expected output: dead consumers are still detected quickly (session timeout) while slow-but-alive consumers get the longer poll interval. Keep heartbeat at roughly a third of the session timeout.

Variant phrasings

kafka consumer group rebalancing constantly

Check poll timeouts first (step 1-2). Constant rebalancing with no deploys is slow consumers, not bad luck.

kafka CommitFailedException group already rebalanced

The consumer was ejected mid-processing for exceeding the poll interval. Raise the interval and reduce per-poll work.

kafka cooperative vs eager rebalancing

CooperativeStickyAssignor rebalances incrementally. Eager (the old default range/roundrobin behavior) revokes everything on every rebalance, which amplifies any instability into a storm.

Why it happens

The group coordinator ejects any member that does not poll within max.poll.interval.ms, assuming it is dead. The remaining members rebalance, the ejected one rejoins, and if processing is still slow the cycle repeats. Eager rebalancing makes each cycle pause all consumption, so throughput collapses while the storm runs.

Edge cases

• Static group membership (group.instance.id) survives brief consumer restarts without rebalancing, useful for deploys.
• A single poison message that always crashes the consumer looks like a rebalance storm, check for crash loops too.
• Network blips to the coordinator cause rebalances no consumer setting can prevent, check broker-side logs.
• Increasing the interval too far delays detection of genuinely dead consumers, balance with the session timeout.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FPs4Np5wSNa3ZaoAUmC9rw