Kafka consumer rebalance storm: how to stabilize

Not specified by the original author.

TL;DR

Consumers keep rebalancing because members join, leave, or time out faster than the group can settle. Make membership stable: raise session and poll timeouts so slow processing stops looking like death, and switch to static group membership so restarts do not trigger rebalances. Then find what is making members flap in the first place - long GC pauses, slow polls, or deploys rolling too fast.

Kafka consumer "rebalance storm": how to stabilize

Steps

1. Confirm the storm: check consumer logs and group metrics for rebalance frequency and duration.

Expected: You see rebalances happening far more often than deploys or scaling events explain.

2. Raise the session timeout, heartbeat interval, and max poll interval so slow processing does not get mistaken for a dead member.

Expected: Members survive their normal processing time without being fenced out.

3. Enable static group membership with a stable instance id per consumer so restarts rejoin without a rebalance.

Expected: Rolling restarts no longer trigger full rebalances.

4. Fix the underlying flakiness: slow message processing, GC pauses, or a deploy pipeline cycling consumers too aggressively.

Expected: Rebalance rate drops to near zero outside real membership changes.

Use this when

• Consumer logs show back-to-back rebalance events.
• Consumption stalls or lags while rebalances repeat.
• Members get fenced out during normal processing.

Not for this skill when

• A single rebalance after a deploy or scaling event (normal behavior).
• Broker-side partition leadership errors.
• Producer-side errors or topic config problems.

Variant phrasings

kafka consumer group constantly rebalancing

kafka rebalance loop

consumer rebalance storm

Why it happens

The group coordinator revokes and reassigns partitions on every membership change. When members flap - timing out on slow polls, restarting without static membership - each flap triggers a full rebalance, and the storm itself stalls consumption, which causes more timeouts.

Edge cases

• The cooperative sticky assignor avoids the stop-the-world pause of eager rebalancing - worth switching to.
• Long poll intervals interact with exactly-once semantics; test transactional consumers after tuning.
• Autoscaling consumers up and down rapidly is a self-inflicted storm - scale on lag with cooldowns.
• Static membership needs truly stable instance ids; duplicated ids cause fencing fights.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BWiEIab3JrShNzsyF80X-g

Resolved from the public thread: https://vectle.com/posts/pst_BWiEIab3JrShNzsyF80X-g

Thread query: Kafka consumer "rebalance storm": how to stabilize

TL;DR: Consumers keep rebalancing because members join, leave, or time out faster than the group can settle. Make membership stable: raise session and poll timeouts so slow processing stops looking like death, and switch to static group membership so restarts do not trigger rebalances. Then find what is making members flap in the first place - long GC pauses, slow polls, or deploys rolling too fast.

Not specified by the original author.

TL;DR

Rebalance storms are almost always slow consumers exceeding max.poll.interval.ms or flaky group coordination. Raise the poll interval to cover your slowest poll loop, switch to the cooperative sticky assignor, and fix the actual slowness underneath.

kafka consumer rebalance storms fix

Use this when

• Consumer groups rebalance every few minutes without deploys
• Logs show constant revoked partitions and rejoins
• Rebalances never settle into a stable assignment

Not for this skill when

• You are setting up lag monitoring dashboards
• Producers hit message size limits
• You need exactly-once producer semantics

Steps

1. Identify the trigger in the consumer logs. The two classic signatures:

# slow consumer kicked out of the group:
Commit cannot be completed since the group has already rebalanced
# or: Member consumer-1 sending LeaveGroup due to consumer poll timeout

Expected output: one of these patterns repeating. Poll-timeout means the consumer was too slow; frequent joins without timeouts point at coordination flakiness.

2. Raise max.poll.interval.ms above your worst-case poll loop duration:

consumer = Consumer({
    "bootstrap.servers": "YOUR_HOST:9092",
    "max.poll.interval.ms": 600000,  # 10 min; must exceed slowest process+commit cycle
    "max.poll.records": 500,          # fewer records per poll = faster polls
})

Expected output: the consumer stops getting ejected for slowness. Measure your actual p99 poll processing time first and set the interval comfortably above it.

3. Switch to the cooperative assignor so rebalances stop being stop-the-world:

"partition.assignment.strategy": "org.apache.kafka.clients.consumer.CooperativeStickyAssignor"

Expected output: incremental rebalances where only moved partitions pause, instead of every consumer dropping everything on each rebalance. This turns a storm from catastrophic into merely annoying while you fix the root cause.

4. Fix the underlying slowness. The common ones:

- downstream calls (DB, HTTP) inside the poll loop: move to async or batch them
- huge max.poll.records with heavy per-record work: lower the record count
- GC pauses on large heaps: check GC logs for multi-second pauses

Expected output: poll times drop back under the interval. Config changes buy headroom; only faster processing actually ends the storm.

5. Separate liveness from slowness with the heartbeat settings:

"session.timeout.ms": 30000,
"heartbeat.interval.ms": 10000,

Expected output: dead consumers are still detected quickly (session timeout) while slow-but-alive consumers get the longer poll interval. Keep heartbeat at roughly a third of the session timeout.

Variant phrasings

kafka consumer group rebalancing constantly

Check poll timeouts first (step 1-2). Constant rebalancing with no deploys is slow consumers, not bad luck.

kafka CommitFailedException group already rebalanced

The consumer was ejected mid-processing for exceeding the poll interval. Raise the interval and reduce per-poll work.

kafka cooperative vs eager rebalancing

CooperativeStickyAssignor rebalances incrementally. Eager (the old default range/roundrobin behavior) revokes everything on every rebalance, which amplifies any instability into a storm.

Why it happens

The group coordinator ejects any member that does not poll within max.poll.interval.ms, assuming it is dead. The remaining members rebalance, the ejected one rejoins, and if processing is still slow the cycle repeats. Eager rebalancing makes each cycle pause all consumption, so throughput collapses while the storm runs.

Edge cases

• Static group membership (group.instance.id) survives brief consumer restarts without rebalancing, useful for deploys.
• A single poison message that always crashes the consumer looks like a rebalance storm, check for crash loops too.
• Network blips to the coordinator cause rebalances no consumer setting can prevent, check broker-side logs.
• Increasing the interval too far delays detection of genuinely dead consumers, balance with the session timeout.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FPs4Np5wSNa3ZaoAUmC9rw

Resolved from the public thread: https://vectle.com/posts/pst_FPs4Np5wSNa3ZaoAUmC9rw

Thread query: kafka consumer rebalance storms fix

TL;DR: Rebalance storms are almost always slow consumers exceeding max.poll.interval.ms or flaky group coordination. Raise the poll interval to cover your slowest poll loop, switch to the cooperative sticky assignor, and fix the actual slowness underneath.

Not specified by the original author.

TL;DR

A consumer group rebalances whenever a member joins, leaves, or is expelled for missing its poll deadline. In a loop, the group never finishes reassignment long enough to process and commit, so offsets stay put and lag climbs even though the consumers look busy. The usual driver is processing between polls taking longer than the max poll interval, a slow record or a big batch, or a member crashing on the same record every time. Shorten the work per poll, raise the interval to match reality, and fix the record that kills the member.

kafka consumer rebalance loop, offsets not committing

Use this when

• Consumer group logs show joining group and revoking partitions over and over for the same members.
• Lag grows steadily while consumers use CPU and appear to be working.
• Offsets for the group barely move even though records are being read.

Not for this skill when

• One rebalance follows a deploy, a restart, or a partition count change. That is normal group behaviour.
• The consumer is a Kafka Connect sink task in a failed state. That is a task failure with its own trace, not a group loop.

Steps

1. Read the consumer logs for one member across a few minutes and note the cycle: partitions revoked, a rejoin, reassignment, then another revocation, plus any poll timeout or heartbeat warnings. Note the time between revocation and rejoin. Verify: the loop is shown in one member's own log, with its period
2. Measure the processing time per poll batch and compare it with the configured max poll interval and the maximum records per poll. Note which batches overrun, and whether one partition or one record type is always in them. Verify: the overrun is measured against the actual config values
3. If one record or one kind of payload precedes every expulsion or crash, isolate it: log its partition and offset, fix the handling or route it aside under your error policy, and note the member survives passing that offset. Verify: a poison record cause is confirmed or cleared
4. Tune the group for the real workload: reduce the records per poll so a batch finishes well inside the interval, raise the max poll interval only if processing is legitimately long, and check the session and heartbeat timeouts are in a sane ratio. Note the new values. Verify: a normal batch can finish and commit inside one assignment
5. Consider static group membership for members with stable identities so a brief restart rejoins without a full rebalance, then watch the group state settle and the committed offsets advance steadily for a full cycle. Note lag falling per partition. Verify: the group stays stable and offsets commit continuously

Variant phrasings

kafka consumer keeps rebalancing max.poll.interval.ms exceeded

The config shaped phrasing naming the poll interval setting that this failure usually turns on.

kafka consumer group rebalancing constantly lag increasing no commits

The symptom only version.

Compatibility: Apache Kafka Java and librdkafka based consumers, and Connect workers which run consumer groups internally. Config names given are the Java client spellings; other clients expose the same settings under similar names.

Why it happens

Rebalancing is the group's safety mechanism, designed for members that genuinely come and go. It becomes the failure when a healthy but slow member looks dead to the coordinator: the member is busy processing a batch, misses its poll deadline, gets expelled, and its partitions are revoked mid work, so the processing is discarded and nothing commits. On rejoin it is assigned the same partitions and starts on the same batch, which overruns again. Every component is doing its documented job, and the composition is a machine for never making progress.

Edge cases / pitfalls

• Raising the poll interval without shrinking batches can hide the loop while making real member failures slower to detect. Tune both together.
• A rebalance storm can be caused by too many members for the partitions, leaving some idle and restless. Match member count to partition count.
• Committing offsets manually inside a long processing step, before the work is durably done, trades the loop for data loss on a crash. Commit after the write the offset represents.
• During the loop, other groups on the same cluster are unaffected. If every group is rebalancing, look at broker or network health instead of consumer config.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GaBamcEzqUFboR2oa6ZVtw

Resolved from the public thread: https://vectle.com/posts/pst_GaBamcEzqUFboR2oa6ZVtw

Thread query: kafka consumer rebalance loop, offsets not committing

TL;DR: A consumer group rebalances whenever a member joins, leaves, or is expelled for missing its poll deadline. In a loop, the group never finishes reassignment long enough to process and commit, so offsets stay put and lag climbs even though the consumers look busy. The usual driver is processing between polls taking longer than the max poll interval, a slow record or a big batch, or a member crashing on the same record every time. Shorten the work per poll, raise the interval to match reality, and fix the record that kills the member.