kafka consumer lag growing, connector failed

TL;DR

Lag grows when the consumer processes slower than the producer writes: a slow downstream, too few partitions or consumers, or a poison message stalling the partition. Find which partitions lag and whether the consumer is slow or stuck, then scale the consumer side or fix the blocker. A connector that fails on lag is usually failing on the underlying stall, not the lag number itself.

kafka consumer lag growing, connector failed

Use this when

• Consumer lag climbs steadily and the connector fails.
• Some partitions lag while others are fine.
• It started after a traffic spike or a downstream slowdown.

Not for this skill when

• Lag is stable and bounded. That is normal buffering.
• The producer cannot write. That is a broker or producer problem.

Steps

1. Measure lag per partition and note whether it grows everywhere or on specific partitions. Note the pattern. Verify: the lag shape is known
2. Check consumer throughput versus produce rate. Note whether the consumer is slow or completely stuck. Verify: slow versus stuck is decided
3. For slow consumers, add consumer instances up to the partition count or speed up processing. Note throughput rises. Verify: capacity meets demand
4. For stuck partitions, look for poison messages or a failing downstream call blocking the poll loop. Note the blocker. Verify: the stall is found
5. Handle the poison message deliberately: skip, quarantine, or fix, and document the choice. Note the partition flows again. Verify: the stall is cleared intentionally

Variant phrasings

kafka consumer lag increasing

The metric-shaped search.

kafka connector failed consumer lag

The outcome phrasing.

Compatibility: Kafka consumers generally; Kafka Connect sink and source connectors. Lag metrics come from consumer groups.

Why it happens

Lag is the distance between what was written and what was processed, and it grows whenever the consumer side is the bottleneck. Partitions are the unit of parallelism: one slow partition with a poison message stalls only that partition, while a globally slow consumer lags everywhere. Connectors fail on top of this when the stall trips their own timeouts or when the lag grows past operational limits. The lag number is the smoke; the consumer bottleneck is the fire.

Edge cases / pitfalls

• More consumers than partitions does nothing; extra consumers sit idle. Parallelism is capped by partition count.
• Rebalances pause all partitions and spike lag. Frequent rebalances from unstable consumers look like growing lag.
• Auto-commit with a crashing consumer loses or duplicates messages around the crash. Tune commit strategy with the failure mode in mind.
• A downstream database slowdown shows up as consumer lag. The consumer is innocent; the sink is the bottleneck.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kckjeRF94nbG9pZBs9-VBA