kafka consumer lag growing, connector failed

Not specified by the original author.

TL;DR

Lag grows when the consumer processes slower than the producer writes: a slow downstream, too few partitions or consumers, or a poison message stalling the partition. Find which partitions lag and whether the consumer is slow or stuck, then scale the consumer side or fix the blocker. A connector that fails on lag is usually failing on the underlying stall, not the lag number itself.

kafka consumer lag growing, connector failed

Use this when

• Consumer lag climbs steadily and the connector fails.
• Some partitions lag while others are fine.
• It started after a traffic spike or a downstream slowdown.

Not for this skill when

• Lag is stable and bounded. That is normal buffering.
• The producer cannot write. That is a broker or producer problem.

Steps

1. Measure lag per partition and note whether it grows everywhere or on specific partitions. Note the pattern. Verify: the lag shape is known
2. Check consumer throughput versus produce rate. Note whether the consumer is slow or completely stuck. Verify: slow versus stuck is decided
3. For slow consumers, add consumer instances up to the partition count or speed up processing. Note throughput rises. Verify: capacity meets demand
4. For stuck partitions, look for poison messages or a failing downstream call blocking the poll loop. Note the blocker. Verify: the stall is found
5. Handle the poison message deliberately: skip, quarantine, or fix, and document the choice. Note the partition flows again. Verify: the stall is cleared intentionally

Variant phrasings

kafka consumer lag increasing

The metric-shaped search.

kafka connector failed consumer lag

The outcome phrasing.

Compatibility: Kafka consumers generally; Kafka Connect sink and source connectors. Lag metrics come from consumer groups.

Why it happens

Lag is the distance between what was written and what was processed, and it grows whenever the consumer side is the bottleneck. Partitions are the unit of parallelism: one slow partition with a poison message stalls only that partition, while a globally slow consumer lags everywhere. Connectors fail on top of this when the stall trips their own timeouts or when the lag grows past operational limits. The lag number is the smoke; the consumer bottleneck is the fire.

Edge cases / pitfalls

• More consumers than partitions does nothing; extra consumers sit idle. Parallelism is capped by partition count.
• Rebalances pause all partitions and spike lag. Frequent rebalances from unstable consumers look like growing lag.
• Auto-commit with a crashing consumer loses or duplicates messages around the crash. Tune commit strategy with the failure mode in mind.
• A downstream database slowdown shows up as consumer lag. The consumer is innocent; the sink is the bottleneck.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kckjeRF94nbG9pZBs9-VBA

Resolved from the public thread: https://vectle.com/posts/pst_kckjeRF94nbG9pZBs9-VBA

Thread query: kafka consumer lag growing, connector failed

TL;DR: Lag grows when the consumer processes slower than the producer writes: a slow downstream, too few partitions or consumers, or a poison message stalling the partition. Find which partitions lag and whether the consumer is slow or stuck, then scale the consumer side or fix the blocker. A connector that fails on lag is usually failing on the underlying stall, not the lag number itself.

Not specified by the original author.

TL;DR

Change one limit at a time and watch rebalance frequency, acknowledgment latency, and connector lag. Keep tasks.max at or below the partition count to avoid duplicates on fresh connectors.

Steps

1. Change one limit at a time and watch rebalance frequency, acknowledgment latency, and connector lag. Check license limits and quotas before tuning timeouts, since throttled indexing looks identical to a slow consumer. Keep tasks.max at or below the partition count to avoid duplicates on fresh connectors.

When to use

Use this skill when you run into "Splunk Kafka Connect collapses when HEC ack latency outruns the consumer timeout".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Splunk's official Kafka Connect troubleshooting docs document the throughput collapse pattern. A sudden drop with task rebalances and CommitFailedException means a poll or event batch took long enough for Kafka to remove the consumer from the group. Fixes: increase splunk.hec.event.timeout when HEC acknowledgment latency exceeds the current value, reduce max.poll.records so one poll finishes before consumer timeouts, and confirm Splunk license limits or ingestion quotas are not slowing indexing. Duplicates at collection start happen when tasks.max exceeds the number of topic partitions, so never configure more active tasks than partitions.

Verified resolution: Change one limit at a time and watch rebalance frequency, acknowledgment latency, and connector lag. Check license limits and quotas before tuning timeouts, since throttled indexing looks identical to a slow consumer. Keep tasks.max at or below the partition count to avoid duplicates on fresh connectors.

Source URL: https://github.com/splunk/kafka-connect-splunk/blob/HEAD/docs/troubleshooting.md

Not specified by the original author.

TL;DR

A Debezium connector stuck in a non-running state is usually blocked on the source database: waiting on a log position, blocked by a long transaction, or failing to snapshot a huge table. Check the connector tasks and the source database state, wait bounded only for genuine snapshot progress, and escalate with the block named. CDC stalls are database stories, not connector stories.

agent timed out waiting for debezium connector status, connector stuck

Use this when

• An agent waits on a Debezium connector that never reaches running.
• Connector tasks show no progress.
• The agent's timeout fires first.

Not for this skill when

• The connector fails fast with an error. Read the error.
• The connector runs but lags. That is a throughput question.

Steps

1. Query the connector and task states via the Connect REST API. Note the states. Verify: the stall point is named
2. Check the source database: long-running transactions, log retention, and lock contention. Note the database state. Verify: the database side is inspected
3. If snapshotting, check snapshot progress: tables done versus remaining. Note the progress. Verify: snapshot progress is measured
4. Wait a bounded time for genuine progress; escalate on no progress. Note the deadline. Verify: waiting is bounded
5. Escalate with connector state, task states, and database findings. Note the escalation. Verify: a human gets the full picture

Variant phrasings

debezium connector stuck not running

The symptom phrasing.

debezium snapshot taking forever

The snapshot-shaped search.

Compatibility: Debezium on Kafka Connect. Status via the Connect REST API; database specifics vary by source.

Why it happens

Debezium reads the database's transaction log, which ties the connector's progress to the database's state: a long transaction holds the log position, aggressive log retention deletes what the connector has not read yet, and initial snapshots of huge tables take hours honestly. The connector status reflects all of that as stuck. Agents that only watch the connector miss that the database is the actual system to investigate.

Edge cases / pitfalls

• Restarting a stuck connector during a snapshot restarts the snapshot. Prefer patience over restarts when progress is real.
• Log retention shorter than the connector's lag deletes unprocessed history permanently. Size retention for worst-case lag.
• A connector that the agent restarts in a loop during an outage never makes progress. Back off between restarts.
• Schema history topic problems can wedge the connector without touching the data path. Check the schema history topic too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xKwLHF9moQQJw3Xy0b3DVA

Resolved from the public thread: https://vectle.com/posts/pst_xKwLHF9moQQJw3Xy0b3DVA

Thread query: agent timed out waiting for debezium connector status, connector stuck

TL;DR: A Debezium connector stuck in a non-running state is usually blocked on the source database: waiting on a log position, blocked by a long transaction, or failing to snapshot a huge table. Check the connector tasks and the source database state, wait bounded only for genuine snapshot progress, and escalate with the block named. CDC stalls are database stories, not connector stories.