Splunk Kafka Connect collapses when HEC ack latency outruns the consumer timeout

Change one limit at a time and watch rebalance frequency, acknowledgment latency, and connector lag. Check license limits and quotas before tuning timeouts, since throttled indexing looks identical to a slow consumer. Keep tasks.max at or below the partition count to avoid duplicates on fresh connectors.

Context: Splunk's official Kafka Connect troubleshooting docs document the throughput collapse pattern. A sudden drop with task rebalances and CommitFailedException means a poll or event batch took long enough for Kafka to remove the consumer from the group. Fixes: increase splunk.hec.event.timeout when HEC acknowledgment latency exceeds the current value, reduce max.poll.records so one poll finishes before consumer timeouts, and confirm Splunk license limits or ingestion quotas are not slowing indexing. Duplicates at collection start happen when tasks.max exceeds the number of topic partitions, so never configure more active tasks than partitions.

Matched source

Source: Published skill Original query: "Splunk Kafka Connect collapses when HEC ack latency outruns the consumer timeout" Key terms: collapses, connect, consumer, kafka, latency, outruns, splunk, timeout

Source: Splunk's official Kafka Connect troubleshooting docs document the throughput collapse pattern. A sudden drop with task rebalances and CommitFailedException means a poll or event batch took long enough for Kafka to remove the consumer from the group. Fixes: increase splunk.hec.event.timeout when HEC acknowledgment latency exceeds the current value, reduce max.poll.records so one poll finishes before consumer timeouts, and confirm Splunk license limits or ingestion quotas are not slowing indexing. Duplicates at collection start happen when tasks.max exceeds the number of topic partitions, so never configure more active tasks than partitions.

Verified resolution: Change one limit at a time and watch rebalance frequency, acknowledgment latency, and connector lag. Check license limits and quotas before tuning timeouts, since throttled indexing looks identical to a slow consumer. Keep tasks.max at or below the partition count to avoid duplicates on fresh connectors.

Source URL: https://github.com/splunk/kafka-connect-splunk/blob/HEAD/docs/troubleshooting.md