Splunk Kafka Connect collapses when HEC ack latency outruns the consumer timeout

TL;DR

Change one limit at a time and watch rebalance frequency, acknowledgment latency, and connector lag. Keep tasks.max at or below the partition count to avoid duplicates on fresh connectors.

Steps

1. Change one limit at a time and watch rebalance frequency, acknowledgment latency, and connector lag. Check license limits and quotas before tuning timeouts, since throttled indexing looks identical to a slow consumer. Keep tasks.max at or below the partition count to avoid duplicates on fresh connectors.

When to use

Use this skill when you run into "Splunk Kafka Connect collapses when HEC ack latency outruns the consumer timeout".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Splunk's official Kafka Connect troubleshooting docs document the throughput collapse pattern. A sudden drop with task rebalances and CommitFailedException means a poll or event batch took long enough for Kafka to remove the consumer from the group. Fixes: increase splunk.hec.event.timeout when HEC acknowledgment latency exceeds the current value, reduce max.poll.records so one poll finishes before consumer timeouts, and confirm Splunk license limits or ingestion quotas are not slowing indexing. Duplicates at collection start happen when tasks.max exceeds the number of topic partitions, so never configure more active tasks than partitions.

Verified resolution: Change one limit at a time and watch rebalance frequency, acknowledgment latency, and connector lag. Check license limits and quotas before tuning timeouts, since throttled indexing looks identical to a slow consumer. Keep tasks.max at or below the partition count to avoid duplicates on fresh connectors.

Source URL: https://github.com/splunk/kafka-connect-splunk/blob/HEAD/docs/troubleshooting.md