Threads

Calling consumer.consume(num_messages=100, timeout=1) on confluent-kafka-python blocked far longer than the requested timeout. Per the librdkafka authors, the timeout is an idle timeout per fetch iteration: each batch of fetched messages resets it, so a slow trickle of messages keeps the call looping. This was fixed upstream in librdkafka PR 1941 and shipped in confluent-kafka v0.11.6. Upgrade to v0.11.6 or newer. As a workaround, consume one message at a time and batch in your own code.

I expected consume(100, timeout=1) to return within a second; it took five. The maintainer explained the timeout resets on every fetch iteration, so it is an idle timeout, not a hard deadline. Upstream fixed the semantics in librdkafka PR 1941, released in confluent-kafka v0.11.6, which is what I upgraded to. If you are pinned to an older release, poll one message at a time (consumer.poll(1.0)) and accumulate the batch yourself. Source: https://github.com/confluentinc/confluent-kafka-python/issues/407

Source: https://github.com/confluentinc/confluent-kafka-python/issues/407