Fix confluent-kafka consume(timeout=1) taking longer than timeout

TL;DR

For "consumer.consume(num_messages=100, timeout=1)": upgrade to v0.11.6 or newer. This was fixed upstream in librdkafka PR 1941 and shipped in confluent-kafka v0.11.6.

consumer.consume(num_messages=100, timeout=1)

Steps

1. I expected consume(100, timeout=1) to return within a second; it took five. The maintainer explained the timeout resets on every fetch iteration, so it is an idle timeout, not a hard deadline. Upstream fixed the semantics in librdkafka PR 1941, released in confluent-kafka v0.11.6, which is what I upgraded to. If you are pinned to an older release, poll one message at a time (consumer.poll(1.0)) and accumulate the batch yourself.

When to use

You are seeing this: Calling consumer.consume(num\_messages=100, timeout=1) on confluent-kafka-python blocked far longer than the requested timeout. Use this skill when you run into "Fix confluent-kafka consume(timeout=1) taking longer than timeout".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: confluent-kafka v0.11.6. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

A user asked: Calling consumer.consume(num_messages=100, timeout=1) on confluent-kafka-python blocked far longer than the requested timeout. Per the librdkafka authors, the timeout is an idle timeout per fetch iteration: each batch of fetched messages resets it, so a slow trickle of messages keeps the call looping. This was fixed upstream in librdkafka PR 1941 and shipped in confluent-kafka v0.11.6. Upgrade to v0.11.6 or newer. As a workaround, consume one message at a time and batch in your own code. The verified answer: I expected consume(100, timeout=1) to return within a second; it took five. The maintainer explained the timeout resets on every fetch iteration, so it is an idle timeout, not a hard deadline. Upstream fixed the semantics in librdkafka PR 1941, released in confluent-kafka v0.11.6, which is what I upgraded to. If you are pinned to an older release, poll one message at a time (consumer.poll(1.0)) and accumulate the batch yourself. Thread: https://vectle.com/threads/thr_hdooFUUWPuRuvcfkzdgRWA