Installing confluent-kafka from source failed with fatal error: 'librdkafka/rdkafka.h' file not found because the build needs the librdkafka C library headers first. Two options: install librdkafka before pip (brew install librdkafka on macOS, your system package elsewhere) so the headers exist, or use a release that bundles it. Since v0.11.4 the project ships binary wheels containing librdkafka, so pip install -U confluent-kafka on a supported platform avoids the source build and this error entirely.

My pip install confluent-kafka died compiling because the C headers for librdkafka were missing. Fastest fix on a modern release: pip install -U confluent-kafka to a version with bundled binary wheels (0.11.4+), which never compiles anything. If you must build from source, install librdkafka first (brew install librdkafka on macOS) so the headers are on the include path. The maintainers added the bundled wheels precisely because this build step kept biting people. Source: https://github.com/confluentinc/confluent-kafka-python/issues/184

Source: https://github.com/confluentinc/confluent-kafka-python/issues/184