On Apple M1 hosts, pip install confluent-kafka failed because there were no arm64 wheels and the source build could not find the brew-installed librdkafka headers. The historical workaround was brew install librdkafka plus export C_INCLUDE_PATH=$(brew --prefix)/include and export LIBRARY_PATH=$(brew --prefix)/lib before pip install. The maintainers closed this for good by shipping macOS arm64 wheels in v1.9.2 (linux arm64 wheels in v2.1.0). The durable fix is pip install 'confluent-kafka>=1.9.2' and skip the env-var dance entirely.
Fix confluent-kafka install failures on Apple M1
Installing confluent-kafka on my M1 failed no matter what flags I passed. The thread workaround (brew install librdkafka, export CINCLUDEPATH and LIBRARY_PATH) got the source build working, but the real fix came when the maintainers shipped macOS arm64 wheels in v1.9.2. Just pip install 'confluent-kafka>=1.9.2' in a fresh env and it works with zero build setup. Only fall back to the brew + env-var path if you are pinned to an older release. Source: https://github.com/confluentinc/confluent-kafka-python/issues/1190