Kafka "UNKNOWN_TOPIC_OR_PARTITION" in automation: topic debugging
This skill debugs Kafka UNKNOWN_TOPIC_OR_PARTITION errors in automation. It shows how to verify the topic exists, check partition leadership, catch name and cluster mismatches, and make pipelines create topics before producing. Use it when producers or consumers fail right after a deploy or topic change.
TL;DR
UNKNOWN_TOPIC_OR_PARTITION in automation means the producer or consumer asked for a topic that does not exist on the broker, or asked for it before metadata refreshed. Stop the code from auto-creating topics by accident, confirm the topic exists with the admin client, and make the automation wait for metadata. The fix is usually in the deploy config, not in Kafka.
UNKNOWN_TOPIC_OR_PARTITIONSteps
- Check what topics the broker actually knows about.
kafka-topics --bootstrap-server [BROKER_LIST] --listExpected: a plain list of topic names. If your topic is missing, the automation raced creation or targeted the wrong cluster.
- Describe the topic to check partitions and replication.
kafka-topics --bootstrap-server [BROKER_LIST] --describe --topic [TOPIC_NAME]Expected: partition rows with leaders assigned. Leader: none rows mean partitions without a leader, which clients report as unknown.
- Confirm your client is pointed at the right cluster and has the exact topic name.
kafka-console-producer --bootstrap-server [BROKER_LIST] --topic [TOPIC_NAME]Expected: the producer starts and accepts input. A name mismatch (dash vs underscore, missing environment prefix) is the most common cause in automation.
- In the automation, create the topic explicitly before the producer or consumer starts, and enable metadata retries.
Expected: the pipeline creates [TOPIC_NAME] with the intended partition count as a separate step, then starts the workload. Clients configured with a metadata refresh interval and a few retries survive a topic that appears seconds after startup.
When to use
- A producer or consumer fails with
UNKNOWN_TOPIC_OR_PARTITIONin CI, a cron job, or a freshly deployed service. - The error appears right after a deploy, topic migration, or cluster failover.
- You are writing automation that creates topics and immediately uses them.
When not to use
- Every topic is unknown (that is a connectivity or broker-down problem, not a topic problem).
- The error is
LEADER_NOT_AVAILABLEorNOT_LEADER_FOR_PARTITION(those are leader-election issues with different fixes). - You are debugging consumer lag on an existing, working topic.
Why it happens
Clients cache broker metadata, and a brand-new topic takes time to propagate. Automation that creates a topic and instantly produces to it loses the race. Typos and environment prefixes (prod-orders vs staging-orders) are the human version of the same race. Some teams disable auto-creation, so a producer that used to self-heal now fails loudly, which is actually the desired behavior once you see the error.
Edge cases
- Topic names are case-sensitive;
[Topic]and[topic]are different topics. - If the topic exists but some partitions show no leader, wait out the election or check for under-replicated partitions rather than recreating the topic.
- Deleting and recreating a topic with the same name while old metadata is cached produces this error until metadata expires; restart the client or wait out the refresh interval.
- Cross-cluster mirroring setups can report the topic on the wrong cluster; verify which broker list the automation is really using.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_aedW3ck44hxpbHoYXYMxYA