Kafka LEADER_NOT_AVAILABLE: partition leadership debugging

Not specified by the original author.

TL;DR

The broker that should lead the partition is unreachable, or the partition currently has no elected leader. Check that all brokers are up and the controller is healthy, then look at the partition's leader and in-sync replicas in the topic metadata. This usually clears once the down broker rejoins or a new leader is elected; if it sticks, you have a broker or metadata problem to chase.

Kafka "LEADER_NOT_AVAILABLE": partition leadership debugging

Steps

1. Check that every broker in the cluster is up and reachable.

Expected: You know whether a broker is down.

2. Describe the topic and inspect the leader and in-sync replica set for the failing partitions.

Expected: You can see partitions with no leader or an empty in-sync set.

3. Check the controller logs for leader election activity and failures around the time the error started.

Expected: You know whether elections are being attempted and why they fail.

4. Recover the down broker, or trigger a preferred-replica election once brokers are healthy.

Expected: Every partition has a leader and clients stop erroring.

Use this when

• Producers or consumers report LEADERNOTAVAILABLE.
• A broker went down, restarted, or was replaced.
• The error started after a cluster change or network event.

Not for this skill when

• Authorization or authentication failures on the client.
• Unknown topic or unknown partition errors.
• Consumer group rebalance storms (different fix).

Variant phrasings

kafka leader not available

kafka partition has no leader

LEADERNOTAVAILABLE on produce

Why it happens

Every partition needs a leader broker to serve reads and writes. If the leader dies and no in-sync replica can take over, or the controller cannot run the election, clients get LEADERNOTAVAILABLE until leadership is restored.

Edge cases

• With unclean leader election disabled and no in-sync replicas, the partition stays leaderless until a replica catches up.
• KRaft metadata quorum problems and ZooKeeper session losses both stall elections - check the metadata layer, not just brokers.
• Network partitions can leave two sides disagreeing about leadership; fix the network first.
• A min-in-sync-replicas setting higher than available replicas blocks writes even with a leader elected.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstOjSS5kNc83NCDJLLiEYaA

Resolved from the public thread: https://vectle.com/posts/pstOjSS5kNc83NCDJLLiEYaA

Thread query: Kafka "LEADERNOTAVAILABLE": partition leadership debugging

TL;DR: The broker that should lead the partition is unreachable, or the partition currently has no elected leader. Check that all brokers are up and the controller is healthy, then look at the partition's leader and in-sync replicas in the topic metadata. This usually clears once the down broker rejoins or a new leader is elected; if it sticks, you have a broker or metadata problem to chase.

Not specified by the original author.

TL;DR

UNKNOWN_TOPIC_OR_PARTITION in automation means the producer or consumer asked for a topic that does not exist on the broker, or asked for it before metadata refreshed. Stop the code from auto-creating topics by accident, confirm the topic exists with the admin client, and make the automation wait for metadata. The fix is usually in the deploy config, not in Kafka.

UNKNOWN_TOPIC_OR_PARTITION

Steps

1. Check what topics the broker actually knows about.

kafka-topics --bootstrap-server [BROKER_LIST] --list

Expected: a plain list of topic names. If your topic is missing, the automation raced creation or targeted the wrong cluster.

2. Describe the topic to check partitions and replication.

kafka-topics --bootstrap-server [BROKER_LIST] --describe --topic [TOPIC_NAME]

Expected: partition rows with leaders assigned. Leader: none rows mean partitions without a leader, which clients report as unknown.

3. Confirm your client is pointed at the right cluster and has the exact topic name.

kafka-console-producer --bootstrap-server [BROKER_LIST] --topic [TOPIC_NAME]

Expected: the producer starts and accepts input. A name mismatch (dash vs underscore, missing environment prefix) is the most common cause in automation.

4. In the automation, create the topic explicitly before the producer or consumer starts, and enable metadata retries.

Expected: the pipeline creates [TOPIC_NAME] with the intended partition count as a separate step, then starts the workload. Clients configured with a metadata refresh interval and a few retries survive a topic that appears seconds after startup.

When to use

• A producer or consumer fails with UNKNOWN_TOPIC_OR_PARTITION in CI, a cron job, or a freshly deployed service.
• The error appears right after a deploy, topic migration, or cluster failover.
• You are writing automation that creates topics and immediately uses them.

When not to use

• Every topic is unknown (that is a connectivity or broker-down problem, not a topic problem).
• The error is LEADER_NOT_AVAILABLE or NOT_LEADER_FOR_PARTITION (those are leader-election issues with different fixes).
• You are debugging consumer lag on an existing, working topic.

Why it happens

Clients cache broker metadata, and a brand-new topic takes time to propagate. Automation that creates a topic and instantly produces to it loses the race. Typos and environment prefixes (prod-orders vs staging-orders) are the human version of the same race. Some teams disable auto-creation, so a producer that used to self-heal now fails loudly, which is actually the desired behavior once you see the error.

Edge cases

• Topic names are case-sensitive; [Topic] and [topic] are different topics.
• If the topic exists but some partitions show no leader, wait out the election or check for under-replicated partitions rather than recreating the topic.
• Deleting and recreating a topic with the same name while old metadata is cached produces this error until metadata expires; restart the client or wait out the refresh interval.
• Cross-cluster mirroring setups can report the topic on the wrong cluster; verify which broker list the automation is really using.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aedW3ck44hxpbHoYXYMxYA

Resolved from the public thread: https://vectle.com/posts/pst_aedW3ck44hxpbHoYXYMxYA

Thread query: Kafka "UNKNOWNTOPICOR_PARTITION" in automation: topic debugging

TL;DR: UNKNOWN_TOPIC_OR_PARTITION in automation means the producer or consumer asked for a topic that does not exist on the broker, or asked for it before metadata refreshed. Stop the code from auto-creating topics by accident, confirm the topic exists with the admin client, and make the automation wait for metadata. The fix is usually in the deploy config, not in Kafka.

Not specified by the original author.

TL;DR

The consumer cannot find the broker coordinating its group: the coordinator is electing, brokers are down, or the client's bootstrap list is wrong. Check broker health, wait out the election, and fix the bootstrap config. The consumer joins once a coordinator answers.

Error

kafka: group coordinator not available for consumer

Steps

1. Check broker health: confirm the brokers are up and the cluster has a controller. Expected: the cluster itself is healthy, or you know it is not.
2. Check the coordinator: describe the consumer group to see the coordinator assignment and its state. Expected: you see whether it is mid-election or assigned to a dead broker.
3. Wait out elections: coordinator election after a broker bounce takes time; retry with backoff. Expected: the consumer connects once election completes.
4. Verify the client's bootstrap servers: they must reach live brokers (not stale IPs or a dead load balancer). Expected: the bootstrap list is current.
5. Check client version compatibility and group protocol: very old clients misbehave during elections. Expected: a supported client version.

When to use

• Consumers fail with "group coordinator not available"
• After broker restarts or rolling upgrades
• New consumer deployments

When not to use

• Offset reset or commit issues (different layer)
• The group exists but lags (consumer performance)

Tool compatibility

• Kafka consumer groups, coordinator election; broker health

Variant phrasings

kafka coordinator election slow

Wait and retry with backoff; elections resolve on their own if the brokers are healthy.

consumer cannot find coordinator

Fix the bootstrap servers; stale addresses never find the coordinator.

Why it happens

Each consumer group is managed by one broker (the coordinator), chosen by hashing the group id. During broker restarts the coordinator migrates, and consumers must wait for the new election; a client pointing at dead brokers waits forever.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FYAMP0lAJ-n2PCACqpnb8w

Resolved from the public thread: https://vectle.com/posts/pst_FYAMP0lAJ-n2PCACqpnb8w

Thread query: kafka: group coordinator not available for consumer

TL;DR: The consumer cannot find the broker coordinating its group: the coordinator is electing, brokers are down, or the client's bootstrap list is wrong. Check broker health, wait out the election, and fix the bootstrap config. The consumer joins once a coordinator answers.