Understanding Kafka Topics and Partitions

TL;DR: This post already has answers, but I am adding my view with a few pictures from Kafka Definitive Guide Before answering the questions, let's look at an overview of producer components: When a producer is producing a message, it will specify the topic it wants to send the message to. Is that right? Does it care about partitions?

Fix

1. This post already has answers, but I am adding my view with a few pictures from Kafka Definitive Guide Before answering the questions, let's look at an overview of producer components: When a producer is producing a message, it will specify the topic it wants to send the message to. Expected: this specific failure stops.

2. Is that right? Expected: this specific failure stops.

3. Does it care about partitions? Expected: this specific failure stops.

4. The producer will decide target partition to place any message, depending on: Partition id, if it's specified within the message key % num partitions , if no partition id is mentioned Round robin if neither partition id nor message key is available in the message means only the value is available When a subscriber is running - Does it specify its group id so that it can be part of a cluster of consumers of the same topic or several topics that this group of consumers is interested in? Expected: this specific failure stops.

Details

This post already has answers, but I am adding my view with a few pictures from Kafka Definitive Guide Before answering the questions, let's look at an overview of producer components: When a producer is producing a message, it will specify the topic it wants to send the message to. Is that right? Does it care about partitions? You should always configure group.id unless you are using the simple assignment API and you don’t need to store offsets in Kafka. It will not be a part of any group. Source . Does each consumer group have a corresponding partition on the broker or does each consumer have one? In one consumer group, each partition will be processed by one consumer onl

When to use

You hit exactly this: Understanding Kafka Topics and Partitions in Understanding Kafka Topics.

When not to use

A different error, or the same symptom in a different tool. This page only covers the failure above.

Compatibility

Understanding Kafka Topics.

Root cause

The producer will decide target partition to place any message, depending on: Partition id, if it's specified within the message key % num partitions , if no partition id is mentioned Round robin if neither partition id nor message key is available in the message means only the value is available When a subscriber is running - Does it specify its group id so that it can be part of a cluster of consumers of the same topic or several topics that this group of consumers is interested in?

Asked on Stack Overflow so:38024514: Stack Overflow #38024514 (score 415, accepted answer, 447 votes): I am starting to learn Kafka. During my readings, some questions came to my mind:

When a producer is producing a message, it will specify the topic it wants to send the message to. Is that right? Does it care about partitions?

When a subscriber is running, does it specify its group id so that it can be part of a cluster of consumers of the same topic or several topics that this group of consumers is interested in?

Does each consumer group have a corresponding partition on the broker or does each consumer have one? The verified fix/answer: This post already has answers, but I am adding my view with a few pictures from Kafka Definitive Guide Before answering the questions, let's look at an overview of producer components:

When a producer is producing a message, it will specify the topic it wants to send the message to. Is that right? Does it care about partitions?

The producer will decide target partition to place any message, depending on:

Partition id, if it's specified within the message key % num partitions , if no partition id is mentioned Round robin if neither partition id nor message key is available in the message means only the value is available

When a subscriber is running - Does it specify its group id so that it can be part of a cluster of consumers of the same topic or several topics that this group of consumers is interested in?

You should always configure group.id unless you are using the simple assignment API and you don’t need to store offsets in Kafka. It will not be a part of any group. Source .

Does each consumer group have a corresponding partition on the broker or does each consumer have one?

In one consumer group, each partition will be processed by one consumer onl Source: https://stackoverflow.com/questions/38024514/understanding-kafka-topics-and-partitions