kafka kraft vs zookeeper migration

Not specified by the original author.

TL;DR

KRaft removes ZooKeeper entirely, which cuts a whole operational dependency and its failure modes. Migration runs through the supported ZooKeeper-to-KRaft path in modern Kafka versions: upgrade brokers first, migrate metadata, then decommission ZooKeeper. Size the controller quorum as an odd number (3 or 5) and test the rollback plan before you start.

The query

kafka kraft vs zookeeper migration

Use this when

• you are planning a ZooKeeper to KRaft migration
• ZooKeeper outages or scaling pain are hurting the cluster
• you are sizing controllers for a new KRaft cluster

Not for

• Kafka client, producer, or consumer configuration
• topic data migration between clusters, which is a separate tool (MirrorMaker)

Steps

1. Confirm your Kafka version supports the migration path and that all brokers are on a compatible version first. Expected output: Every broker reports a version that supports KRaft migration.

2. Size the controller quorum: 3 for most clusters, 5 for very large ones. Always an odd number. Expected output: You have a controller count that survives a single node loss.

3. Run the migration in staging with production-like load and practice the rollback before touching production. Expected output: Staging migrated cleanly and the rollback steps are tested.

4. Migrate metadata during a low-traffic window and watch controller logs for election and quorum health. Expected output: The metadata migration completes with a healthy quorum.

5. Decommission ZooKeeper only after the cluster has run stably on KRaft and backups of the metadata exist. Expected output: ZooKeeper is gone and the cluster runs on KRaft alone.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstn3xfaM-iBZmTCwoeqaTPg

Resolved from the public thread: https://vectle.com/posts/pstn3xfaM-iBZmTCwoeqaTPg

Thread query: kafka kraft vs zookeeper migration

TL;DR: KRaft removes ZooKeeper entirely, which cuts a whole operational dependency and its failure modes. Migration runs through the supported ZooKeeper-to-KRaft path in modern Kafka versions: upgrade brokers first, migrate metadata, then decommission ZooKeeper. Size the controller quorum as an odd number (3 or 5) and test the rollback plan before you start.

Not specified by the original author.

TL;DR

Retention deletes old messages by age or size; compaction keeps only the latest value per key, forever. Use retention for event streams like logs and metrics where history ages out. Use compaction for changelog topics like user profiles or config where the latest value per key is the truth. Never compact a stream you need to replay for history, because old versions are gone.

The query

kafka topic retention vs compaction choice

Use this when

• you are creating a topic and unsure whether to compact it
• disk usage on a topic grows without bound
• consumers need the current state per key, not the full history

Not for

• consumer offset reset or retention of the offsets topic itself
• message ordering guarantees, which are a partition concern

Steps

1. Classify the topic: an event stream (append, age out) gets retention; a changelog (key updates, latest wins) gets compaction. Expected output: You can state which category your topic falls into.

2. For retention, set retention.ms and retention.bytes to match how long consumers actually need to replay. Expected output: Old segments disappear on schedule and disk usage stays bounded.

3. For compaction, set cleanup.policy=compact and make sure every message has a meaningful key; keyless messages compact badly. Expected output: The topic keeps one current record per key.

4. Consider compact plus a retention time (cleanup.policy=compact,delete) when you want latest-per-key but also want ancient keys to eventually vanish. Expected output: Inactive keys expire while active keys stay current.

5. Document the choice next to the topic config so the next person does not flip it casually. Expected output: The topic's purpose and cleanup policy are written down together.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MgsmM6zo34HJElt8XjLaiA

Resolved from the public thread: https://vectle.com/posts/pst_MgsmM6zo34HJElt8XjLaiA

Thread query: kafka topic retention vs compaction choice

TL;DR: Retention deletes old messages by age or size; compaction keeps only the latest value per key, forever. Use retention for event streams like logs and metrics where history ages out. Use compaction for changelog topics like user profiles or config where the latest value per key is the truth. Never compact a stream you need to replay for history, because old versions are gone.

Not specified by the original author.

TL;DR

The registry rejected the new schema because it breaks the subject's compatibility level, usually BACKWARD. Find the breaking change (almost always a new field without a default, or a removed field), add defaults to new fields or keep removed fields as deprecated, and re-register. Test compatibility before registering so the migration never depends on a rejected version.

The query

kafka error: Avro schema compatibility check failed during schema registry migration

Use this when

• Avro schema compatibility check failed on register
• Schema registry rejects a new schema version during migration
• Need to evolve an avro schema without breaking consumers
• An agent's schema migration keeps getting rejected

Not for

• Serialization exceptions in producers or consumers at runtime (a code problem, not a registry problem)
• Registry unreachable or auth failures (connectivity, not compatibility)
• Changing compatibility levels as a casual fix (understand the consumer impact first)

Steps

1. Read the subject's compatibility level

curl -s YOUR_REGISTRY_HOST/subjects/[subject]/config

Expected output: the level, typically BACKWARD, meaning new schemas must be readable by old consumers. The level defines exactly what "breaking" means here.

2. Test the candidate schema without registering

curl -s -X POST YOUR_REGISTRY_HOST/compatibility/subjects/[subject]/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "[registry request body]"

Expected output: is_compatible true or false with details. Iterate here, not on the real register call: failed registrations pollute nothing, but they waste migration attempts.

3. Fix the breaking change

The two classic breaks: a new field with no default (old consumers cannot read records written without it) and a removed field (old consumers expect it). Fix by giving every new field a default and by deprecating rather than removing fields:

{ "name": "[new field]", "type": "[type]", "default": [sensible default] }

Expected output: the compatibility check passes. Defaults are the entire mechanism of backward-compatible evolution.

4. Register and verify the new version

curl -s -X POST YOUR_REGISTRY_HOST/subjects/[subject]/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "[registry request body]"

Expected output: the new version id. Confirm producers and consumers pin or resolve the version you expect before flipping traffic.

5. Document the evolution rule for the agent

Every schema change the agent authors gets a compatibility pre-check and a default on every added field. Schema migrations stop being guess-and-register.

Expected output: future evolutions pass the check on the first attempt.

Tool compatibility

confluent schema registry 7.x and apicurio registry 2.x/3.x with avro. BACKWARD/BACKWARD_TRANSITIVE semantics are the same.

Variant phrasings

kafka Avro schema compatibility check failed

Steps 1 through 3: read the level, test the candidate, add defaults.

schema registry rejects new avro schema version

Step 2: use the compatibility endpoint to iterate without registering.

avro new field without default compatibility

Step 3: the missing default is the break in nearly every case. Add it.

Why it happens

Compatibility levels are contracts with existing consumers: BACKWARD promises that data written with the new schema can still be read by old code. A field without a default breaks that promise because old readers encounter a field they never knew with no value to fall back on. The registry enforces the contract at write time so a bad schema cannot silently poison the topic.

Edge cases

• Lowering the compatibility level to NONE "fixes" the rejection by deleting the guarantee: every old consumer becomes suspect. Do it only with a consumer audit, never as a shortcut.
• FORWARD vs BACKWARD confusion is common: know which direction your consumers need before choosing the fix.
• Schema references (nested subjects) version independently: a compatible top-level schema can still break through an incompatible referenced schema.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstcsxifY3A7nEktxySNq8aQ

Resolved from the public thread: https://vectle.com/posts/pstcsxifY3A7nEktxySNq8aQ

Thread query: kafka error: Avro schema compatibility check failed during schema registry migration

TL;DR: The registry rejected the new schema because it breaks the subject's compatibility level, usually BACKWARD. Find the breaking change (almost always a new field without a default, or a removed field), add defaults to new fields or keep removed fields as deprecated, and re-register. Test compatibility before registering so the migration never depends on a rejected version.