kafka error: Avro schema compatibility check failed during schema registry migration

TL;DR

The registry rejected the new schema because it breaks the subject's compatibility level, usually BACKWARD. Find the breaking change (almost always a new field without a default, or a removed field), add defaults to new fields or keep removed fields as deprecated, and re-register. Test compatibility before registering so the migration never depends on a rejected version.

The query

kafka error: Avro schema compatibility check failed during schema registry migration

Use this when

• Avro schema compatibility check failed on register
• Schema registry rejects a new schema version during migration
• Need to evolve an avro schema without breaking consumers
• An agent's schema migration keeps getting rejected

Not for

• Serialization exceptions in producers or consumers at runtime (a code problem, not a registry problem)
• Registry unreachable or auth failures (connectivity, not compatibility)
• Changing compatibility levels as a casual fix (understand the consumer impact first)

Steps

1. Read the subject's compatibility level

curl -s YOUR_REGISTRY_HOST/subjects/[subject]/config

Expected output: the level, typically BACKWARD, meaning new schemas must be readable by old consumers. The level defines exactly what "breaking" means here.

2. Test the candidate schema without registering

curl -s -X POST YOUR_REGISTRY_HOST/compatibility/subjects/[subject]/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "[registry request body]"

Expected output: is_compatible true or false with details. Iterate here, not on the real register call: failed registrations pollute nothing, but they waste migration attempts.

3. Fix the breaking change

The two classic breaks: a new field with no default (old consumers cannot read records written without it) and a removed field (old consumers expect it). Fix by giving every new field a default and by deprecating rather than removing fields:

{ "name": "[new field]", "type": "[type]", "default": [sensible default] }

Expected output: the compatibility check passes. Defaults are the entire mechanism of backward-compatible evolution.

4. Register and verify the new version

curl -s -X POST YOUR_REGISTRY_HOST/subjects/[subject]/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "[registry request body]"

Expected output: the new version id. Confirm producers and consumers pin or resolve the version you expect before flipping traffic.

5. Document the evolution rule for the agent

Every schema change the agent authors gets a compatibility pre-check and a default on every added field. Schema migrations stop being guess-and-register.

Expected output: future evolutions pass the check on the first attempt.

Tool compatibility

confluent schema registry 7.x and apicurio registry 2.x/3.x with avro. BACKWARD/BACKWARD_TRANSITIVE semantics are the same.

Variant phrasings

kafka Avro schema compatibility check failed

Steps 1 through 3: read the level, test the candidate, add defaults.

schema registry rejects new avro schema version

Step 2: use the compatibility endpoint to iterate without registering.

avro new field without default compatibility

Step 3: the missing default is the break in nearly every case. Add it.

Why it happens

Compatibility levels are contracts with existing consumers: BACKWARD promises that data written with the new schema can still be read by old code. A field without a default breaks that promise because old readers encounter a field they never knew with no value to fall back on. The registry enforces the contract at write time so a bad schema cannot silently poison the topic.

Edge cases

• Lowering the compatibility level to NONE "fixes" the rejection by deleting the guarantee: every old consumer becomes suspect. Do it only with a consumer audit, never as a shortcut.
• FORWARD vs BACKWARD confusion is common: know which direction your consumers need before choosing the fix.
• Schema references (nested subjects) version independently: a compatible top-level schema can still break through an incompatible referenced schema.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstcsxifY3A7nEktxySNq8aQ