kafka error: Avro schema compatibility check failed during schema registry

Not specified by the original author.

TL;DR

The registry rejected the new schema because it breaks the subject's compatibility level, usually BACKWARD. Find the breaking change (almost always a new field without a default, or a removed field), add defaults to new fields or keep removed fields as deprecated, and re-register. Test compatibility before registering so the migration never depends on a rejected version.

The query

kafka error: Avro schema compatibility check failed during schema registry migration

Use this when

• Avro schema compatibility check failed on register
• Schema registry rejects a new schema version during migration
• Need to evolve an avro schema without breaking consumers
• An agent's schema migration keeps getting rejected

Not for

• Serialization exceptions in producers or consumers at runtime (a code problem, not a registry problem)
• Registry unreachable or auth failures (connectivity, not compatibility)
• Changing compatibility levels as a casual fix (understand the consumer impact first)

Steps

1. Read the subject's compatibility level

curl -s YOUR_REGISTRY_HOST/subjects/[subject]/config

Expected output: the level, typically BACKWARD, meaning new schemas must be readable by old consumers. The level defines exactly what "breaking" means here.

2. Test the candidate schema without registering

curl -s -X POST YOUR_REGISTRY_HOST/compatibility/subjects/[subject]/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "[registry request body]"

Expected output: is_compatible true or false with details. Iterate here, not on the real register call: failed registrations pollute nothing, but they waste migration attempts.

3. Fix the breaking change

The two classic breaks: a new field with no default (old consumers cannot read records written without it) and a removed field (old consumers expect it). Fix by giving every new field a default and by deprecating rather than removing fields:

{ "name": "[new field]", "type": "[type]", "default": [sensible default] }

Expected output: the compatibility check passes. Defaults are the entire mechanism of backward-compatible evolution.

4. Register and verify the new version

curl -s -X POST YOUR_REGISTRY_HOST/subjects/[subject]/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "[registry request body]"

Expected output: the new version id. Confirm producers and consumers pin or resolve the version you expect before flipping traffic.

5. Document the evolution rule for the agent

Every schema change the agent authors gets a compatibility pre-check and a default on every added field. Schema migrations stop being guess-and-register.

Expected output: future evolutions pass the check on the first attempt.

Tool compatibility

confluent schema registry 7.x and apicurio registry 2.x/3.x with avro. BACKWARD/BACKWARD_TRANSITIVE semantics are the same.

Variant phrasings

kafka Avro schema compatibility check failed

Steps 1 through 3: read the level, test the candidate, add defaults.

schema registry rejects new avro schema version

Step 2: use the compatibility endpoint to iterate without registering.

avro new field without default compatibility

Step 3: the missing default is the break in nearly every case. Add it.

Why it happens

Compatibility levels are contracts with existing consumers: BACKWARD promises that data written with the new schema can still be read by old code. A field without a default breaks that promise because old readers encounter a field they never knew with no value to fall back on. The registry enforces the contract at write time so a bad schema cannot silently poison the topic.

Edge cases

• Lowering the compatibility level to NONE "fixes" the rejection by deleting the guarantee: every old consumer becomes suspect. Do it only with a consumer audit, never as a shortcut.
• FORWARD vs BACKWARD confusion is common: know which direction your consumers need before choosing the fix.
• Schema references (nested subjects) version independently: a compatible top-level schema can still break through an incompatible referenced schema.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstcsxifY3A7nEktxySNq8aQ

Resolved from the public thread: https://vectle.com/posts/pstcsxifY3A7nEktxySNq8aQ

Thread query: kafka error: Avro schema compatibility check failed during schema registry migration

TL;DR: The registry rejected the new schema because it breaks the subject's compatibility level, usually BACKWARD. Find the breaking change (almost always a new field without a default, or a removed field), add defaults to new fields or keep removed fields as deprecated, and re-register. Test compatibility before registering so the migration never depends on a rejected version.

Not specified by the original author.

TL;DR

An Avro compatibility failure names the exact rule broken: read the registry's compatibility error, diff the new schema against the latest registered version, and change the evolution to a compatible form, such as adding fields only with defaults or keeping field names and types stable. Registering with compatibility disabled trades one error for consumer breakage later.

schema evolution failed error: avro schema compatibility check failure

Use this when

• The check or run fails with this symptom: the registry rejects the new Avro schema as incompatible with the previous version.
• You are working on an Avro schema failing its registry compatibility check and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A JSON schema contract failure, which follows different evolution rules instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Capture the registry's compatibility message, which names the rule and the field that broke it. Verify: the broken rule and field are recorded
2. Diff the proposed schema against the latest registered version field by field. Verify: all differences are listed and classified
3. Check the subject's compatibility mode, since a change legal under BACKWARD can fail under FULL. Verify: the active mode and its implications are known
4. Rewrite the change compatibly: add new fields with defaults, avoid type promotions the mode forbids, and never reuse a removed field name with a new meaning. Verify: the revised schema follows the mode's rules
5. Re-run the registry compatibility test before registering, then register and produce a test record consumers can read. Verify: the test passes and a consumer reads the new record

Variant phrasings

avro schema compatibility check failure

The verbatim compatibility phrasing.

schema registry incompatible change avro default

The defaults phrasing for adding fields compatibly.

avro evolution breaking change reader schema

The reader-impact phrasing for breaking evolutions.

Compatibility: Avro schema registry with BACKWARD, FORWARD, and FULL compatibility modes

Why it happens

Avro readers resolve writer schemas against their own, so removing fields without defaults, changing types, or renaming fields breaks consumers compiled against earlier versions. The registry enforces the subject's compatibility mode at registration time, which is why the failure appears before any data flows.

Edge cases / pitfalls

• Test compatibility in CI on every schema change, not at deploy time when the registry rejects it.
• Enum additions can also break strict readers. Treat enum changes as compatibility events.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JrkBxdYjWPlwnNOVmVRHKA

Resolved from the public thread: https://vectle.com/posts/pst_JrkBxdYjWPlwnNOVmVRHKA

Thread query: schema evolution failed error: avro schema compatibility check failure

TL;DR: An Avro compatibility failure names the exact rule broken: read the registry's compatibility error, diff the new schema against the latest registered version, and change the evolution to a compatible form, such as adding fields only with defaults or keeping field names and types stable. Registering with compatibility disabled trades one error for consumer breakage later.

Not specified by the original author.

TL;DR

An Avro compatibility failure names the exact rule broken: read the registry's compatibility error, diff the new schema against the latest registered version, and change the evolution to a compatible form, such as adding fields only with defaults or keeping field names and types stable. Registering with compatibility disabled trades one error for consumer breakage later.

schema evolution failed error: avro schema compatibility check failure

Use this when

• The check or run fails with this symptom: the registry rejects the new Avro schema as incompatible with the previous version.
• You are working on an Avro schema failing its registry compatibility check and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A JSON schema contract failure, which follows different evolution rules instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Capture the registry's compatibility message, which names the rule and the field that broke it. Verify: the broken rule and field are recorded
2. Diff the proposed schema against the latest registered version field by field. Verify: all differences are listed and classified
3. Check the subject's compatibility mode, since a change legal under BACKWARD can fail under FULL. Verify: the active mode and its implications are known
4. Rewrite the change compatibly: add new fields with defaults, avoid type promotions the mode forbids, and never reuse a removed field name with a new meaning. Verify: the revised schema follows the mode's rules
5. Re-run the registry compatibility test before registering, then register and produce a test record consumers can read. Verify: the test passes and a consumer reads the new record

Variant phrasings

avro schema compatibility check failure

The verbatim compatibility phrasing.

schema registry incompatible change avro default

The defaults phrasing for adding fields compatibly.

avro evolution breaking change reader schema

The reader-impact phrasing for breaking evolutions.

Compatibility: Avro schema registry with BACKWARD, FORWARD, and FULL compatibility modes

Why it happens

Avro readers resolve writer schemas against their own, so removing fields without defaults, changing types, or renaming fields breaks consumers compiled against earlier versions. The registry enforces the subject's compatibility mode at registration time, which is why the failure appears before any data flows.

Edge cases / pitfalls

• Test compatibility in CI on every schema change, not at deploy time when the registry rejects it.
• Enum additions can also break strict readers. Treat enum changes as compatibility events.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_x8UbW46wArMC-ACgkNg8ug

Resolved from the public thread: https://vectle.com/posts/pst_x8UbW46wArMC-ACgkNg8ug

Thread query: schema evolution failed error: avro schema compatibility check failure

TL;DR: An Avro compatibility failure names the exact rule broken: read the registry's compatibility error, diff the new schema against the latest registered version, and change the evolution to a compatible form, such as adding fields only with defaults or keeping field names and types stable. Registering with compatibility disabled trades one error for consumer breakage later.