delta lake merge failed error "cannot change data type of existing column" on schema evolution

TL;DR

Change the column type through a replace, not a merge: Delta MERGE refuses to change an existing column type, so rewrite the table with the new type in a controlled replace, or land the widened values in a new column and migrate readers. Delta Lake protects existing column types during merge. A safe widening such as int to long still fails inside MERGE, because merge never rewrites the schema of matched files.

delta lake merge failed error "cannot change data type of existing column" on schema evolution

Use this when

• The check or run fails with this symptom: a Delta Lake merge fails with cannot change data type of existing column.
• You are working on a Delta table updated by MERGE whose source column type changed and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A narrowing change such as string to int, which needs value validation and cleansing rather than a schema rewrite.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the merge error and identify the column and the from and to types. Verify: the exact column and type pair are known
2. Check whether the change is a safe widening, int to long or float to double, or a real conversion. Verify: the change class decides the fix path
3. For a widening, rewrite the table once with the new schema outside the merge, using a replace or a create-or-replace from a typed select. Verify: the table schema shows the new type and row counts match before and after
4. For a conversion, add a new typed column, backfill it with a cast that surfaces bad values, and switch readers before dropping the old column. Verify: bad values are listed instead of silently nulled
5. Rerun the original merge. Verify: the merge completes with the schema already at the target type

Variant phrasings

delta lake merge cannot change data type of existing column

The shorter phrasing focused on the named failure.

delta merge schema evolution type change failed

The symptom phrasing a searcher tries next.

cannot change data type existing column delta fix

The variant naming the failing stage or tool directly.

Compatibility: Delta Lake 2.x and 3.x on Spark and Delta standalone writers

Why it happens

MERGE updates files in place and validates the source schema against the target schema strictly. Type changes are a schema evolution operation that must happen as a table rewrite, so a pipeline that only ever merges has no path to apply them.

Edge cases / pitfalls

• Enabling schema merge options on MERGE does not permit type changes; it only allows adding columns.
• A replace rewrite drops table history beyond the retention used for the rewrite; snapshot the table first if time travel matters.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pUXxZ-IlPRmyGBiUZdVLqg