iceberg schema evolution failed error "cannot add required field without default value"
Give the new field a default, or add it as optional first: Iceberg refuses a required field with no default because existing files have no value for it, so add the field as optional, backfill it, then promote it to required if the format version allows. Use when an Iceberg table where a new non-null column is being added to a populated table. Not for a brand new empty table, where defining the field as required in
TL;DR
Give the new field a default, or add it as optional first: Iceberg refuses a required field with no default because existing files have no value for it, so add the field as optional, backfill it, then promote it to required if the format version allows. Iceberg validates schema evolution against existing data files. A required field without a default would make every existing row invalid the moment the field appears.
iceberg schema evolution failed error "cannot add required field without default value"Use this when
- The check or run fails with this symptom: an Iceberg schema evolution fails with cannot add required field without default value.
- You are working on an Iceberg table where a new non-null column is being added to a populated table and the failure repeats on the same data or configuration.
- You need the failing rows, rule, or stage isolated before changing thresholds or code.
Not for this skill when
- A brand new empty table, where defining the field as required in the initial schema is the right move.
- The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.
Steps
- Confirm the table already has data files, which is what triggers the rule. Verify: the table is populated, so the staged path applies
- Add the field as optional, or with an explicit default value where the engine supports initial defaults. Verify: the schema change succeeds
- Backfill the field for existing rows with the agreed value and verify no nulls remain. Verify: a count of null values in the new field returns zero
- Promote the field to required if the catalog and format support it, or enforce non-null in the writer and a data quality check. Verify: readers see the field populated on all rows
- Run a compaction or a metadata check after the change. Verify: the table reads cleanly with the new schema
Variant phrasings
iceberg cannot add required field without default value
The shorter phrasing focused on the named failure.
iceberg schema evolution required field failed
The symptom phrasing a searcher tries next.
add required column iceberg existing table fix
The variant naming the failing stage or tool directly.
Compatibility: Apache Iceberg format v2 and v3, Spark, Trino, and Flink writers
Why it happens
Iceberg schema evolution is designed so old files stay readable forever. Requiring a value that old files cannot supply breaks that guarantee, so the spec demands a default or an optional field followed by a backfill.
Edge cases / pitfalls
- Defaults in Iceberg apply on read for missing values in some engines and on write in others; verify how your engine surfaces the default before relying on it.
- Adding the field as optional and never backfilling leaves a permanent two-population column; track the backfill as part of the change.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_Nti59U6gaYm7XH3TBT9ECw