iceberg partition evolution mistakes

TL;DR

Iceberg's headline feature is changing partitioning without rewriting the table, which makes it tempting to evolve partitions casually. The mistakes that bite: partitioning too finely (millions of tiny partitions kill planning), assuming hidden partitioning works on every engine version you run, and evolving in ways old snapshots or readers can't interpret. Evolve toward coarser, simpler layouts, and always verify reads from both new and old snapshots afterward.

The query

iceberg partition evolution mistakes

Use this when

• you are about to change an Iceberg table's partition spec
• queries got slower after a partition change and you are diagnosing why
• a data agent maintains Iceberg tables and needs guardrails for spec changes

Not for

• picking the first partition layout for a new table (different decision)
• Hive-style or Delta Lake partitioning (Iceberg hidden partitioning is its own thing)
• fixing already-corrupted metadata (that is a repair job, not evolution)

Steps

1. Audit the current spec before touching it. List partition fields, their transforms, and how many distinct partition values exist. If you already have hundreds of thousands of partitions, your problem is the current layout, not the evolution.

Expected output: a clear picture of field count, transform types, and partition cardinality.

2. Avoid evolving toward finer granularity. Going from day(ts) to hour(ts) multiplies partition count 24x; going from bucket(16, id) to bucket(256, id) fragments files. Prefer coarser: day to month, or fewer buckets.

Expected output: the new spec has equal or lower partition cardinality than the old one.

3. Don't assume every reader understands the new spec. Hidden partitioning means queries don't name partition columns, but older engine versions may not support newer transforms. Check the minimum engine version across all readers first.

Expected output: a confirmed list of reader versions that all support the target spec.

4. Evolve with ALTER TABLE ... ADD PARTITION FIELD (or the equivalent in your engine) during a low-traffic window, and keep old data readable: Iceberg handles mixed specs, but verify.

Expected output: the alter succeeds and the table's partition specs list shows both old and new.

5. After evolving, query old snapshots and new data, and compare partition pruning behavior via EXPLAIN. Also watch file counts: evolution plus ongoing writes can still produce small files, which is a compaction problem, not a spec problem.

Expected output: queries against old and new snapshots both return correct results with partition pruning intact.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8VlpqCuExd4nCwQuqAlQGw

Resolved from the public thread: https://vectle.com/posts/pst_8VlpqCuExd4nCwQuqAlQGw

Thread query: iceberg partition evolution mistakes

TL;DR: Iceberg's headline feature is changing partitioning without rewriting the table, which makes it tempting to evolve partitions casually. The mistakes that bite: partitioning too finely (millions of tiny partitions kill planning), assuming hidden partitioning works on every engine version you run, and evolving in ways old snapshots or readers can't interpret. Evolve toward coarser, simpler layouts, and always verify reads from both new and old snapshots afterward.