agent wrote parquet with dictionary encoding mismatch, spark read failed

TL;DR

Spark reads a Parquet dataset as one logical table, and files written with different writer settings, dictionary encoding on for some string columns and off for others, or a different writer version, can produce footers and pages the reader trips on, especially with older Spark or when merging schemas. Identify the writer and its settings from the file footers, rewrite the affected files with one explicit writer configuration, dictionary encoding set deliberately per column type, and a consistent schema, then have Spark read the whole folder again.

agent wrote parquet with dictionary encoding mismatch, spark read failed

Use this when

• Spark fails reading a Parquet path that an agent or a script wrote, with an encoding or page decoding style error.
• Individual files read fine in one tool but the folder fails in Spark, or the reverse.
• Files in the folder were written by more than one job or library version.

Not for this skill when

• The error is a schema mismatch of different column names or types across files. That is schema drift; encoding is a separate layer.
• A single file is truncated or zero length from an interrupted write. Rewrite that file, encoding settings are not the story.

Steps

1. Read the exact Spark error and note the file it names and the decoding or encoding term it mentions. Note whether it fails on the first file or partway through the folder. Verify: the failing file and the encoding complaint are pinned down
2. Inspect the footers of a failing file and a working file with a Parquet metadata tool: writer version, encodings used per column, and dictionary page presence. Note the differences. Verify: the encoding difference is shown from the files themselves
3. Check the writing code and its library defaults: which writer produced the bad set, and whether dictionary encoding was left to per file heuristics based on cardinality. Note the settings that varied. Verify: the cause is traced to a writer configuration, not the reader
4. Rewrite the affected files with one explicit configuration: a stated writer version compatible with your Spark version, dictionary encoding on for low cardinality strings or off entirely for consistency, and the same schema for every file. Note the configuration in the writing code. Verify: every file in the dataset shares one writer profile
5. Read the full folder in Spark with schema merging off, run a count and a sample query per partition, and compare the count with the source records. Note the folder reads clean and counts agree. Verify: Spark reads the whole dataset, not just the repaired sample

Variant phrasings

spark parquet dictionary encoding fallback error reading files

The dictionary fallback phrasing.

parquet files written by different writer version spark read error

The writer version version of the problem.

Compatibility: Apache Spark reading Parquet written by common writers such as parquet-mr and Arrow based writers, on Spark 3.x. Encoding and writer version metadata are part of the Parquet format itself.

Why it happens

Parquet is self describing per file, but a dataset read assumes the files are siblings, and readers exercise encoding paths per column chunk. Dictionary encoding is chosen by writers based on the data they see, so two runs of the same job can make different choices for the same column when one file's values repeat and another's do not. Add a second writer library with different defaults and the folder becomes a mix no single read was tested against. Explicit writer settings turn an emergent per file choice into a dataset level contract.

Edge cases / pitfalls

• Turning dictionary encoding off everywhere costs file size and scan speed on string heavy data. Prefer consistent on for the columns that benefit, set in the writer, rather than a blanket off.
• Very old Spark versions lack support for newer encodings entirely. The compatible writer version ceiling is set by your oldest reader, including other tools.
• Do not mix repaired and original files in the same partition folder during the fix. Swap a partition atomically after verifying the new set.
• Other readers, such as a warehouse external table, may accept the folder Spark rejected. A file set that only one reader can read is still a liability.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xsAW3szL26oyuqhWJgtjFQ