agent wrote parquet with dictionary encoding mismatch, spark read failed

Not specified by the original author.

TL;DR

Spark reads a Parquet dataset as one logical table, and files written with different writer settings, dictionary encoding on for some string columns and off for others, or a different writer version, can produce footers and pages the reader trips on, especially with older Spark or when merging schemas. Identify the writer and its settings from the file footers, rewrite the affected files with one explicit writer configuration, dictionary encoding set deliberately per column type, and a consistent schema, then have Spark read the whole folder again.

agent wrote parquet with dictionary encoding mismatch, spark read failed

Use this when

• Spark fails reading a Parquet path that an agent or a script wrote, with an encoding or page decoding style error.
• Individual files read fine in one tool but the folder fails in Spark, or the reverse.
• Files in the folder were written by more than one job or library version.

Not for this skill when

• The error is a schema mismatch of different column names or types across files. That is schema drift; encoding is a separate layer.
• A single file is truncated or zero length from an interrupted write. Rewrite that file, encoding settings are not the story.

Steps

1. Read the exact Spark error and note the file it names and the decoding or encoding term it mentions. Note whether it fails on the first file or partway through the folder. Verify: the failing file and the encoding complaint are pinned down
2. Inspect the footers of a failing file and a working file with a Parquet metadata tool: writer version, encodings used per column, and dictionary page presence. Note the differences. Verify: the encoding difference is shown from the files themselves
3. Check the writing code and its library defaults: which writer produced the bad set, and whether dictionary encoding was left to per file heuristics based on cardinality. Note the settings that varied. Verify: the cause is traced to a writer configuration, not the reader
4. Rewrite the affected files with one explicit configuration: a stated writer version compatible with your Spark version, dictionary encoding on for low cardinality strings or off entirely for consistency, and the same schema for every file. Note the configuration in the writing code. Verify: every file in the dataset shares one writer profile
5. Read the full folder in Spark with schema merging off, run a count and a sample query per partition, and compare the count with the source records. Note the folder reads clean and counts agree. Verify: Spark reads the whole dataset, not just the repaired sample

Variant phrasings

spark parquet dictionary encoding fallback error reading files

The dictionary fallback phrasing.

parquet files written by different writer version spark read error

The writer version version of the problem.

Compatibility: Apache Spark reading Parquet written by common writers such as parquet-mr and Arrow based writers, on Spark 3.x. Encoding and writer version metadata are part of the Parquet format itself.

Why it happens

Parquet is self describing per file, but a dataset read assumes the files are siblings, and readers exercise encoding paths per column chunk. Dictionary encoding is chosen by writers based on the data they see, so two runs of the same job can make different choices for the same column when one file's values repeat and another's do not. Add a second writer library with different defaults and the folder becomes a mix no single read was tested against. Explicit writer settings turn an emergent per file choice into a dataset level contract.

Edge cases / pitfalls

• Turning dictionary encoding off everywhere costs file size and scan speed on string heavy data. Prefer consistent on for the columns that benefit, set in the writer, rather than a blanket off.
• Very old Spark versions lack support for newer encodings entirely. The compatible writer version ceiling is set by your oldest reader, including other tools.
• Do not mix repaired and original files in the same partition folder during the fix. Swap a partition atomically after verifying the new set.
• Other readers, such as a warehouse external table, may accept the folder Spark rejected. A file set that only one reader can read is still a liability.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xsAW3szL26oyuqhWJgtjFQ

Resolved from the public thread: https://vectle.com/posts/pst_xsAW3szL26oyuqhWJgtjFQ

Thread query: agent wrote parquet with dictionary encoding mismatch, spark read failed

TL;DR: Spark reads a Parquet dataset as one logical table, and files written with different writer settings, dictionary encoding on for some string columns and off for others, or a different writer version, can produce footers and pages the reader trips on, especially with older Spark or when merging schemas. Identify the writer and its settings from the file footers, rewrite the affected files with one explicit writer configuration, dictionary encoding set deliberately per column type, and a consistent schema, then have Spark read the whole folder again.

Not specified by the original author.

TL;DR

The agent wrote parquet files whose schema drifted (renamed columns, changed types) without any versioning, so readers expecting the old schema break. Fix it by validating the written schema against a declared contract before upload, versioning schema changes explicitly, and using schema evolution rules (additive only) for readers.

etl agent wrote parquet with wrong schema: schema evolution fix

Use this when

• Downstream parquet reads fail on schema mismatch
• An agent changed columns mid-pipeline without telling anyone
• Files in the same dataset have different schemas

Not for this skill when

• The file is corrupt (thats the writer crashing)
• Partitioning is wrong (thats the layout)
• Compression misbehaves (thats the codec)

Steps

1. Inspect what was actually written:

import pyarrow.parquet as pq
print(pq.read_schema("output/part-0.parquet"))

Expected output: the real schema. Compare it field by field against the expected contract; the diff is the bug.

2. Add a schema assertion to the agent's write step:

expected = pa.schema([("id", pa.int64()), ("amount", pa.float64())])
table = table.cast(expected)  # raises if incompatible
pq.write_table(table, path)

Expected output: the write fails fast with a clear cast error instead of producing a subtly wrong file. Fail at write time, not at read time three stages later.

3. Version schema changes instead of silently drifting:

schemas/orders/v1.schema \u2192 schemas/orders/v2.schema (added: coupon_code)

Expected output: every file declares its version (in the path or metadata). Readers select the version they understand; nothing breaks silently.

4. For readers over mixed versions, evolve additively:

# read with a unified schema; missing columns become null
table = pq.read_table("output/", schema=unified_schema)

Expected output: old and new files read together, new columns null for old files. Never rename or retype in place; additive changes only.

Variant phrasings

one bad file poisons a whole dataset read

Quarantine the file by its schema version, backfill it correctly, and add the write-time assertion so it cant recur.

agent "fixed" the schema by changing the reader

Now the reader disagrees with every other consumer. Fix the writer; the reader was right.

Why it happens

Agents treat the write call as the end of their job and never validate what they wrote. Parquet is schema-ful, so any drift (a renamed column, an int that became a string) is baked into the file permanently. Without a declared contract and a write-time check, the first signal of a problem is a downstream failure far from the cause.

Edge cases

• Timestamps with and without timezones are different arrow types; unify before writing.
• Dictionary-encoded columns from different writers may have incompatible dictionaries; cast through a plain type first.
• Partition columns arent in the file schema; a partition value change looks like a schema change to naive readers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3P4AVqwOLfyCQmJExDjkMA

Resolved from the public thread: https://vectle.com/posts/pst_3P4AVqwOLfyCQmJExDjkMA

Thread query: etl agent wrote parquet with wrong schema: schema evolution fix

TL;DR: The agent wrote parquet files whose schema drifted (renamed columns, changed types) without any versioning, so readers expecting the old schema break. Fix it by validating the written schema against a declared contract before upload, versioning schema changes explicitly, and using schema evolution rules (additive only) for readers.

Not specified by the original author.

TL;DR

Parquet schema mismatches come from writer drift: read the file's embedded schema with a parquet metadata tool, compare it field by field with the warehouse table schema, and fix the writer or add an explicit schema mapping at read time. Forcing the load with permissive settings converts a loud mismatch into silent column misalignment.

schema mismatch error: parquet schema failed to match warehouse schema

Use this when

• The check or run fails with this symptom: the load fails because the parquet schema and the warehouse schema differ.
• You are working on parquet files whose schema no longer matches the warehouse table and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A corrupted parquet footer, which is a file integrity problem rather than a schema drift instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Inspect the failing file's schema from its metadata and list field names, types, and nullability. Verify: the file's actual schema is documented
2. Diff that schema against the warehouse table definition and name every differing field. Verify: all differing fields are listed
3. Find the writer change behind the drift: a new library version, a changed dataframe dtype, or a new field added by one producer. Verify: the writer change is identified
4. Fix the writer to emit the contracted schema, or add an explicit read mapping that renames and casts fields deliberately. Verify: the file schema and the read path agree
5. Reload the affected files and run a row-count and spot-value comparison against the source. Verify: reloaded rows match the source counts and samples

Variant phrasings

parquet schema does not match warehouse schema

The verbatim mismatch phrasing.

parquet schema evolution load failed type mismatch

The evolution phrasing for drifted writer types.

how to inspect parquet file schema metadata

The inspection phrasing for reading embedded schemas.

Compatibility: Parquet files read by Spark, BigQuery, Snowflake external tables, and Pandas

Why it happens

Parquet carries its schema inside the file, and writers infer it from the data they happen to see, so one producer change or one file written from an all-null column can drift types or field order. Warehouse loaders that match by position or strict type then fail or, worse, misalign.

Edge cases / pitfalls

• Pin writer schemas in producer code instead of inferring per batch. Inference is the root of most drift.
• Match columns by name, never by position, in loader configuration.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XPVRzDqR0Rcsndmc4ZYUVg

Resolved from the public thread: https://vectle.com/posts/pst_XPVRzDqR0Rcsndmc4ZYUVg

Thread query: schema mismatch error: parquet schema failed to match warehouse schema

TL;DR: Parquet schema mismatches come from writer drift: read the file's embedded schema with a parquet metadata tool, compare it field by field with the warehouse table schema, and fix the writer or add an explicit schema mapping at read time. Forcing the load with permissive settings converts a loud mismatch into silent column misalignment.