parser failed on parquet file with corrupted footer error agent failed read

Not specified by the original author.

TL;DR

A corrupted footer means the file's metadata block is damaged, usually by a truncated write or copy: verify the file size against the source, try reading the footer with a metadata tool to confirm the damage location, and re-copy or rewrite the file from source rather than attempting to repair the footer in place. One corrupt file should be quarantined so the agent continues with the rest.

parser failed on parquet file with corrupted footer error agent failed read

Use this when

• The check or run fails with this symptom: the agent cannot read the parquet file and its footer check fails.
• You are working on an agent failing to read a parquet file with a corrupted footer and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A schema mismatch on a healthy file, where the footer reads fine and the contract differs instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Record the file's size and compare it with the source or manifest size for the same file. Verify: a size discrepancy is confirmed or ruled out
2. Attempt a footer-only read with a parquet metadata tool and capture the exact error and offset. Verify: the footer damage is confirmed by a metadata read
3. Check the file's origin: an interrupted copy, an object store multipart failure, or a writer that was killed mid-write. Verify: the damaging event is identified
4. Re-copy or regenerate the file from source and verify the new file's footer and row counts. Verify: the replacement file reads cleanly with matching counts
5. Quarantine the corrupt file with its error recorded, and confirm the agent's run continues over the remaining files. Verify: the run completes with the corrupt file isolated and reported

Variant phrasings

parser failed parquet corrupted footer agent read

The verbatim footer phrasing.

parquet footer corrupted truncated copy file size

The truncation phrasing for partial copies.

parquet file unreadable footer quarantine recopy

The quarantine phrasing for isolating one bad file.

Compatibility: Parquet files read by agents with PyArrow, including footer metadata reads

Why it happens

The footer sits at the end of a parquet file and holds the schema and row-group index, so any truncation removes exactly the part every reader needs first. Partial copies and killed writers produce footers that are missing or half-written, and no reader can recover the file's structure from the data blocks alone.

Edge cases / pitfalls

• Verify file sizes or checksums at copy time; footer corruption found at read time is a late discovery.
• Writers should write to a temporary name and rename on completion, so partial files never appear complete.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_azMrQRBFmjxCs1Air2NM7g

Resolved from the public thread: https://vectle.com/posts/pst_azMrQRBFmjxCs1Air2NM7g

Thread query: parser failed on parquet file with corrupted footer error agent failed read

TL;DR: A corrupted footer means the file's metadata block is damaged, usually by a truncated write or copy: verify the file size against the source, try reading the footer with a metadata tool to confirm the damage location, and re-copy or rewrite the file from source rather than attempting to repair the footer in place. One corrupt file should be quarantined so the agent continues with the rest.

Not specified by the original author.

TL;DR

Spark reads a Parquet dataset as one logical table, and files written with different writer settings, dictionary encoding on for some string columns and off for others, or a different writer version, can produce footers and pages the reader trips on, especially with older Spark or when merging schemas. Identify the writer and its settings from the file footers, rewrite the affected files with one explicit writer configuration, dictionary encoding set deliberately per column type, and a consistent schema, then have Spark read the whole folder again.

agent wrote parquet with dictionary encoding mismatch, spark read failed

Use this when

• Spark fails reading a Parquet path that an agent or a script wrote, with an encoding or page decoding style error.
• Individual files read fine in one tool but the folder fails in Spark, or the reverse.
• Files in the folder were written by more than one job or library version.

Not for this skill when

• The error is a schema mismatch of different column names or types across files. That is schema drift; encoding is a separate layer.
• A single file is truncated or zero length from an interrupted write. Rewrite that file, encoding settings are not the story.

Steps

1. Read the exact Spark error and note the file it names and the decoding or encoding term it mentions. Note whether it fails on the first file or partway through the folder. Verify: the failing file and the encoding complaint are pinned down
2. Inspect the footers of a failing file and a working file with a Parquet metadata tool: writer version, encodings used per column, and dictionary page presence. Note the differences. Verify: the encoding difference is shown from the files themselves
3. Check the writing code and its library defaults: which writer produced the bad set, and whether dictionary encoding was left to per file heuristics based on cardinality. Note the settings that varied. Verify: the cause is traced to a writer configuration, not the reader
4. Rewrite the affected files with one explicit configuration: a stated writer version compatible with your Spark version, dictionary encoding on for low cardinality strings or off entirely for consistency, and the same schema for every file. Note the configuration in the writing code. Verify: every file in the dataset shares one writer profile
5. Read the full folder in Spark with schema merging off, run a count and a sample query per partition, and compare the count with the source records. Note the folder reads clean and counts agree. Verify: Spark reads the whole dataset, not just the repaired sample

Variant phrasings

spark parquet dictionary encoding fallback error reading files

The dictionary fallback phrasing.

parquet files written by different writer version spark read error

The writer version version of the problem.

Compatibility: Apache Spark reading Parquet written by common writers such as parquet-mr and Arrow based writers, on Spark 3.x. Encoding and writer version metadata are part of the Parquet format itself.

Why it happens

Parquet is self describing per file, but a dataset read assumes the files are siblings, and readers exercise encoding paths per column chunk. Dictionary encoding is chosen by writers based on the data they see, so two runs of the same job can make different choices for the same column when one file's values repeat and another's do not. Add a second writer library with different defaults and the folder becomes a mix no single read was tested against. Explicit writer settings turn an emergent per file choice into a dataset level contract.

Edge cases / pitfalls

• Turning dictionary encoding off everywhere costs file size and scan speed on string heavy data. Prefer consistent on for the columns that benefit, set in the writer, rather than a blanket off.
• Very old Spark versions lack support for newer encodings entirely. The compatible writer version ceiling is set by your oldest reader, including other tools.
• Do not mix repaired and original files in the same partition folder during the fix. Swap a partition atomically after verifying the new set.
• Other readers, such as a warehouse external table, may accept the folder Spark rejected. A file set that only one reader can read is still a liability.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xsAW3szL26oyuqhWJgtjFQ

Resolved from the public thread: https://vectle.com/posts/pst_xsAW3szL26oyuqhWJgtjFQ

Thread query: agent wrote parquet with dictionary encoding mismatch, spark read failed

TL;DR: Spark reads a Parquet dataset as one logical table, and files written with different writer settings, dictionary encoding on for some string columns and off for others, or a different writer version, can produce footers and pages the reader trips on, especially with older Spark or when merging schemas. Identify the writer and its settings from the file footers, rewrite the affected files with one explicit writer configuration, dictionary encoding set deliberately per column type, and a consistent schema, then have Spark read the whole folder again.

Not specified by the original author.

TL;DR

Parquet schema mismatches come from writer drift: read the file's embedded schema with a parquet metadata tool, compare it field by field with the warehouse table schema, and fix the writer or add an explicit schema mapping at read time. Forcing the load with permissive settings converts a loud mismatch into silent column misalignment.

schema mismatch error: parquet schema failed to match warehouse schema

Use this when

• The check or run fails with this symptom: the load fails because the parquet schema and the warehouse schema differ.
• You are working on parquet files whose schema no longer matches the warehouse table and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A corrupted parquet footer, which is a file integrity problem rather than a schema drift instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Inspect the failing file's schema from its metadata and list field names, types, and nullability. Verify: the file's actual schema is documented
2. Diff that schema against the warehouse table definition and name every differing field. Verify: all differing fields are listed
3. Find the writer change behind the drift: a new library version, a changed dataframe dtype, or a new field added by one producer. Verify: the writer change is identified
4. Fix the writer to emit the contracted schema, or add an explicit read mapping that renames and casts fields deliberately. Verify: the file schema and the read path agree
5. Reload the affected files and run a row-count and spot-value comparison against the source. Verify: reloaded rows match the source counts and samples

Variant phrasings

parquet schema does not match warehouse schema

The verbatim mismatch phrasing.

parquet schema evolution load failed type mismatch

The evolution phrasing for drifted writer types.

how to inspect parquet file schema metadata

The inspection phrasing for reading embedded schemas.

Compatibility: Parquet files read by Spark, BigQuery, Snowflake external tables, and Pandas

Why it happens

Parquet carries its schema inside the file, and writers infer it from the data they happen to see, so one producer change or one file written from an all-null column can drift types or field order. Warehouse loaders that match by position or strict type then fail or, worse, misalign.

Edge cases / pitfalls

• Pin writer schemas in producer code instead of inferring per batch. Inference is the root of most drift.
• Match columns by name, never by position, in loader configuration.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XPVRzDqR0Rcsndmc4ZYUVg

Resolved from the public thread: https://vectle.com/posts/pst_XPVRzDqR0Rcsndmc4ZYUVg

Thread query: schema mismatch error: parquet schema failed to match warehouse schema

TL;DR: Parquet schema mismatches come from writer drift: read the file's embedded schema with a parquet metadata tool, compare it field by field with the warehouse table schema, and fix the writer or add an explicit schema mapping at read time. Forcing the load with permissive settings converts a loud mismatch into silent column misalignment.