etl agent wrote parquet with wrong schema: schema evolution fix

TL;DR

The agent wrote parquet files whose schema drifted (renamed columns, changed types) without any versioning, so readers expecting the old schema break. Fix it by validating the written schema against a declared contract before upload, versioning schema changes explicitly, and using schema evolution rules (additive only) for readers.

etl agent wrote parquet with wrong schema: schema evolution fix

Use this when

• Downstream parquet reads fail on schema mismatch
• An agent changed columns mid-pipeline without telling anyone
• Files in the same dataset have different schemas

Not for this skill when

• The file is corrupt (thats the writer crashing)
• Partitioning is wrong (thats the layout)
• Compression misbehaves (thats the codec)

Steps

1. Inspect what was actually written:

import pyarrow.parquet as pq
print(pq.read_schema("output/part-0.parquet"))

Expected output: the real schema. Compare it field by field against the expected contract; the diff is the bug.

2. Add a schema assertion to the agent's write step:

expected = pa.schema([("id", pa.int64()), ("amount", pa.float64())])
table = table.cast(expected)  # raises if incompatible
pq.write_table(table, path)

Expected output: the write fails fast with a clear cast error instead of producing a subtly wrong file. Fail at write time, not at read time three stages later.

3. Version schema changes instead of silently drifting:

schemas/orders/v1.schema \u2192 schemas/orders/v2.schema (added: coupon_code)

Expected output: every file declares its version (in the path or metadata). Readers select the version they understand; nothing breaks silently.

4. For readers over mixed versions, evolve additively:

# read with a unified schema; missing columns become null
table = pq.read_table("output/", schema=unified_schema)

Expected output: old and new files read together, new columns null for old files. Never rename or retype in place; additive changes only.

Variant phrasings

one bad file poisons a whole dataset read

Quarantine the file by its schema version, backfill it correctly, and add the write-time assertion so it cant recur.

agent "fixed" the schema by changing the reader

Now the reader disagrees with every other consumer. Fix the writer; the reader was right.

Why it happens

Agents treat the write call as the end of their job and never validate what they wrote. Parquet is schema-ful, so any drift (a renamed column, an int that became a string) is baked into the file permanently. Without a declared contract and a write-time check, the first signal of a problem is a downstream failure far from the cause.

Edge cases

• Timestamps with and without timezones are different arrow types; unify before writing.
• Dictionary-encoded columns from different writers may have incompatible dictionaries; cast through a plain type first.
• Partition columns arent in the file schema; a partition value change looks like a schema change to naive readers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3P4AVqwOLfyCQmJExDjkMA