## TL;DR
The agent wrote parquet files whose schema drifted (renamed columns, changed types) without any versioning, so readers expecting the old schema break. Fix it by validating the written schema against a declared contract before upload, versioning schema changes explicitly, and using schema evolution rules (additive only) for readers.

```text
etl agent wrote parquet with wrong schema: schema evolution fix
```

## Use this when
- Downstream parquet reads fail on schema mismatch
- An agent changed columns mid-pipeline without telling anyone
- Files in the same dataset have different schemas

## Not for this skill when
- The file is corrupt (thats the writer crashing)
- Partitioning is wrong (thats the layout)
- Compression misbehaves (thats the codec)

## Steps

1. Inspect what was actually written:

```python
import pyarrow.parquet as pq
print(pq.read_schema("output/part-0.parquet"))
```
Expected output: the real schema. Compare it field by field against the expected contract; the diff is the bug.

2. Add a schema assertion to the agent's write step:

```python
expected = pa.schema([("id", pa.int64()), ("amount", pa.float64())])
table = table.cast(expected)  # raises if incompatible
pq.write_table(table, path)
```
Expected output: the write fails fast with a clear cast error instead of producing a subtly wrong file. Fail at write time, not at read time three stages later.

3. Version schema changes instead of silently drifting:

```text
schemas/orders/v1.schema \u2192 schemas/orders/v2.schema (added: coupon_code)
```
Expected output: every file declares its version (in the path or metadata). Readers select the version they understand; nothing breaks silently.

4. For readers over mixed versions, evolve additively:

```python
# read with a unified schema; missing columns become null
table = pq.read_table("output/", schema=unified_schema)
```
Expected output: old and new files read together, new columns null for old files. Never rename or retype in place; additive changes only.

## Variant phrasings

### one bad file poisons a whole dataset read
Quarantine the file by its schema version, backfill it correctly, and add the write-time assertion so it cant recur.

### agent "fixed" the schema by changing the reader
Now the reader disagrees with every other consumer. Fix the writer; the reader was right.

## Why it happens
Agents treat the write call as the end of their job and never validate what they wrote. Parquet is schema-ful, so any drift (a renamed column, an int that became a string) is baked into the file permanently. Without a declared contract and a write-time check, the first signal of a problem is a downstream failure far from the cause.

## Edge cases
- Timestamps with and without timezones are different arrow types; unify before writing.
- Dictionary-encoded columns from different writers may have incompatible dictionaries; cast through a plain type first.
- Partition columns arent in the file schema; a partition value change looks like a schema change to naive readers.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3P4AVqwOLfyCQmJExDjkMA