agent mis-parsed csv header drift as a schema change error

TL;DR

CSV headers drift cosmetically: case changes, whitespace, reordered columns, renamed-but-equivalent labels. Normalize before comparing (trim, case-fold, map known aliases), and only flag structural changes: added, removed, or retyped columns. An agent that diffs raw header strings will cry schema change on every file.

agent mis-parsed csv header drift as a schema change error

Use this when

• An agent flags CSV header differences that are cosmetic.
• Column order or casing changed but the data is the same.
• Known alias labels trigger false schema alarms.

Not for this skill when

• Columns were really added or dropped. That is a schema change.
• The data under the headers changed type. That is a type problem.

Steps

1. Collect the flagged header diffs and classify each as cosmetic or structural. Note the classification. Verify: the noise is separated
2. Build normalization: trim whitespace, case-fold, apply the alias map. Note the rules. Verify: comparison is semantic
3. Re-run the comparison normalized and confirm the cosmetic diffs disappear. Note the clean diff. Verify: the noise is gone
4. Keep structural checks strict: added, removed, or retyped columns still alarm. Note the real guards. Verify: real changes still fire
5. Document the alias map where the team can see it. Note the documentation. Verify: the knowledge is shared

Variant phrasings

csv header normalization schema check

The technique phrasing.

csv column drift false positive

The symptom search.

Compatibility: CSV parsing generally. The pattern applies to any header-based ingestion.

Why it happens

CSV headers are human-written labels, not schema definitions: the source team renames a column for readability, exports with different casing, or reorders columns, and the data is identical. A raw string diff treats all of that as a breaking change. Normalization encodes what the team knows, that these labels mean the same thing, so the diff only fires on changes that actually alter the data contract. Without it, every file is an incident.

Edge cases / pitfalls

• An alias map that grows forever hides real renames. Review aliases when the source team changes.
• Case-folding hides case-sensitive column bugs on some warehouses. Normalize for comparison, but load the original names.
• A header row that shifts (extra row at the top) looks like total header drift. Detect header-row position first.
• BOM characters at the file start corrupt the first header name invisibly. Strip the BOM before comparing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FxUBpOYFfqlLhBnTsXAKLA