agent's csv parser broke with encoding error on the contact export
Fixes CSV parsing crashes on contact exports with mixed encodings: detect the encoding, normalize to UTF-8, and parse defensively. Use when CSV reads fail on encoding. Not for delimiter or mapping issues.
TL;DR
The export mixes encodings (UTF-8, Windows-1252, stray BOMs), and the parser chokes on the first non-UTF-8 byte. Detect the encoding per file, decode to UTF-8, strip the BOM, and parse with errors handled explicitly. Never assume an export is clean UTF-8.
Error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 412: invalid start byteSteps
- Detect the encoding with a detection library on a sample of the file. Expected: you learn it is Windows-1252, UTF-8 with BOM, or mixed.
- Decode with the detected encoding and re-encode to UTF-8, stripping any BOM. Expected: a clean UTF-8 file.
- Parse with the CSV reader set to handle bad rows explicitly (skip or quarantine, never crash). Expected: the whole file parses.
- Quarantine rows that still fail and log their line numbers. Expected: a small quarantine file, not a dead pipeline.
- Make detection and normalization the first step of every import pipeline. Expected: encoding never breaks a run again.
When to use
- CSV parsing fails with UnicodeDecodeError.
- Exports from different sources break the same parser.
- An agent reads CRM exports and crashes on special characters.
When not to use
- Wrong delimiter (different parse error).
- Column mapping issues (parse succeeds, mapping fails).
Tool compatibility
- Python csv module and encoding detection libraries.
- Any ETL reading CRM exports.
Variant phrasings
invalid start byte on csv read
Mixed encoding; detect and normalize.
BOM breaking the header
Strip the BOM after decoding.
Why it happens
CRMs accept pasted text from Word, Outlook, and web forms in whatever encoding the client used. Exports faithfully reproduce the mess, and strict UTF-8 parsers die on it.
Edge cases
- Smart quotes and em dashes are the usual Windows-1252 culprits; they decode fine once the encoding is right.
- Mixed encodings in one file need per-chunk handling; detect on samples from multiple offsets.
- Writing the normalized file with a BOM reintroduces the problem downstream; write BOM-free.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_8iMQIfu4oFmVPchAmUOBNQ