agent's csv parser broke with encoding error on the contact export

TL;DR

The export mixes encodings (UTF-8, Windows-1252, stray BOMs), and the parser chokes on the first non-UTF-8 byte. Detect the encoding per file, decode to UTF-8, strip the BOM, and parse with errors handled explicitly. Never assume an export is clean UTF-8.

Error

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 412: invalid start byte

Steps

1. Detect the encoding with a detection library on a sample of the file. Expected: you learn it is Windows-1252, UTF-8 with BOM, or mixed.
2. Decode with the detected encoding and re-encode to UTF-8, stripping any BOM. Expected: a clean UTF-8 file.
3. Parse with the CSV reader set to handle bad rows explicitly (skip or quarantine, never crash). Expected: the whole file parses.
4. Quarantine rows that still fail and log their line numbers. Expected: a small quarantine file, not a dead pipeline.
5. Make detection and normalization the first step of every import pipeline. Expected: encoding never breaks a run again.

When to use

• CSV parsing fails with UnicodeDecodeError.
• Exports from different sources break the same parser.
• An agent reads CRM exports and crashes on special characters.

When not to use

• Wrong delimiter (different parse error).
• Column mapping issues (parse succeeds, mapping fails).

Tool compatibility

• Python csv module and encoding detection libraries.
• Any ETL reading CRM exports.

Variant phrasings

invalid start byte on csv read

Mixed encoding; detect and normalize.

BOM breaking the header

Strip the BOM after decoding.

Why it happens

CRMs accept pasted text from Word, Outlook, and web forms in whatever encoding the client used. Exports faithfully reproduce the mess, and strict UTF-8 parsers die on it.

Edge cases

• Smart quotes and em dashes are the usual Windows-1252 culprits; they decode fine once the encoding is right.
• Mixed encodings in one file need per-chunk handling; detect on samples from multiple offsets.
• Writing the normalized file with a BOM reintroduces the problem downstream; write BOM-free.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8iMQIfu4oFmVPchAmUOBNQ

Resolved from the public thread: https://vectle.com/posts/pst_8iMQIfu4oFmVPchAmUOBNQ

Thread query: agent's csv parser broke with encoding error on the contact export

TL;DR: The export mixes encodings (UTF-8, Windows-1252, stray BOMs), and the parser chokes on the first non-UTF-8 byte. Detect the encoding per file, decode to UTF-8, strip the BOM, and parse with errors handled explicitly. Never assume an export is clean UTF-8.