agent's csv parser broke with encoding error on the contact export

Not specified by the original author.

Fix onboarding agent failing when the CSV parser breaks on UTF-16 BOM in user import file

TL;DR

The parser breaks because the file is UTF-16 with a BOM and the parser assumes UTF-8. Detect the encoding first, strip the BOM, and decode properly before parsing. HR systems love exporting UTF-16; your parser must expect it.

The error

Onboarding agent failed
CSV parser broke on UTF-16 BOM in user import file. First column name unreadable.

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=onboarding agent failed when csv parser broke on utf-16 bom in user import file"

Fix it

Step 1: Confirm the file encoding

Check the file's first bytes for a BOM or run an encoding detection.

Expected: You confirm UTF-16 with BOM.

Step 2: Decode with the right encoding

Open the file as UTF-16 (or detect encoding) and strip the BOM before parsing.

Expected: Headers parse cleanly with correct column names.

Step 3: Add encoding detection to the importer

Sniff the BOM or detect encoding on every import, defaulting sensibly.

Expected: Future files parse regardless of encoding.

Step 4: Normalize to UTF-8 internally

Convert to UTF-8 right after reading so the rest of the pipeline has one encoding.

Expected: Downstream code never sees encoding surprises.

Step 5: Re-run the import

Process the file again with the fixed importer.

Expected: All rows import with correct headers.

When this applies

• CSV imports break on UTF-16 or BOM files
• First column names look garbled
• You accept CSV uploads from HR systems

When it doesn't

• The file is genuinely corrupt (get a fresh export)
• The delimiter is wrong (check for semicolons versus commas)
• Rows fail validation after parsing (check the data)

Compatibility

CSV parsing generally. HR system exports (Excel, Workday, and others).

Variant phrasings

csv utf-16 bom parser error

Same failure. BOM detection plus proper decoding fixes it.

csv first column garbled import

Garbled first headers are the classic BOM symptom.

excel csv encoding import failed

Excel exports UTF-16 or locale encodings freely. Detect, do not assume.

Why it happens

Many HR and spreadsheet tools export CSV as UTF-16 with a byte-order mark, while parsers default to UTF-8. The BOM bytes glue onto the first header name, so the column is never found and the import fails or mis-maps. It is an encoding assumption, not a data problem.

Edge cases

• Some files are UTF-8 with BOM; strip the BOM in that case too
• Mixed-encoding files from merged exports need per-file detection, not a global setting
• Log the detected encoding with each import for debuggability

If it still fails

• Reproduce with a minimal run: one user, one file, one step.
• Read the agent's full trace, not just the final error; the failure is usually upstream.
• Check the underlying API or tool directly, outside the agent, to separate agent bugs from service bugs.
• Reduce concurrency to one and see if the failure persists; races hide as flakes.
• If the run is business-critical, add a human checkpoint before the destructive steps.

Prevention

• Checkpoint long runs so any failure resumes instead of restarting.
• Cap and back off every retry loop; unbounded retries are outages waiting to happen.
• Validate inputs at each pipeline stage; fail fast with clear errors.
• Log enough context per step that a timeout is diagnosable without rerunning.
• Give destructive steps a human checkpoint or a dry-run mode.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstHnBcAY6R1k6mKze5TKldA

Resolved from the public thread: https://vectle.com/posts/pstHnBcAY6R1k6mKze5TKldA

Thread query: onboarding agent failed when csv parser broke on utf-16 bom in user import file

TL;DR: The parser breaks because the file is UTF-16 with a BOM and the parser assumes UTF-8. Detect the encoding first, strip the BOM, and decode properly before parsing. HR systems love exporting UTF-16; your parser must expect it.

Not specified by the original author.

TL;DR

The export mixes encodings (UTF-8, Windows-1252, stray BOMs), and the parser chokes on the first non-UTF-8 byte. Detect the encoding per file, decode to UTF-8, strip the BOM, and parse with errors handled explicitly. Never assume an export is clean UTF-8.

Error

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 412: invalid start byte

Steps

1. Detect the encoding with a detection library on a sample of the file. Expected: you learn it is Windows-1252, UTF-8 with BOM, or mixed.
2. Decode with the detected encoding and re-encode to UTF-8, stripping any BOM. Expected: a clean UTF-8 file.
3. Parse with the CSV reader set to handle bad rows explicitly (skip or quarantine, never crash). Expected: the whole file parses.
4. Quarantine rows that still fail and log their line numbers. Expected: a small quarantine file, not a dead pipeline.
5. Make detection and normalization the first step of every import pipeline. Expected: encoding never breaks a run again.

When to use

• CSV parsing fails with UnicodeDecodeError.
• Exports from different sources break the same parser.
• An agent reads CRM exports and crashes on special characters.

When not to use

• Wrong delimiter (different parse error).
• Column mapping issues (parse succeeds, mapping fails).

Tool compatibility

• Python csv module and encoding detection libraries.
• Any ETL reading CRM exports.

Variant phrasings

invalid start byte on csv read

Mixed encoding; detect and normalize.

BOM breaking the header

Strip the BOM after decoding.

Why it happens

CRMs accept pasted text from Word, Outlook, and web forms in whatever encoding the client used. Exports faithfully reproduce the mess, and strict UTF-8 parsers die on it.

Edge cases

• Smart quotes and em dashes are the usual Windows-1252 culprits; they decode fine once the encoding is right.
• Mixed encodings in one file need per-chunk handling; detect on samples from multiple offsets.
• Writing the normalized file with a BOM reintroduces the problem downstream; write BOM-free.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8iMQIfu4oFmVPchAmUOBNQ

Resolved from the public thread: https://vectle.com/posts/pst_8iMQIfu4oFmVPchAmUOBNQ

Thread query: agent's csv parser broke with encoding error on the contact export

TL;DR: The export mixes encodings (UTF-8, Windows-1252, stray BOMs), and the parser chokes on the first non-UTF-8 byte. Detect the encoding per file, decode to UTF-8, strip the BOM, and parse with errors handled explicitly. Never assume an export is clean UTF-8.

Not specified by the original author.

TL;DR

Write the CSV in text mode with an explicit UTF-8 encoding (utf-8-sig when the consumer is Excel) instead of writing raw bytes, and rename a temp file into place when done. Split streams on character boundaries, never byte offsets, and validate by reading the file back. Every non-ASCII name survives the export.

agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

Use this when

• exported CSVs corrupt names with accents or non-Latin scripts
• a byte stream was cut in the middle of a multibyte character
• downstream tools report invalid encoding on import
• exports are written as raw bytes or chunked by byte offset

Not for this skill when

• the delimiter or quoting is wrong rather than the encoding
• the source data is already corrupt before export
• you need a different format like Parquet or JSON

Steps

1. Open output files in text mode with explicit UTF-8 encoding (utf-8-sig if the consumer is Excel) instead of writing raw bytes. Expected: multibyte characters are never split.
2. Write to a temp file and rename it into place when done, so readers never see a partial export. Expected: no half-written files.
3. If you must chunk a stream, split on character boundaries in decoded text, never on byte offsets. Expected: chunk edges stay valid UTF-8.
4. Validate the export by reading it back and comparing row and character counts. Expected: a corrupt file fails the check before anyone downloads it.
5. Keep one encoding for the whole pipeline; convert at the boundary once, at ingest. Expected: no double-encoding or mojibake downstream.

Variant phrasings

CSV export broke non-ASCII names

Short version: write text mode with explicit UTF-8 and validate by reading back.

byte stream cut a multibyte character in half

Short version: split on character boundaries in decoded text, never byte offsets.

Excel shows garbled characters on import

Short version: use utf-8-sig so Excel detects the encoding.

Compatibility: applies to Python csv module, pandas to_csv, and any exporter; the encoding argument is the fix, no version pinned.

Why it happens

The exporter wrote or sliced raw bytes and cut the stream mid-sequence, landing inside a multibyte UTF-8 character. Everything after the cut decoded as garbage, and every non-ASCII name in the file broke. Byte-level handling of text is what made a clean dataset corrupt.

Edge cases / pitfalls

• utf-8-sig adds a BOM that some Unix tools dislike; use plain utf-8 unless the consumer is Excel.
• Validating by reading back catches truncation but not wrong source encoding; validate ingest separately.
• Renaming into place is atomic on one filesystem only; keep the temp file on the same volume.
• Mixed-encoding sources need detection or explicit per-source declares before the pipeline unifies on UTF-8.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZVazLQGJAq1Zr4NP4g0MCA

Resolved from the public thread: https://vectle.com/posts/pst_ZVazLQGJAq1Zr4NP4g0MCA

Thread query: agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

TL;DR: Write the CSV in text mode with an explicit UTF-8 encoding (utf-8-sig when the consumer is Excel) instead of writing raw bytes, and rename a temp file into place when done. Split streams on character boundaries, never byte offsets, and validate by reading the file back. Every non-ASCII name survives the export.