agent failed to parse the failed-rows csv: encoding error on unicode names

TL;DR

The failed-rows CSV contains unicode names the parser choked on. The agent should retry reading with utf-8-sig encoding (which handles byte-order marks), fall back to replacing undecodable bytes, or skip the CSV entirely and query the failures table in the warehouse.

Error

agent failed to parse the failed-rows csv: encoding error on unicode names

Steps

1. Retry the read specifying utf-8-sig encoding. Expected: byte-order-mark and unicode issues resolve.
2. If it still fails, read with errors set to replace undecodable bytes. Expected: the file parses with placeholder characters for the bad bytes.
3. Better: skip the CSV and query the failures table directly in the warehouse. Expected: no encoding round-trip at all.
4. Parse the resulting rows. Expected: the unicode names are intact (from the warehouse path) or legible (from the CSV path).
5. Proceed with the failure analysis. Expected: the encoding issue no longer blocks the fix.

When to use

• CSV parsing fails with encoding or codec errors on unicode content.
• Failed-rows exports with international names.

When not to use

• The CSV is structurally broken (wrong delimiters, ragged rows); that is a format problem.
• The failures table is available (query it instead of fighting the CSV).

Tool compatibility

• Python csv with encoding parameters; any warehouse for the direct-query path.

Variant phrasings

Byte-order mark breaks the header row

utf-8-sig strips the BOM; plain utf-8 leaves it attached to the first column name.

Mojibake in names after parsing

The file was read with the wrong encoding; re-read with the right one rather than fixing the text.

Why it happens

CSV exports carry whatever encoding the exporter used, often with a BOM, while parsers default to plain UTF-8 or platform encodings. Unicode names outside ASCII expose the mismatch immediately.

Edge cases

• Excel-generated CSVs are the classic BOM source; always try utf-8-sig first.
• Replacement mode loses the original characters; use it for analysis, not for writing data back.
• The warehouse-direct path preserves full fidelity and is preferred whenever available.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RFvukDE3bmQCx1ZjxvWkZQ