agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

TL;DR

Write the CSV in text mode with an explicit UTF-8 encoding (utf-8-sig when the consumer is Excel) instead of writing raw bytes, and rename a temp file into place when done. Split streams on character boundaries, never byte offsets, and validate by reading the file back. Every non-ASCII name survives the export.

agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

Use this when

• exported CSVs corrupt names with accents or non-Latin scripts
• a byte stream was cut in the middle of a multibyte character
• downstream tools report invalid encoding on import
• exports are written as raw bytes or chunked by byte offset

Not for this skill when

• the delimiter or quoting is wrong rather than the encoding
• the source data is already corrupt before export
• you need a different format like Parquet or JSON

Steps

1. Open output files in text mode with explicit UTF-8 encoding (utf-8-sig if the consumer is Excel) instead of writing raw bytes. Expected: multibyte characters are never split.
2. Write to a temp file and rename it into place when done, so readers never see a partial export. Expected: no half-written files.
3. If you must chunk a stream, split on character boundaries in decoded text, never on byte offsets. Expected: chunk edges stay valid UTF-8.
4. Validate the export by reading it back and comparing row and character counts. Expected: a corrupt file fails the check before anyone downloads it.
5. Keep one encoding for the whole pipeline; convert at the boundary once, at ingest. Expected: no double-encoding or mojibake downstream.

Variant phrasings

CSV export broke non-ASCII names

Short version: write text mode with explicit UTF-8 and validate by reading back.

byte stream cut a multibyte character in half

Short version: split on character boundaries in decoded text, never byte offsets.

Excel shows garbled characters on import

Short version: use utf-8-sig so Excel detects the encoding.

Compatibility: applies to Python csv module, pandas to_csv, and any exporter; the encoding argument is the fix, no version pinned.

Why it happens

The exporter wrote or sliced raw bytes and cut the stream mid-sequence, landing inside a multibyte UTF-8 character. Everything after the cut decoded as garbage, and every non-ASCII name in the file broke. Byte-level handling of text is what made a clean dataset corrupt.

Edge cases / pitfalls

• utf-8-sig adds a BOM that some Unix tools dislike; use plain utf-8 unless the consumer is Excel.
• Validating by reading back catches truncation but not wrong source encoding; validate ingest separately.
• Renaming into place is atomic on one filesystem only; keep the temp file on the same volume.
• Mixed-encoding sources need detection or explicit per-source declares before the pipeline unifies on UTF-8.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZVazLQGJAq1Zr4NP4g0MCA