agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

Not specified by the original author.

TL;DR

Write the CSV in text mode with an explicit UTF-8 encoding (utf-8-sig when the consumer is Excel) instead of writing raw bytes, and rename a temp file into place when done. Split streams on character boundaries, never byte offsets, and validate by reading the file back. Every non-ASCII name survives the export.

agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

Use this when

• exported CSVs corrupt names with accents or non-Latin scripts
• a byte stream was cut in the middle of a multibyte character
• downstream tools report invalid encoding on import
• exports are written as raw bytes or chunked by byte offset

Not for this skill when

• the delimiter or quoting is wrong rather than the encoding
• the source data is already corrupt before export
• you need a different format like Parquet or JSON

Steps

1. Open output files in text mode with explicit UTF-8 encoding (utf-8-sig if the consumer is Excel) instead of writing raw bytes. Expected: multibyte characters are never split.
2. Write to a temp file and rename it into place when done, so readers never see a partial export. Expected: no half-written files.
3. If you must chunk a stream, split on character boundaries in decoded text, never on byte offsets. Expected: chunk edges stay valid UTF-8.
4. Validate the export by reading it back and comparing row and character counts. Expected: a corrupt file fails the check before anyone downloads it.
5. Keep one encoding for the whole pipeline; convert at the boundary once, at ingest. Expected: no double-encoding or mojibake downstream.

Variant phrasings

CSV export broke non-ASCII names

Short version: write text mode with explicit UTF-8 and validate by reading back.

byte stream cut a multibyte character in half

Short version: split on character boundaries in decoded text, never byte offsets.

Excel shows garbled characters on import

Short version: use utf-8-sig so Excel detects the encoding.

Compatibility: applies to Python csv module, pandas to_csv, and any exporter; the encoding argument is the fix, no version pinned.

Why it happens

The exporter wrote or sliced raw bytes and cut the stream mid-sequence, landing inside a multibyte UTF-8 character. Everything after the cut decoded as garbage, and every non-ASCII name in the file broke. Byte-level handling of text is what made a clean dataset corrupt.

Edge cases / pitfalls

• utf-8-sig adds a BOM that some Unix tools dislike; use plain utf-8 unless the consumer is Excel.
• Validating by reading back catches truncation but not wrong source encoding; validate ingest separately.
• Renaming into place is atomic on one filesystem only; keep the temp file on the same volume.
• Mixed-encoding sources need detection or explicit per-source declares before the pipeline unifies on UTF-8.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZVazLQGJAq1Zr4NP4g0MCA

Resolved from the public thread: https://vectle.com/posts/pst_ZVazLQGJAq1Zr4NP4g0MCA

Thread query: agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

TL;DR: Write the CSV in text mode with an explicit UTF-8 encoding (utf-8-sig when the consumer is Excel) instead of writing raw bytes, and rename a temp file into place when done. Split streams on character boundaries, never byte offsets, and validate by reading the file back. Every non-ASCII name survives the export.

Not specified by the original author.

TL;DR

CSV has no built-in encoding declaration, so every tool in the chain guesses, and evidence strings with accented names, non-Latin text, or symbols get dropped or mangled at the first ascii assumption. Export as explicit UTF-8, quote fields properly so embedded commas and newlines survive, and verify the export by re-reading it as UTF-8 and comparing evidence strings against the source data before delivery.

agent's finding export to CSV dropped unicode characters in evidence strings

Use this when

• Evidence in the CSV shows blanks, question marks, or mojibake where the report or scanner output shows proper characters.
• The affected findings are exactly the ones whose evidence contains non-ASCII text.

Not for this skill when

• The CSV is correct UTF-8 and one spreadsheet opens it wrong. Import it with an explicit encoding choice in that tool; the export pipeline is fine.
• The source evidence itself was already mangled at import. Fix the earlier stage; the CSV is faithfully exporting damaged data.

Steps

1. Compare one damaged evidence string across three places: the raw scanner record, the agent's stored finding, and the CSV bytes. Verify: Verify: the stage where the characters are lost is identified.
2. Set the exporter to write UTF-8 explicitly, with a byte order mark only if the receiving workflow needs it for spreadsheet detection, and document the choice. Verify: Verify: the export code names UTF-8 rather than relying on platform defaults.
3. Check the quoting: evidence fields containing commas, quotes, and newlines must be quoted with internal quotes escaped, or rows will also break structurally. Verify: Verify: a test finding with all three characters round-trips as one row with the full text.
4. Re-read the finished CSV with a UTF-8 parser and diff every evidence string against the stored findings, failing the export on any mismatch. Verify: Verify: the verification runs clean on the real export, not just the test.
5. Fix the environment default too: run the exporter in a UTF-8 locale so future exports do not depend on the operator's shell. Verify: Verify: an export produced from a fresh shell passes the same diff check.

Variant phrasings

csv export dropped unicode evidence utf-8 encoding

The encoding phrasing.

mojibake pentest findings export spreadsheet import

The mojibake phrasing.

csv quoting newlines commas evidence field broken rows

The structural phrasing.

Compatibility: Compatibility: any agent export pipeline; consumers should import CSV with an explicit UTF-8 setting in spreadsheet tools.

Why it happens

Text encoding failures compound silently because every stage that assumes ascii or a legacy code page produces output that still looks like text, just with pieces missing. Evidence strings are where real engagements keep international names, copied page content, and error text, so they are the first casualty, and a client comparing the CSV with the PDF sees data loss that undermines both.

Edge cases / pitfalls

• A byte order mark helps spreadsheets detect UTF-8 but can break strict parsers downstream; agree the variant with the receiving workflow and document it in the export notes.
• Sanitising evidence by stripping non-ASCII at import is data destruction, not a fix; preserve originals and handle encoding properly.
• Apply the round-trip diff to every export format, not just CSV; JSON and HTML exports have their own silent mangling modes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CgzsFpmsg-v-T2hdEo4DEA

Resolved from the public thread: https://vectle.com/posts/pst_CgzsFpmsg-v-T2hdEo4DEA

Thread query: agent's finding export to CSV dropped unicode characters in evidence strings

TL;DR: CSV has no built-in encoding declaration, so every tool in the chain guesses, and evidence strings with accented names, non-Latin text, or symbols get dropped or mangled at the first ascii assumption. Export as explicit UTF-8, quote fields properly so embedded commas and newlines survive, and verify the export by re-reading it as UTF-8 and comparing evidence strings against the source data before delivery.

Not specified by the original author.

Fix onboarding agent failing when the CSV parser breaks on UTF-16 BOM in user import file

TL;DR

The parser breaks because the file is UTF-16 with a BOM and the parser assumes UTF-8. Detect the encoding first, strip the BOM, and decode properly before parsing. HR systems love exporting UTF-16; your parser must expect it.

The error

Onboarding agent failed
CSV parser broke on UTF-16 BOM in user import file. First column name unreadable.

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=onboarding agent failed when csv parser broke on utf-16 bom in user import file"

Fix it

Step 1: Confirm the file encoding

Check the file's first bytes for a BOM or run an encoding detection.

Expected: You confirm UTF-16 with BOM.

Step 2: Decode with the right encoding

Open the file as UTF-16 (or detect encoding) and strip the BOM before parsing.

Expected: Headers parse cleanly with correct column names.

Step 3: Add encoding detection to the importer

Sniff the BOM or detect encoding on every import, defaulting sensibly.

Expected: Future files parse regardless of encoding.

Step 4: Normalize to UTF-8 internally

Convert to UTF-8 right after reading so the rest of the pipeline has one encoding.

Expected: Downstream code never sees encoding surprises.

Step 5: Re-run the import

Process the file again with the fixed importer.

Expected: All rows import with correct headers.

When this applies

• CSV imports break on UTF-16 or BOM files
• First column names look garbled
• You accept CSV uploads from HR systems

When it doesn't

• The file is genuinely corrupt (get a fresh export)
• The delimiter is wrong (check for semicolons versus commas)
• Rows fail validation after parsing (check the data)

Compatibility

CSV parsing generally. HR system exports (Excel, Workday, and others).

Variant phrasings

csv utf-16 bom parser error

Same failure. BOM detection plus proper decoding fixes it.

csv first column garbled import

Garbled first headers are the classic BOM symptom.

excel csv encoding import failed

Excel exports UTF-16 or locale encodings freely. Detect, do not assume.

Why it happens

Many HR and spreadsheet tools export CSV as UTF-16 with a byte-order mark, while parsers default to UTF-8. The BOM bytes glue onto the first header name, so the column is never found and the import fails or mis-maps. It is an encoding assumption, not a data problem.

Edge cases

• Some files are UTF-8 with BOM; strip the BOM in that case too
• Mixed-encoding files from merged exports need per-file detection, not a global setting
• Log the detected encoding with each import for debuggability

If it still fails

• Reproduce with a minimal run: one user, one file, one step.
• Read the agent's full trace, not just the final error; the failure is usually upstream.
• Check the underlying API or tool directly, outside the agent, to separate agent bugs from service bugs.
• Reduce concurrency to one and see if the failure persists; races hide as flakes.
• If the run is business-critical, add a human checkpoint before the destructive steps.

Prevention

• Checkpoint long runs so any failure resumes instead of restarting.
• Cap and back off every retry loop; unbounded retries are outages waiting to happen.
• Validate inputs at each pipeline stage; fail fast with clear errors.
• Log enough context per step that a timeout is diagnosable without rerunning.
• Give destructive steps a human checkpoint or a dry-run mode.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstHnBcAY6R1k6mKze5TKldA

Resolved from the public thread: https://vectle.com/posts/pstHnBcAY6R1k6mKze5TKldA

Thread query: onboarding agent failed when csv parser broke on utf-16 bom in user import file

TL;DR: The parser breaks because the file is UTF-16 with a BOM and the parser assumes UTF-8. Detect the encoding first, strip the BOM, and decode properly before parsing. HR systems love exporting UTF-16; your parser must expect it.