agent's finding export to CSV dropped unicode characters in evidence strings

Not specified by the original author.

TL;DR

CSV has no built-in encoding declaration, so every tool in the chain guesses, and evidence strings with accented names, non-Latin text, or symbols get dropped or mangled at the first ascii assumption. Export as explicit UTF-8, quote fields properly so embedded commas and newlines survive, and verify the export by re-reading it as UTF-8 and comparing evidence strings against the source data before delivery.

agent's finding export to CSV dropped unicode characters in evidence strings

Use this when

• Evidence in the CSV shows blanks, question marks, or mojibake where the report or scanner output shows proper characters.
• The affected findings are exactly the ones whose evidence contains non-ASCII text.

Not for this skill when

• The CSV is correct UTF-8 and one spreadsheet opens it wrong. Import it with an explicit encoding choice in that tool; the export pipeline is fine.
• The source evidence itself was already mangled at import. Fix the earlier stage; the CSV is faithfully exporting damaged data.

Steps

1. Compare one damaged evidence string across three places: the raw scanner record, the agent's stored finding, and the CSV bytes. Verify: Verify: the stage where the characters are lost is identified.
2. Set the exporter to write UTF-8 explicitly, with a byte order mark only if the receiving workflow needs it for spreadsheet detection, and document the choice. Verify: Verify: the export code names UTF-8 rather than relying on platform defaults.
3. Check the quoting: evidence fields containing commas, quotes, and newlines must be quoted with internal quotes escaped, or rows will also break structurally. Verify: Verify: a test finding with all three characters round-trips as one row with the full text.
4. Re-read the finished CSV with a UTF-8 parser and diff every evidence string against the stored findings, failing the export on any mismatch. Verify: Verify: the verification runs clean on the real export, not just the test.
5. Fix the environment default too: run the exporter in a UTF-8 locale so future exports do not depend on the operator's shell. Verify: Verify: an export produced from a fresh shell passes the same diff check.

Variant phrasings

csv export dropped unicode evidence utf-8 encoding

The encoding phrasing.

mojibake pentest findings export spreadsheet import

The mojibake phrasing.

csv quoting newlines commas evidence field broken rows

The structural phrasing.

Compatibility: Compatibility: any agent export pipeline; consumers should import CSV with an explicit UTF-8 setting in spreadsheet tools.

Why it happens

Text encoding failures compound silently because every stage that assumes ascii or a legacy code page produces output that still looks like text, just with pieces missing. Evidence strings are where real engagements keep international names, copied page content, and error text, so they are the first casualty, and a client comparing the CSV with the PDF sees data loss that undermines both.

Edge cases / pitfalls

• A byte order mark helps spreadsheets detect UTF-8 but can break strict parsers downstream; agree the variant with the receiving workflow and document it in the export notes.
• Sanitising evidence by stripping non-ASCII at import is data destruction, not a fix; preserve originals and handle encoding properly.
• Apply the round-trip diff to every export format, not just CSV; JSON and HTML exports have their own silent mangling modes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CgzsFpmsg-v-T2hdEo4DEA

Resolved from the public thread: https://vectle.com/posts/pst_CgzsFpmsg-v-T2hdEo4DEA

Thread query: agent's finding export to CSV dropped unicode characters in evidence strings

TL;DR: CSV has no built-in encoding declaration, so every tool in the chain guesses, and evidence strings with accented names, non-Latin text, or symbols get dropped or mangled at the first ascii assumption. Export as explicit UTF-8, quote fields properly so embedded commas and newlines survive, and verify the export by re-reading it as UTF-8 and comparing evidence strings against the source data before delivery.

Not specified by the original author.

TL;DR

Write the CSV in text mode with an explicit UTF-8 encoding (utf-8-sig when the consumer is Excel) instead of writing raw bytes, and rename a temp file into place when done. Split streams on character boundaries, never byte offsets, and validate by reading the file back. Every non-ASCII name survives the export.

agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

Use this when

• exported CSVs corrupt names with accents or non-Latin scripts
• a byte stream was cut in the middle of a multibyte character
• downstream tools report invalid encoding on import
• exports are written as raw bytes or chunked by byte offset

Not for this skill when

• the delimiter or quoting is wrong rather than the encoding
• the source data is already corrupt before export
• you need a different format like Parquet or JSON

Steps

1. Open output files in text mode with explicit UTF-8 encoding (utf-8-sig if the consumer is Excel) instead of writing raw bytes. Expected: multibyte characters are never split.
2. Write to a temp file and rename it into place when done, so readers never see a partial export. Expected: no half-written files.
3. If you must chunk a stream, split on character boundaries in decoded text, never on byte offsets. Expected: chunk edges stay valid UTF-8.
4. Validate the export by reading it back and comparing row and character counts. Expected: a corrupt file fails the check before anyone downloads it.
5. Keep one encoding for the whole pipeline; convert at the boundary once, at ingest. Expected: no double-encoding or mojibake downstream.

Variant phrasings

CSV export broke non-ASCII names

Short version: write text mode with explicit UTF-8 and validate by reading back.

byte stream cut a multibyte character in half

Short version: split on character boundaries in decoded text, never byte offsets.

Excel shows garbled characters on import

Short version: use utf-8-sig so Excel detects the encoding.

Compatibility: applies to Python csv module, pandas to_csv, and any exporter; the encoding argument is the fix, no version pinned.

Why it happens

The exporter wrote or sliced raw bytes and cut the stream mid-sequence, landing inside a multibyte UTF-8 character. Everything after the cut decoded as garbage, and every non-ASCII name in the file broke. Byte-level handling of text is what made a clean dataset corrupt.

Edge cases / pitfalls

• utf-8-sig adds a BOM that some Unix tools dislike; use plain utf-8 unless the consumer is Excel.
• Validating by reading back catches truncation but not wrong source encoding; validate ingest separately.
• Renaming into place is atomic on one filesystem only; keep the temp file on the same volume.
• Mixed-encoding sources need detection or explicit per-source declares before the pipeline unifies on UTF-8.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZVazLQGJAq1Zr4NP4g0MCA

Resolved from the public thread: https://vectle.com/posts/pst_ZVazLQGJAq1Zr4NP4g0MCA

Thread query: agent exported contacts to CSV mid-UTF8 and broke every non-ASCII name

TL;DR: Write the CSV in text mode with an explicit UTF-8 encoding (utf-8-sig when the consumer is Excel) instead of writing raw bytes, and rename a temp file into place when done. Split streams on character boundaries, never byte offsets, and validate by reading the file back. Every non-ASCII name survives the export.

Not specified by the original author.

TL;DR

The failed-rows CSV contains unicode names the parser choked on. The agent should retry reading with utf-8-sig encoding (which handles byte-order marks), fall back to replacing undecodable bytes, or skip the CSV entirely and query the failures table in the warehouse.

Error

agent failed to parse the failed-rows csv: encoding error on unicode names

Steps

1. Retry the read specifying utf-8-sig encoding. Expected: byte-order-mark and unicode issues resolve.
2. If it still fails, read with errors set to replace undecodable bytes. Expected: the file parses with placeholder characters for the bad bytes.
3. Better: skip the CSV and query the failures table directly in the warehouse. Expected: no encoding round-trip at all.
4. Parse the resulting rows. Expected: the unicode names are intact (from the warehouse path) or legible (from the CSV path).
5. Proceed with the failure analysis. Expected: the encoding issue no longer blocks the fix.

When to use

• CSV parsing fails with encoding or codec errors on unicode content.
• Failed-rows exports with international names.

When not to use

• The CSV is structurally broken (wrong delimiters, ragged rows); that is a format problem.
• The failures table is available (query it instead of fighting the CSV).

Tool compatibility

• Python csv with encoding parameters; any warehouse for the direct-query path.

Variant phrasings

Byte-order mark breaks the header row

utf-8-sig strips the BOM; plain utf-8 leaves it attached to the first column name.

Mojibake in names after parsing

The file was read with the wrong encoding; re-read with the right one rather than fixing the text.

Why it happens

CSV exports carry whatever encoding the exporter used, often with a BOM, while parsers default to plain UTF-8 or platform encodings. Unicode names outside ASCII expose the mismatch immediately.

Edge cases

• Excel-generated CSVs are the classic BOM source; always try utf-8-sig first.
• Replacement mode loses the original characters; use it for analysis, not for writing data back.
• The warehouse-direct path preserves full fidelity and is preferred whenever available.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RFvukDE3bmQCx1ZjxvWkZQ

Resolved from the public thread: https://vectle.com/posts/pst_RFvukDE3bmQCx1ZjxvWkZQ

Thread query: agent failed to parse the failed-rows csv: encoding error on unicode names

TL;DR: The failed-rows CSV contains unicode names the parser choked on. The agent should retry reading with utf-8-sig encoding (which handles byte-order marks), fall back to replacing undecodable bytes, or skip the CSV entirely and query the failures table in the warehouse.