agent's finding export to CSV dropped unicode characters in evidence strings

TL;DR

CSV has no built-in encoding declaration, so every tool in the chain guesses, and evidence strings with accented names, non-Latin text, or symbols get dropped or mangled at the first ascii assumption. Export as explicit UTF-8, quote fields properly so embedded commas and newlines survive, and verify the export by re-reading it as UTF-8 and comparing evidence strings against the source data before delivery.

agent's finding export to CSV dropped unicode characters in evidence strings

Use this when

• Evidence in the CSV shows blanks, question marks, or mojibake where the report or scanner output shows proper characters.
• The affected findings are exactly the ones whose evidence contains non-ASCII text.

Not for this skill when

• The CSV is correct UTF-8 and one spreadsheet opens it wrong. Import it with an explicit encoding choice in that tool; the export pipeline is fine.
• The source evidence itself was already mangled at import. Fix the earlier stage; the CSV is faithfully exporting damaged data.

Steps

1. Compare one damaged evidence string across three places: the raw scanner record, the agent's stored finding, and the CSV bytes. Verify: Verify: the stage where the characters are lost is identified.
2. Set the exporter to write UTF-8 explicitly, with a byte order mark only if the receiving workflow needs it for spreadsheet detection, and document the choice. Verify: Verify: the export code names UTF-8 rather than relying on platform defaults.
3. Check the quoting: evidence fields containing commas, quotes, and newlines must be quoted with internal quotes escaped, or rows will also break structurally. Verify: Verify: a test finding with all three characters round-trips as one row with the full text.
4. Re-read the finished CSV with a UTF-8 parser and diff every evidence string against the stored findings, failing the export on any mismatch. Verify: Verify: the verification runs clean on the real export, not just the test.
5. Fix the environment default too: run the exporter in a UTF-8 locale so future exports do not depend on the operator's shell. Verify: Verify: an export produced from a fresh shell passes the same diff check.

Variant phrasings

csv export dropped unicode evidence utf-8 encoding

The encoding phrasing.

mojibake pentest findings export spreadsheet import

The mojibake phrasing.

csv quoting newlines commas evidence field broken rows

The structural phrasing.

Compatibility: Compatibility: any agent export pipeline; consumers should import CSV with an explicit UTF-8 setting in spreadsheet tools.

Why it happens

Text encoding failures compound silently because every stage that assumes ascii or a legacy code page produces output that still looks like text, just with pieces missing. Evidence strings are where real engagements keep international names, copied page content, and error text, so they are the first casualty, and a client comparing the CSV with the PDF sees data loss that undermines both.

Edge cases / pitfalls

• A byte order mark helps spreadsheets detect UTF-8 but can break strict parsers downstream; agree the variant with the receiving workflow and document it in the export notes.
• Sanitising evidence by stripping non-ASCII at import is data destruction, not a fix; preserve originals and handle encoding properly.
• Apply the round-trip diff to every export format, not just CSV; JSON and HTML exports have their own silent mangling modes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CgzsFpmsg-v-T2hdEo4DEA