license agent's Okta CSV parser dropped rows with unicode names, audit undercounted
Fixes license audit undercounts when the CSV parser drops rows with unicode names. Use when the audit count is low and dropped rows contain non-ASCII names. Not for schema mismatches.
TL;DR
Read the CSV as UTF-8 with proper error handling. The parser drops rows because it decodes with the wrong codec, and names with non-ASCII characters are the casualties.
Error
(audit counted 900 of 1,000 users; dropped rows all have non-ASCII names; no parser error logged)Steps
- Confirm the pattern: check whether every dropped row has a non-ASCII character in the name. Expected: the unicode correlation established.
- Open the CSV with explicit UTF-8 decoding and inspect the previously dropped rows. Expected: the rows parse cleanly.
- Fix the agent's parser to specify UTF-8 (and handle the BOM if present) instead of relying on the platform default. Expected: the parser configured correctly.
- Re-run the parse and the audit. Expected: the full row count and correct seat totals.
When to use
- CSV parsing silently drops rows with unicode names
- The audit undercounts in regions with non-ASCII names
- The platform default encoding is not UTF-8
When not to use
- Rows are dropped for malformed quoting (a CSV dialect problem)
- The export itself is truncated (an export problem)
- The schema changed (a column-mapping problem)
Tool compatibility
Any CSV parsing in the audit agent (Python csv, pandas, etc.).
Variant phrasings
parser errors instead of dropping rows
Same root cause, louder symptom. Specify UTF-8.
BOM breaks the first column name
Strip the BOM or use UTF-8-SIG decoding.
Why it happens
CSV has no declared encoding, so parsers guess. The platform default (often ASCII or a locale codec) fails on non-ASCII bytes, and lenient parsers skip the row instead of raising.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_wYLgUMrLd5mlhCLkqzfGoA