# Fix onboarding agent failing when the CSV parser breaks on UTF-16 BOM in user import file

## TL;DR
The parser breaks because the file is UTF-16 with a BOM and the parser assumes UTF-8. Detect the encoding first, strip the BOM, and decode properly before parsing. HR systems love exporting UTF-16; your parser must expect it.

## The error
```text
Onboarding agent failed
CSV parser broke on UTF-16 BOM in user import file. First column name unreadable.
```

## Use this skill
Agent? Search Vectle for this error:
```bash
curl -s "https://vectle.com/api/v1/search?q=onboarding agent failed when csv parser broke on utf-16 bom in user import file"
```

## Fix it

### Step 1: Confirm the file encoding

```bash
Check the file's first bytes for a BOM or run an encoding detection.
```

Expected: You confirm UTF-16 with BOM.

### Step 2: Decode with the right encoding

```bash
Open the file as UTF-16 (or detect encoding) and strip the BOM before parsing.
```

Expected: Headers parse cleanly with correct column names.

### Step 3: Add encoding detection to the importer

```bash
Sniff the BOM or detect encoding on every import, defaulting sensibly.
```

Expected: Future files parse regardless of encoding.

### Step 4: Normalize to UTF-8 internally

```bash
Convert to UTF-8 right after reading so the rest of the pipeline has one encoding.
```

Expected: Downstream code never sees encoding surprises.

### Step 5: Re-run the import

```bash
Process the file again with the fixed importer.
```

Expected: All rows import with correct headers.

## When this applies

- CSV imports break on UTF-16 or BOM files
- First column names look garbled
- You accept CSV uploads from HR systems

## When it doesn't

- The file is genuinely corrupt (get a fresh export)
- The delimiter is wrong (check for semicolons versus commas)
- Rows fail validation after parsing (check the data)

## Compatibility

CSV parsing generally. HR system exports (Excel, Workday, and others).

## Variant phrasings

### csv utf-16 bom parser error

Same failure. BOM detection plus proper decoding fixes it.

### csv first column garbled import

Garbled first headers are the classic BOM symptom.

### excel csv encoding import failed

Excel exports UTF-16 or locale encodings freely. Detect, do not assume.

## Why it happens

Many HR and spreadsheet tools export CSV as UTF-16 with a byte-order mark, while parsers default to UTF-8. The BOM bytes glue onto the first header name, so the column is never found and the import fails or mis-maps. It is an encoding assumption, not a data problem.

## Edge cases

- Some files are UTF-8 with BOM; strip the BOM in that case too
- Mixed-encoding files from merged exports need per-file detection, not a global setting
- Log the detected encoding with each import for debuggability

## If it still fails

- Reproduce with a minimal run: one user, one file, one step.
- Read the agent's full trace, not just the final error; the failure is usually upstream.
- Check the underlying API or tool directly, outside the agent, to separate agent bugs from service bugs.
- Reduce concurrency to one and see if the failure persists; races hide as flakes.
- If the run is business-critical, add a human checkpoint before the destructive steps.

## Prevention

- Checkpoint long runs so any failure resumes instead of restarting.
- Cap and back off every retry loop; unbounded retries are outages waiting to happen.
- Validate inputs at each pipeline stage; fail fast with clear errors.
- Log enough context per step that a timeout is diagnosable without rerunning.
- Give destructive steps a human checkpoint or a dry-run mode.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HnBcAY6R1k6mKze5TKld_A