## TL;DR
pandas raises "Error tokenizing data" when the CSV's structure breaks its parser: ragged rows, unescaped quotes, or a wrong delimiter. Inspect the offending line the error names, then fix the read with the right `sep`, `quoting`, or `on_bad_lines` policy instead of guessing.

```text
pandas.errors.ParserError: Error tokenizing data
```

## Use this when
- `read_csv` fails with ParserError naming a line number
- An agent ingests a CSV export and dies partway through
- The file opens fine in a spreadsheet but pandas chokes

## Not for this skill when
- The failure is a UnicodeDecodeError (thats encoding, try `encoding=`)
- The file doesnt exist (thats the path)
- Parsing works but dtypes are wrong (thats conversion, not tokenizing)

## Steps

1. Look at the exact line the error names:

```bash
sed -n '1234p;1235p' messy.csv | cat -A | head -5
```
Expected output: the raw bytes of the bad line, with `cat -A` revealing hidden characters. Replace 1234 with your error's line number.

2. Count fields per line to find ragged rows:

```bash
awk -F',' '{print NF}' messy.csv | sort | uniq -c | sort -rn | head
```
Expected output: the dominant field count and the outliers. Outliers are your malformed rows.

3. If a few rows are just bad, skip them explicitly rather than failing:

```python
df = pd.read_csv("messy.csv", on_bad_lines="skip")
print(df.shape)
```
Expected output: the read succeeds with fewer rows. Log how many were skipped so a "few" doesnt turn out to be half the file.

4. If the delimiter or quoting is wrong, set it deliberately:

```python
df = pd.read_csv("messy.csv", sep=";", quoting=3)  # quoting=3 is QUOTE_NONE
```
Expected output: the read succeeds. European exports with `;` delimiters and stray quotes are the classic case.

## Variant phrasings

### C error: Expected N fields in line M, saw K
The C parser's version of this error. Same diagnosis: inspect line M, fix the delimiter or skip the row.

### tokenizing error only on some chunks
With `chunksize`, the error names the line within the file, not the chunk. The bad row is real; chunking just changes where you see it.

## Why it happens
pandas' C parser expects every row to have the same field count after quote handling. One unescaped quote swallows the rest of the line (merging two rows), one extra delimiter splits a row in two, and the parser reports the mismatch as a tokenizing error. Spreadsheets tolerate this mess; a strict parser doesnt.

## Edge cases
- Multiline quoted fields are legal CSV but break naive `sed`/`awk` inspection; use Python's csv module to inspect those.
- `engine="python"` is more forgiving than the C engine and gives better error messages; use it for diagnosis even if you keep the C engine for speed.
- A UTF-8 BOM at the file start can corrupt the first column name; `encoding="utf-8-sig"` handles it.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mIRzwfr1dyRct93aAqHF7w