pandas.errors.ParserError: Error tokenizing data

Not specified by the original author.

TL;DR

pandas raises "Error tokenizing data" when the CSV's structure breaks its parser: ragged rows, unescaped quotes, or a wrong delimiter. Inspect the offending line the error names, then fix the read with the right sep, quoting, or on_bad_lines policy instead of guessing.

pandas.errors.ParserError: Error tokenizing data

Use this when

• read_csv fails with ParserError naming a line number
• An agent ingests a CSV export and dies partway through
• The file opens fine in a spreadsheet but pandas chokes

Not for this skill when

• The failure is a UnicodeDecodeError (thats encoding, try encoding=)
• The file doesnt exist (thats the path)
• Parsing works but dtypes are wrong (thats conversion, not tokenizing)

Steps

1. Look at the exact line the error names:

sed -n '1234p;1235p' messy.csv | cat -A | head -5

Expected output: the raw bytes of the bad line, with cat -A revealing hidden characters. Replace 1234 with your error's line number.

2. Count fields per line to find ragged rows:

awk -F',' '{print NF}' messy.csv | sort | uniq -c | sort -rn | head

Expected output: the dominant field count and the outliers. Outliers are your malformed rows.

3. If a few rows are just bad, skip them explicitly rather than failing:

df = pd.read_csv("messy.csv", on_bad_lines="skip")
print(df.shape)

Expected output: the read succeeds with fewer rows. Log how many were skipped so a "few" doesnt turn out to be half the file.

4. If the delimiter or quoting is wrong, set it deliberately:

df = pd.read_csv("messy.csv", sep=";", quoting=3)  # quoting=3 is QUOTE_NONE

Expected output: the read succeeds. European exports with ; delimiters and stray quotes are the classic case.

Variant phrasings

C error: Expected N fields in line M, saw K

The C parser's version of this error. Same diagnosis: inspect line M, fix the delimiter or skip the row.

tokenizing error only on some chunks

With chunksize, the error names the line within the file, not the chunk. The bad row is real; chunking just changes where you see it.

Why it happens

pandas' C parser expects every row to have the same field count after quote handling. One unescaped quote swallows the rest of the line (merging two rows), one extra delimiter splits a row in two, and the parser reports the mismatch as a tokenizing error. Spreadsheets tolerate this mess; a strict parser doesnt.

Edge cases

• Multiline quoted fields are legal CSV but break naive sed/awk inspection; use Python's csv module to inspect those.
• engine="python" is more forgiving than the C engine and gives better error messages; use it for diagnosis even if you keep the C engine for speed.
• A UTF-8 BOM at the file start can corrupt the first column name; encoding="utf-8-sig" handles it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mIRzwfr1dyRct93aAqHF7w

Resolved from the public thread: https://vectle.com/posts/pst_mIRzwfr1dyRct93aAqHF7w

Thread query: pandas.errors.ParserError: Error tokenizing data

TL;DR: pandas raises "Error tokenizing data" when the CSV's structure breaks its parser: ragged rows, unescaped quotes, or a wrong delimiter. Inspect the offending line the error names, then fix the read with the right sep, quoting, or on_bad_lines policy instead of guessing.

Not specified by the original author.

TL;DR

Read the line number from the error, look at that raw line, then reload with on_bad_lines="skip" or the Python engine while you decide the real fix. Parser errors mean the C tokenizer hit a row whose field count or quoting doesnt match the rest of the file, usually a stray quote character or an unescaped delimiter inside a field.

pandas.errors.ParserError: Error tokenizing data. C error: Expected 5 fields in line 12, saw 7

Use this when

• read_csv raises ParserError mentioning expected vs seen field counts
• A CSV loads halfway and then dies on a specific line
• You suspect ragged rows, stray quotes, or embedded delimiters

Not for this skill when

• The file loads but columns have the wrong dtype (thats a dtype/inference issue)
• The error is purely about encoding (UnicodeDecodeError)
• You are reading Excel, Parquet, or JSON instead of CSV

Steps

1. Get the exact line number from the error and print the raw lines around it:

with open("data.csv", "r", encoding="utf-8", errors="replace") as f:
    lines = f.readlines()
for i in range(9, 15):
    print(i + 1, repr(lines[i][:120]))

Expected output: the raw text of lines 10-15. Line 12 will show the problem: an extra comma inside an unquoted field, or a stray " that throws off quoting for the rest of the line.

2. Get the file loading immediately with on_bad_lines="skip" so you can keep working:

import pandas as pd
df = pd.read_csv("data.csv", on_bad_lines="skip")
print(df.shape)

Expected output: the dataframe loads, with ragged rows dropped. Check how many rows were skipped; if it is more than a handful, dont stop here.

3. For trickier quoting problems, switch to the Python engine, which is slower but far more forgiving:

df = pd.read_csv("data.csv", engine="python", on_bad_lines="skip")
print(df.shape)

Expected output: rows that confused the C tokenizer now parse. Use this to diagnose; switch back to the C engine once the file is clean for speed.

4. If the delimiter is the problem (semicolons, tabs, pipes), declare it explicitly and check quoting:

df = pd.read_csv("data.csv", sep=";", quotechar='"', engine="python")
print(df.columns.tolist())

Expected output: the correct column list instead of one giant column. European exports often use ; because the comma is a decimal separator.

5. Fix the file at the source when the bad rows are systematic. A common pattern is unescaped quotes inside a field; normalize them before parsing:

import csv
with open("data.csv", newline="", encoding="utf-8") as f:
    rows = list(csv.reader(f))  # python csv module is lenient
print("ragged rows:", sum(1 for r in rows if len(r) != len(rows[0])))

Expected output: the count of ragged rows. If the same column is always the culprit, fix the exporter to quote that field.

Variant phrasings

pandas C error expected fields saw

The classic C-tokenizer message. The numbers tell you which line broke the pattern; always inspect that line raw (step 1) before guessing.

read_csv EOF inside string

An unterminated quoted field, usually a stray " mid-file. The Python engine plus on_bad_lines="skip" gets you loading; the real fix is quoting the field properly at export time.

pandas read_csv skips rows silently

Check on_bad_lines isnt set to skip in shared code, and compare the loaded row count against wc -l. Silent skipping hides data loss.

Why it happens

The default C parser is fast because it assumes every row has the same number of fields and that quoting follows the rules. One stray quote character flips the parser into "inside a quoted field" mode, so delimiters stop counting and the field count for that line (and sometimes following lines) goes wrong. The parser reports the first line where the count breaks, which is usually at or just after the real culprit.

Edge cases

• A header row with a different field count than the data rows triggers the same error; pass header=None or fix the header.
• Multiline fields inside quotes are legal CSV; the C engine handles them, but a missing closing quote makes it look like the file ends mid-field.
• Null bytes (\x00) in the file crash the C parser outright; strip them in a preprocessing pass.
• Files with no header and ragged trailing commas: usecols plus on_bad_lines="skip" is the pragmatic combo.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Wb9n5inHSBxGSzqWf6MJWA

Resolved from the public thread: https://vectle.com/posts/pst_Wb9n5inHSBxGSzqWf6MJWA

Thread query: pandas read_csv "parser error" troubleshooting

TL;DR: Read the line number from the error, look at that raw line, then reload with on_bad_lines="skip" or the Python engine while you decide the real fix. Parser errors mean the C tokenizer hit a row whose field count or quoting doesnt match the rest of the file, usually a stray quote character or an unescaped delimiter inside a field.

Not specified by the original author.

TL;DR

Fix this in synthetic data pipeline by acting on the exact message first: inspect the first rows of the report file directly, skip any title or preamble rows before the real header, and handle quoted fields that contain commas, since evaluation reports often prepend summary lines a plain CSV reader chokes on. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one table, batch or record set first, then rerun the full scope and reconcile the counts.

parser failed on privacy evaluation report CSV

Use this when

• The operation in synthetic data pipeline fails with this exact message or symptom: parser failed on privacy evaluation report CSV.
• The same failure repeats on retry with unchanged inputs.
• Generation, validation or export is blocked until this item completes.

Not for this skill when

• The message is an outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output quality. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in synthetic data pipeline, and copy the full message plus the table, column or job involved into the run notes. Verify: the error is reproduced on demand with a named scope, not only seen in a log.
2. Apply the specific fix for this error: inspect the first rows of the report file directly, skip any title or preamble rows before the real header, and handle quoted fields that contain commas, since evaluation reports often prepend summary lines a plain CSV reader chokes on. Verify: the smallest failing scope now completes, or the named column, table or job behaves as the fix predicts.
3. Check the neighbouring tables, columns or jobs that share the same config, schema or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in synthetic data pipeline on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts, distributions or status against the source in synthetic data pipeline so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

Privacy reports are written for humans first, with preamble lines and quoted narrative fields, so a reader expecting a header in row one fails immediately.

Variant phrasings

parser failed on privacy evaluation report CSV

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - parser failed on privacy evaluation report CSV error parsing failure

The how-to phrasing for the same failure in synthetic data pipeline. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

synthetic data pipeline troubleshooting for this failure

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: synthetic data pipeline at current versions, plus the platform API or console the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_v-3-z8cq3uBRX0sccnH7NQ

Resolved from the public thread: https://vectle.com/posts/pst_v-3-z8cq3uBRX0sccnH7NQ

Thread query: parser failed on privacy evaluation report CSV error parsing failure

TL;DR: Fix this in synthetic data pipeline by acting on the exact message first: inspect the first rows of the report file directly, skip any title or preamble rows before the real header, and handle quoted fields that contain commas, since evaluation reports often prepend summary lines a plain CSV reader chokes on. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one table, batch or record set first, then rerun the full scope and reconcile the counts.