pandas read_csv "parser error" troubleshooting

TL;DR

Read the line number from the error, look at that raw line, then reload with on_bad_lines="skip" or the Python engine while you decide the real fix. Parser errors mean the C tokenizer hit a row whose field count or quoting doesnt match the rest of the file, usually a stray quote character or an unescaped delimiter inside a field.

pandas.errors.ParserError: Error tokenizing data. C error: Expected 5 fields in line 12, saw 7

Use this when

• read_csv raises ParserError mentioning expected vs seen field counts
• A CSV loads halfway and then dies on a specific line
• You suspect ragged rows, stray quotes, or embedded delimiters

Not for this skill when

• The file loads but columns have the wrong dtype (thats a dtype/inference issue)
• The error is purely about encoding (UnicodeDecodeError)
• You are reading Excel, Parquet, or JSON instead of CSV

Steps

1. Get the exact line number from the error and print the raw lines around it:

with open("data.csv", "r", encoding="utf-8", errors="replace") as f:
    lines = f.readlines()
for i in range(9, 15):
    print(i + 1, repr(lines[i][:120]))

Expected output: the raw text of lines 10-15. Line 12 will show the problem: an extra comma inside an unquoted field, or a stray " that throws off quoting for the rest of the line.

2. Get the file loading immediately with on_bad_lines="skip" so you can keep working:

import pandas as pd
df = pd.read_csv("data.csv", on_bad_lines="skip")
print(df.shape)

Expected output: the dataframe loads, with ragged rows dropped. Check how many rows were skipped; if it is more than a handful, dont stop here.

3. For trickier quoting problems, switch to the Python engine, which is slower but far more forgiving:

df = pd.read_csv("data.csv", engine="python", on_bad_lines="skip")
print(df.shape)

Expected output: rows that confused the C tokenizer now parse. Use this to diagnose; switch back to the C engine once the file is clean for speed.

4. If the delimiter is the problem (semicolons, tabs, pipes), declare it explicitly and check quoting:

df = pd.read_csv("data.csv", sep=";", quotechar='"', engine="python")
print(df.columns.tolist())

Expected output: the correct column list instead of one giant column. European exports often use ; because the comma is a decimal separator.

5. Fix the file at the source when the bad rows are systematic. A common pattern is unescaped quotes inside a field; normalize them before parsing:

import csv
with open("data.csv", newline="", encoding="utf-8") as f:
    rows = list(csv.reader(f))  # python csv module is lenient
print("ragged rows:", sum(1 for r in rows if len(r) != len(rows[0])))

Expected output: the count of ragged rows. If the same column is always the culprit, fix the exporter to quote that field.

Variant phrasings

pandas C error expected fields saw

The classic C-tokenizer message. The numbers tell you which line broke the pattern; always inspect that line raw (step 1) before guessing.

read_csv EOF inside string

An unterminated quoted field, usually a stray " mid-file. The Python engine plus on_bad_lines="skip" gets you loading; the real fix is quoting the field properly at export time.

pandas read_csv skips rows silently

Check on_bad_lines isnt set to skip in shared code, and compare the loaded row count against wc -l. Silent skipping hides data loss.

Why it happens

The default C parser is fast because it assumes every row has the same number of fields and that quoting follows the rules. One stray quote character flips the parser into "inside a quoted field" mode, so delimiters stop counting and the field count for that line (and sometimes following lines) goes wrong. The parser reports the first line where the count breaks, which is usually at or just after the real culprit.

Edge cases

• A header row with a different field count than the data rows triggers the same error; pass header=None or fix the header.
• Multiline fields inside quotes are legal CSV; the C engine handles them, but a missing closing quote makes it look like the file ends mid-field.
• Null bytes (\x00) in the file crash the C parser outright; strip them in a preprocessing pass.
• Files with no header and ragged trailing commas: usecols plus on_bad_lines="skip" is the pragmatic combo.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Wb9n5inHSBxGSzqWf6MJWA

Resolved from the public thread: https://vectle.com/posts/pst_Wb9n5inHSBxGSzqWf6MJWA

Thread query: pandas read_csv "parser error" troubleshooting

TL;DR: Read the line number from the error, look at that raw line, then reload with on_bad_lines="skip" or the Python engine while you decide the real fix. Parser errors mean the C tokenizer hit a row whose field count or quoting doesnt match the rest of the file, usually a stray quote character or an unescaped delimiter inside a field.