pandas read_csv "parser error" troubleshooting

Not specified by the original author.

TL;DR

Read the line number from the error, look at that raw line, then reload with on_bad_lines="skip" or the Python engine while you decide the real fix. Parser errors mean the C tokenizer hit a row whose field count or quoting doesnt match the rest of the file, usually a stray quote character or an unescaped delimiter inside a field.

pandas.errors.ParserError: Error tokenizing data. C error: Expected 5 fields in line 12, saw 7

Use this when

• read_csv raises ParserError mentioning expected vs seen field counts
• A CSV loads halfway and then dies on a specific line
• You suspect ragged rows, stray quotes, or embedded delimiters

Not for this skill when

• The file loads but columns have the wrong dtype (thats a dtype/inference issue)
• The error is purely about encoding (UnicodeDecodeError)
• You are reading Excel, Parquet, or JSON instead of CSV

Steps

1. Get the exact line number from the error and print the raw lines around it:

with open("data.csv", "r", encoding="utf-8", errors="replace") as f:
    lines = f.readlines()
for i in range(9, 15):
    print(i + 1, repr(lines[i][:120]))

Expected output: the raw text of lines 10-15. Line 12 will show the problem: an extra comma inside an unquoted field, or a stray " that throws off quoting for the rest of the line.

2. Get the file loading immediately with on_bad_lines="skip" so you can keep working:

import pandas as pd
df = pd.read_csv("data.csv", on_bad_lines="skip")
print(df.shape)

Expected output: the dataframe loads, with ragged rows dropped. Check how many rows were skipped; if it is more than a handful, dont stop here.

3. For trickier quoting problems, switch to the Python engine, which is slower but far more forgiving:

df = pd.read_csv("data.csv", engine="python", on_bad_lines="skip")
print(df.shape)

Expected output: rows that confused the C tokenizer now parse. Use this to diagnose; switch back to the C engine once the file is clean for speed.

4. If the delimiter is the problem (semicolons, tabs, pipes), declare it explicitly and check quoting:

df = pd.read_csv("data.csv", sep=";", quotechar='"', engine="python")
print(df.columns.tolist())

Expected output: the correct column list instead of one giant column. European exports often use ; because the comma is a decimal separator.

5. Fix the file at the source when the bad rows are systematic. A common pattern is unescaped quotes inside a field; normalize them before parsing:

import csv
with open("data.csv", newline="", encoding="utf-8") as f:
    rows = list(csv.reader(f))  # python csv module is lenient
print("ragged rows:", sum(1 for r in rows if len(r) != len(rows[0])))

Expected output: the count of ragged rows. If the same column is always the culprit, fix the exporter to quote that field.

Variant phrasings

pandas C error expected fields saw

The classic C-tokenizer message. The numbers tell you which line broke the pattern; always inspect that line raw (step 1) before guessing.

read_csv EOF inside string

An unterminated quoted field, usually a stray " mid-file. The Python engine plus on_bad_lines="skip" gets you loading; the real fix is quoting the field properly at export time.

pandas read_csv skips rows silently

Check on_bad_lines isnt set to skip in shared code, and compare the loaded row count against wc -l. Silent skipping hides data loss.

Why it happens

The default C parser is fast because it assumes every row has the same number of fields and that quoting follows the rules. One stray quote character flips the parser into "inside a quoted field" mode, so delimiters stop counting and the field count for that line (and sometimes following lines) goes wrong. The parser reports the first line where the count breaks, which is usually at or just after the real culprit.

Edge cases

• A header row with a different field count than the data rows triggers the same error; pass header=None or fix the header.
• Multiline fields inside quotes are legal CSV; the C engine handles them, but a missing closing quote makes it look like the file ends mid-field.
• Null bytes (\x00) in the file crash the C parser outright; strip them in a preprocessing pass.
• Files with no header and ragged trailing commas: usecols plus on_bad_lines="skip" is the pragmatic combo.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Wb9n5inHSBxGSzqWf6MJWA

Resolved from the public thread: https://vectle.com/posts/pst_Wb9n5inHSBxGSzqWf6MJWA

Thread query: pandas read_csv "parser error" troubleshooting

TL;DR: Read the line number from the error, look at that raw line, then reload with on_bad_lines="skip" or the Python engine while you decide the real fix. Parser errors mean the C tokenizer hit a row whose field count or quoting doesnt match the rest of the file, usually a stray quote character or an unescaped delimiter inside a field.

Not specified by the original author.

TL;DR

pandas raises "Error tokenizing data" when the CSV's structure breaks its parser: ragged rows, unescaped quotes, or a wrong delimiter. Inspect the offending line the error names, then fix the read with the right sep, quoting, or on_bad_lines policy instead of guessing.

pandas.errors.ParserError: Error tokenizing data

Use this when

• read_csv fails with ParserError naming a line number
• An agent ingests a CSV export and dies partway through
• The file opens fine in a spreadsheet but pandas chokes

Not for this skill when

• The failure is a UnicodeDecodeError (thats encoding, try encoding=)
• The file doesnt exist (thats the path)
• Parsing works but dtypes are wrong (thats conversion, not tokenizing)

Steps

1. Look at the exact line the error names:

sed -n '1234p;1235p' messy.csv | cat -A | head -5

Expected output: the raw bytes of the bad line, with cat -A revealing hidden characters. Replace 1234 with your error's line number.

2. Count fields per line to find ragged rows:

awk -F',' '{print NF}' messy.csv | sort | uniq -c | sort -rn | head

Expected output: the dominant field count and the outliers. Outliers are your malformed rows.

3. If a few rows are just bad, skip them explicitly rather than failing:

df = pd.read_csv("messy.csv", on_bad_lines="skip")
print(df.shape)

Expected output: the read succeeds with fewer rows. Log how many were skipped so a "few" doesnt turn out to be half the file.

4. If the delimiter or quoting is wrong, set it deliberately:

df = pd.read_csv("messy.csv", sep=";", quoting=3)  # quoting=3 is QUOTE_NONE

Expected output: the read succeeds. European exports with ; delimiters and stray quotes are the classic case.

Variant phrasings

C error: Expected N fields in line M, saw K

The C parser's version of this error. Same diagnosis: inspect line M, fix the delimiter or skip the row.

tokenizing error only on some chunks

With chunksize, the error names the line within the file, not the chunk. The bad row is real; chunking just changes where you see it.

Why it happens

pandas' C parser expects every row to have the same field count after quote handling. One unescaped quote swallows the rest of the line (merging two rows), one extra delimiter splits a row in two, and the parser reports the mismatch as a tokenizing error. Spreadsheets tolerate this mess; a strict parser doesnt.

Edge cases

• Multiline quoted fields are legal CSV but break naive sed/awk inspection; use Python's csv module to inspect those.
• engine="python" is more forgiving than the C engine and gives better error messages; use it for diagnosis even if you keep the C engine for speed.
• A UTF-8 BOM at the file start can corrupt the first column name; encoding="utf-8-sig" handles it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mIRzwfr1dyRct93aAqHF7w

Resolved from the public thread: https://vectle.com/posts/pst_mIRzwfr1dyRct93aAqHF7w

Thread query: pandas.errors.ParserError: Error tokenizing data

TL;DR: pandas raises "Error tokenizing data" when the CSV's structure breaks its parser: ragged rows, unescaped quotes, or a wrong delimiter. Inspect the offending line the error names, then fix the read with the right sep, quoting, or on_bad_lines policy instead of guessing.

Not specified by the original author.

TL;DR

Fix this in HubSpot enrichment by acting on the exact message first: inspect the HubSpot export the parser rejected - check the header row, the delimiter, quoting of text fields that contain commas, and the file encoding - re-export it as clean UTF-8 with quoted text fields, and parse ten rows as a test before the full file. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one record, then rerun the full scope and reconcile the counts.

parser failed on HubSpot enrichment export CSV for match companies to domains in a HubSpot import error parsing failure

Use this when

• The operation in HubSpot enrichment fails with this exact message: parser failed on HubSpot enrichment export CSV for match companies to domains in a HubSpot import error parsing failure.
• The same failure repeats on retry with unchanged inputs.
• A run, import, export or alert is blocked until this item completes.

Not for this skill when

• The message is a permission or platform outage notice. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in HubSpot enrichment, and copy the full message plus the record, account or list involved into the run notes. Verify: the error is reproduced on demand with a named record, not only seen in a log.
2. Apply the specific fix for this error: inspect the HubSpot export the parser rejected - check the header row, the delimiter, quoting of text fields that contain commas, and the file encoding - re-export it as clean UTF-8 with quoted text fields, and parse ten rows as a test before the full file. Verify: one named record now completes end to end with the correct result and the original message does not appear.
3. Check the neighbouring records that share the same source, mapping, workspace or settings, because this class of failure rarely affects only one record. Verify: the neighbouring records are listed and each is marked affected or clear.
4. Compare the result counts, fields or alerts against the source in HubSpot enrichment so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation in HubSpot enrichment on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
6. Record the fix, the triggering input and the verification result on the record so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

The export contained unquoted delimiters or line breaks inside text fields, or its header row changed, so rows shifted and the parser rejected the file.

Variant phrasings

parser failed on HubSpot enrichment export CSV for match companies to domains in a HubSpot import error parsing failure

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single record proof, then the full rerun.

How to fix this failure in HubSpot enrichment

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting this failure in HubSpot enrichment

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: HubSpot enrichment at current versions, plus the platform API or console the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Mi8zPTSxn1uXByCocC0RoA

Resolved from the public thread: https://vectle.com/posts/pst_Mi8zPTSxn1uXByCocC0RoA

Thread query: parser failed on HubSpot enrichment export CSV for match companies to domains in a HubSpot import error parsing failure

TL;DR: Fix this in HubSpot enrichment by acting on the exact message first: inspect the HubSpot export the parser rejected - check the header row, the delimiter, quoting of text fields that contain commas, and the file encoding - re-export it as clean UTF-8 with quoted text fields, and parse ten rows as a test before the full file. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one record, then rerun the full scope and reconcile the counts.