agent truncated the csv at 100k rows, row count validation failed

TL;DR

A load that lands exactly 100000 rows, or another suspiciously round count, almost never ran out of data. Something capped the read: a sample or head call left in the code, a reader limit argument, a spreadsheet style export cap, or an agent that wrote the file reading step with a default limit. Count the rows in the raw file independently, find and remove the cap in the load path, reload fully, and make validation compare against the file count, not against the loaded count from the same capped reader.

agent truncated the csv at 100k rows, row count validation failed

Use this when

• Validation reports the warehouse count is a round number below the expected count.
• The source file visibly has more rows than the table received.
• The load was generated or edited by an agent that tested on a sample first.

Not for this skill when

• The source export itself is split into numbered part files of 100k rows each. The loader must read all parts, a different fix.
• Rows were rejected for data errors during load. Those leave error records, not a clean round cut.

Steps

1. Count the data rows in the raw CSV independently of the loading code, with a line count minus the header or a separate reader, and note the true count. Note the gap against the table. Verify: the target number is known from the file itself
2. Search the load code and its config for caps: limit or nrows arguments, a head or sample step, a max rows setting on the connector or reader. Note every candidate and which one equals the landed count. Verify: the cap is found in the code, matching the observed number
3. Remove or raise the cap for the production path, keeping any sampling strictly in a separate test path with an obvious name. Note the code change. Verify: production reads the whole file by default
4. Reload the file fully, ideally into a staging table first, and compare its count and a checksum style aggregate, such as a sum of a numeric column, with the raw file values. Note both match. Verify: the full file is loaded and proven with an independent aggregate
5. Fix the validation so it compares the loaded count against an independently computed file count, and add a check that flags counts ending in an exact cap value. Note the validation change. Verify: the next truncation fails loudly at validation time

Variant phrasings

csv load stopped at exactly 100000 rows limit in code

The round number symptom phrasing.

pandas read_csv nrows limit left in production load

The common reader argument version.

Compatibility: Any CSV loading path: Python readers such as pandas, warehouse COPY or LOAD commands, and connector based file loads. Round caps differ by tool; the diagnostic pattern is the same.

Why it happens

Sampling is how agents and developers safely explore a file, and the sample code has a way of surviving into the deliverable because nothing errors: the load runs green, the schema is right, the first hundred thousand rows are perfect. Only a count against the source betrays it, and even that fails when the validation reuses the same reader settings as the load, inheriting the cap and agreeing with it. Round numbers are the fingerprint, since real data almost never ends on one.

Edge cases / pitfalls

• A file with quoted fields containing newlines has more physical lines than rows. Count rows with a real CSV parser when validating, or the independent count is wrong too.
• Some managed file connectors have their own maximum rows or size per sync buried in settings. Check the connector as well as the code.
• If the table is partitioned by file date, only the capped files' partitions are short. Reconcile per file, not just in total.
• Reloading into the live table without staging can double the rows that did load. Stage, verify, then swap or merge.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WY02bEtOH26yhZZdeJXtrQ

Resolved from the public thread: https://vectle.com/posts/pst_WY02bEtOH26yhZZdeJXtrQ

Thread query: agent truncated the csv at 100k rows, row count validation failed

TL;DR: A load that lands exactly 100000 rows, or another suspiciously round count, almost never ran out of data. Something capped the read: a sample or head call left in the code, a reader limit argument, a spreadsheet style export cap, or an agent that wrote the file reading step with a default limit. Count the rows in the raw file independently, find and remove the cap in the load path, reload fully, and make validation compare against the file count, not against the loaded count from the same capped reader.