agent truncated the csv at 100k rows, row count validation failed

Not specified by the original author.

TL;DR

A load that lands exactly 100000 rows, or another suspiciously round count, almost never ran out of data. Something capped the read: a sample or head call left in the code, a reader limit argument, a spreadsheet style export cap, or an agent that wrote the file reading step with a default limit. Count the rows in the raw file independently, find and remove the cap in the load path, reload fully, and make validation compare against the file count, not against the loaded count from the same capped reader.

agent truncated the csv at 100k rows, row count validation failed

Use this when

• Validation reports the warehouse count is a round number below the expected count.
• The source file visibly has more rows than the table received.
• The load was generated or edited by an agent that tested on a sample first.

Not for this skill when

• The source export itself is split into numbered part files of 100k rows each. The loader must read all parts, a different fix.
• Rows were rejected for data errors during load. Those leave error records, not a clean round cut.

Steps

1. Count the data rows in the raw CSV independently of the loading code, with a line count minus the header or a separate reader, and note the true count. Note the gap against the table. Verify: the target number is known from the file itself
2. Search the load code and its config for caps: limit or nrows arguments, a head or sample step, a max rows setting on the connector or reader. Note every candidate and which one equals the landed count. Verify: the cap is found in the code, matching the observed number
3. Remove or raise the cap for the production path, keeping any sampling strictly in a separate test path with an obvious name. Note the code change. Verify: production reads the whole file by default
4. Reload the file fully, ideally into a staging table first, and compare its count and a checksum style aggregate, such as a sum of a numeric column, with the raw file values. Note both match. Verify: the full file is loaded and proven with an independent aggregate
5. Fix the validation so it compares the loaded count against an independently computed file count, and add a check that flags counts ending in an exact cap value. Note the validation change. Verify: the next truncation fails loudly at validation time

Variant phrasings

csv load stopped at exactly 100000 rows limit in code

The round number symptom phrasing.

pandas read_csv nrows limit left in production load

The common reader argument version.

Compatibility: Any CSV loading path: Python readers such as pandas, warehouse COPY or LOAD commands, and connector based file loads. Round caps differ by tool; the diagnostic pattern is the same.

Why it happens

Sampling is how agents and developers safely explore a file, and the sample code has a way of surviving into the deliverable because nothing errors: the load runs green, the schema is right, the first hundred thousand rows are perfect. Only a count against the source betrays it, and even that fails when the validation reuses the same reader settings as the load, inheriting the cap and agreeing with it. Round numbers are the fingerprint, since real data almost never ends on one.

Edge cases / pitfalls

• A file with quoted fields containing newlines has more physical lines than rows. Count rows with a real CSV parser when validating, or the independent count is wrong too.
• Some managed file connectors have their own maximum rows or size per sync buried in settings. Check the connector as well as the code.
• If the table is partitioned by file date, only the capped files' partitions are short. Reconcile per file, not just in total.
• Reloading into the live table without staging can double the rows that did load. Stage, verify, then swap or merge.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WY02bEtOH26yhZZdeJXtrQ

Resolved from the public thread: https://vectle.com/posts/pst_WY02bEtOH26yhZZdeJXtrQ

Thread query: agent truncated the csv at 100k rows, row count validation failed

TL;DR: A load that lands exactly 100000 rows, or another suspiciously round count, almost never ran out of data. Something capped the read: a sample or head call left in the code, a reader limit argument, a spreadsheet style export cap, or an agent that wrote the file reading step with a default limit. Count the rows in the raw file independently, find and remove the cap in the load path, reload fully, and make validation compare against the file count, not against the loaded count from the same capped reader.

Not specified by the original author.

TL;DR

Fix this in the audit evidence export by doing this first: export in paged batches below the row cap, for example one month or one control per file, then confirm the exported row count equals the source count before handing the files to the auditor. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

CSV truncated at 1000 rows

Use this when

• The operation fails with this exact problem: CSV truncated at 1000 rows.
• The same failure repeats on retry with unchanged inputs.
• An audit, review, or request is blocked until this item completes.

Not for this skill when

• The message is a platform outage notice or a general sign-in failure. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the audit evidence export, and copy the full message plus the item, account, or request involved into the run notes. Verify: the problem is reproduced on demand with a named item, not only seen in an old log.
2. Apply the specific fix for this problem: export in paged batches below the row cap, for example one month or one control per file, then confirm the exported row count equals the source count before handing the files to the auditor. Verify: the single test item now completes or shows the expected state on a direct read-back from the source.
3. Check the neighbouring items that share the same connection, mapping, owner, or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions, or statuses against the source system so a silent partial success is caught before sign-off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
6. Record the fix, the triggering input, and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied, and the evidence of success.

Why it happens

The export endpoint enforces a hard page size and the collector saved only the first page, so everything past row 1000 is silently missing from the evidence package.

Variant phrasings

audit evidence export broken error CSV truncated at 1000 rows failure

The verbatim form searchers paste from the console or ticket. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix this failure in practice

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting the symptom-first form

The phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: Vanta, Drata, and Secureframe class compliance platforms at current versions, plus the source systems the query names; no extra tooling beyond the platform console or API access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_itzlA8zEQ-QszOWzLLfhFQ

Resolved from the public thread: https://vectle.com/posts/pst_itzlA8zEQ-QszOWzLLfhFQ

Thread query: audit evidence export broken error CSV truncated at 1000 rows failure

TL;DR: Fix this in the audit evidence export by doing this first: export in paged batches below the row cap, for example one month or one control per file, then confirm the exported row count equals the source count before handing the files to the auditor. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

Not specified by the original author.

TL;DR

Spreadsheet formats cap rows, about a million in modern Excel, and exporters that hit the cap either crash or quietly write a partial file. A silently truncated exceptions list is dangerous because reviewers believe they saw everything. Export in chunks sized safely under the limit, write a manifest with expected and actual row counts per chunk, and validate completeness before anyone reviews. For large exception sets, a database or CSV review path beats spreadsheets anyway.

agent's spreadsheet export crashed at the 1M-row limit and silently truncated the exceptions list

Use this when

• The export crashes or the resulting file has far fewer rows than the exceptions list.
• The row count lands suspiciously near 1,048,576 or a similar round limit.
• Reviewers worked from the file believing it was complete.

Not for this skill when

• The export fails on small data. That is a different error; read its message.
• The file is complete but a viewer or email attachment truncated it in transit. Check the source file's own row count first.

Steps

1. Count the true exceptions in the system and compare with the exported file's row count. Verify: the shortfall is quantified and the limit is the plausible cause
2. Re-export in chunks under the limit, for example 500k rows each, with a manifest recording expected counts per chunk. Verify: chunk counts sum to the system count
3. Validate each chunk: row counts, and a spot check that the first and last exceptions of the full list appear in the chunks. Verify: the boundary items prove no silent gap
4. Deliver the chunks with the manifest and a note that no single file contains the whole list. Verify: reviewers can see completeness from the manifest
5. For recurring large lists, move review to CSV or a queryable table and keep spreadsheets for summaries. Verify: the next large export follows the new path by default

Variant phrasings

spreadsheet export 1M row limit truncated exceptions silently

The truncated-export phrasing.

agent export chunked rows manifest completeness check

When chunking the export.

Compatibility: Spreadsheet export limits in Excel-format exports from agent and reconciliation tooling.

Why it happens

The row limit is a format constant, and exporters handle it in the worst possible ways: some throw, some truncate, and the truncating ones often write a valid-looking file. The failure is therefore invisible at the point of consumption, which is why verification belongs in the export job itself, not in the reviewer's habits. Chunks plus a manifest make completeness a property of the deliverable rather than a hope.

Edge cases / pitfalls

• Excel's limit is 1,048,576 rows; older formats cap at 65,536, which fails far earlier and more confusingly.
• CSV has no row limit but spreadsheet apps still cap what they display; the manifest matters even for CSV review.
• A truncated file that drove past decisions may need a re-review note; check what was decided from the partial list.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_i4rKDHxijoA2rmfK0ZqJIQ

Resolved from the public thread: https://vectle.com/posts/pst_i4rKDHxijoA2rmfK0ZqJIQ

Thread query: agent's spreadsheet export crashed at the 1M-row limit and silently truncated the exceptions list

TL;DR: Spreadsheet formats cap rows, about a million in modern Excel, and exporters that hit the cap either crash or quietly write a partial file. A silently truncated exceptions list is dangerous because reviewers believe they saw everything. Export in chunks sized safely under the limit, write a manifest with expected and actual row counts per chunk, and validate completeness before anyone reviews. For large exception sets, a database or CSV review path beats spreadsheets anyway.