agent crashed parsing a 2GB usage CSV: out of memory error

TL;DR

Loading a 2GB CSV whole guarantees the crash; the data per account is what matters, not the file. Stream the file row by row with a chunked reader, accumulate only per-account aggregates, spill intermediate results to disk if they grow, and the same job runs in a fraction of the memory regardless of file size.

agent crashed parsing a 2GB usage CSV: out of memory error

Use this when

• The agent dies with memory errors on large usage exports
• Smaller files from the same source process fine
• Memory climbs steadily until the crash

Not for this skill when

• The file is corrupt and the parser loops on one record, inflating memory
• The aggregates themselves (millions of accounts) genuinely exceed memory and need a database

Steps

1. Confirm the crash pattern: memory growth tracks file reading, and the file parses fine in a streaming test read. Verify: the whole-file load is convicted as the cause
2. Rewrite ingestion with a chunked or streaming reader and verify peak memory on the full file. Verify: the full file processes within a bounded memory footprint
3. Replace row storage with incremental per-account aggregation during the stream. Verify: memory no longer scales with row count
4. If aggregates still grow large, spill them to a local database or disk-backed store mid-run. Verify: the job scales to the real account count without redesign
5. Add a pre-flight file-size check that selects the streaming path automatically for large inputs. Verify: no future big export can re-trigger the crash by surprise

Variant phrasings

agent out of memory 2GB CSV

A related phrasing for the same underlying problem.

stream large usage file parsing

A related phrasing for the same underlying problem.

chunked CSV aggregation agent

A related phrasing for the same underlying problem.

Compatibility: Compatibility: large CSV usage exports processed in Python (pandas chunking, csv streaming) or similar runtimes, current versions.

Why it happens

Memory failures on growing files are inevitable with whole-file designs; the file always eventually wins. Streaming with incremental aggregation changes the scaling law from file size to aggregate size, which for usage data is orders of magnitude smaller and stable.

Edge cases / pitfalls

• pandas read in chunks still concatenates badly if you accumulate frames. Accumulate aggregates, not chunks.
• Malformed rows inside a huge file need a counted quarantine, not an exception that wastes a long run.
• Compression helps transfer, not memory: a gzipped 2GB CSV expands to the same problem. Stream the decompression too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4MRcaMDr1Fgdhu6wgCee3g