## TL;DR
Loading a 2GB CSV whole guarantees the crash; the data per account is what matters, not the file. Stream the file row by row with a chunked reader, accumulate only per-account aggregates, spill intermediate results to disk if they grow, and the same job runs in a fraction of the memory regardless of file size.

```text
agent crashed parsing a 2GB usage CSV: out of memory error
```

## Use this when
- The agent dies with memory errors on large usage exports
- Smaller files from the same source process fine
- Memory climbs steadily until the crash

## Not for this skill when
- The file is corrupt and the parser loops on one record, inflating memory
- The aggregates themselves (millions of accounts) genuinely exceed memory and need a database

## Steps
1. Confirm the crash pattern: memory growth tracks file reading, and the file parses fine in a streaming test read. Verify: the whole-file load is convicted as the cause
2. Rewrite ingestion with a chunked or streaming reader and verify peak memory on the full file. Verify: the full file processes within a bounded memory footprint
3. Replace row storage with incremental per-account aggregation during the stream. Verify: memory no longer scales with row count
4. If aggregates still grow large, spill them to a local database or disk-backed store mid-run. Verify: the job scales to the real account count without redesign
5. Add a pre-flight file-size check that selects the streaming path automatically for large inputs. Verify: no future big export can re-trigger the crash by surprise

## Variant phrasings
### agent out of memory 2GB CSV
A related phrasing for the same underlying problem.
### stream large usage file parsing
A related phrasing for the same underlying problem.
### chunked CSV aggregation agent
A related phrasing for the same underlying problem.

Compatibility: Compatibility: large CSV usage exports processed in Python (pandas chunking, csv streaming) or similar runtimes, current versions.

## Why it happens
Memory failures on growing files are inevitable with whole-file designs; the file always eventually wins. Streaming with incremental aggregation changes the scaling law from file size to aggregate size, which for usage data is orders of magnitude smaller and stable.

## Edge cases / pitfalls
- pandas read in chunks still concatenates badly if you accumulate frames. Accumulate aggregates, not chunks.
- Malformed rows inside a huge file need a counted quarantine, not an exception that wastes a long run.
- Compression helps transfer, not memory: a gzipped 2GB CSV expands to the same problem. Stream the decompression too.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4MRcaMDr1Fgdhu6wgCee3g