agent hit context window limit on 100k-row usage export, run failed

TL;DR

At 100k rows there is no prompt-based alternative: aggregate in code, stream the file row by row, and keep the model on the summaries. Extract filtered slices only for targeted drill-downs.

agent hit context window limit on 100k-row usage export, run failed

Use this when

• An agent fails with context overflow on a ~100k-row export.
• The file is tens of megabytes.
• Any attempt to read it into the prompt fails.

Not for this skill when

• The file fits in context. Just read it.
• The overflow is from something else. Find the real large input.
• Only a few rows are needed. Filter first, then read.

Steps

1. Write the aggregation as a standalone script. Verify: it runs without the model.
2. Stream the export row by row, never loading it whole. Verify: memory stays flat.
3. Emit per-app and per-status summaries. Verify: the summaries answer the audit question.
4. Bring only the summaries into context. Verify: the prompt stays small.
5. Confirm the run completes. Verify: no overflow.

Variant phrasings

context window exceeded

The error-first phrasing.

100k rows

The number-first search.

export too large for prompt

The plain-English phrasing.

Compatibility: Any agent runtime with code execution and a context window.

Why it happens

100k rows are tens of millions of tokens. Code is the only path; the model directs the analysis but never holds the data.

Edge cases / pitfalls

• Needing row-level detail for a few outliers: extract by filter, not by loading all.
• The script needing the schema first; sample rows to learn it.
• Summaries that answer the wrong question; define the needed aggregates before coding.
• The agent falling back to reading the file when the script has a bug; fix the script.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Cf0pxQePk2CuoPY3a2n7IA