agent's test run exceeded the context window on large failure logs: how to summarize
Summarizes oversized test failure logs with an extract-dedupe-cap pipeline so the agent works from kilobytes, not megabytes. Use when a full log blows past the context window or the agent truncates it. Not for small logs the agent can read directly.
TL;DR
Never paste a multi-megabyte failure log into the agent. Run a pipeline first: extract the failing test names, pull a bounded window of lines around each traceback, dedupe identical failures, and hand the agent a summary of unique causes with counts. The agent then works from a few kilobytes and cites specific failures instead of drowning.
Error
[agent] test output is 2.4 MB, exceeds my context window, showing first 200 lines...
[first 200 lines] all green setup output, zero failures visible
The actual failures start at line 18,000.Steps
- Extract failing test names only: search the log for FAIL, ERROR, and traceback headers. Expected: a list of failing test names, nothing else.
- For each failing test, grab a bounded window (about 40 lines) around its traceback. Expected: per-failure excerpts, each small enough to read.
- Dedupe: group failures with identical tracebacks and count occurrences. Expected: 200 failures collapse into a handful of unique causes with counts.
- Write a summary file: test name, one-line error, first differing line, occurrence count. Expected: the summary is a few kilobytes.
- Give the agent the summary plus log line numbers for the full context. Expected: the agent names specific failures and their causes without ever reading the whole log.
When to use
- The failure log is larger than the agent's context window.
- The agent truncates the log and misses failures past the cutoff.
- Hundreds of failures share a few root causes.
When not to use
- The log is small enough to read directly (summarizing adds nothing).
- You need the full log for compliance or audit (archive it, but still summarize for the agent).
Tool compatibility
- Any coding agent (Claude Code, Cursor, etc.) triaging pytest, Jest, Vitest, Go test, or JUnit output.
- Extraction with grep, awk, or a small script; any machine-readable reporter (JUnit XML, JSON) makes dedupe trivial.
Variant phrasings
agent truncated the log and missed the failures
Truncation keeps the head, failures live in the tail. The extract step finds failures wherever they are.
too many failures to read one by one
Dedupe turns 200 failures into 6 causes. The agent fixes causes, not instances.
failure log is megabytes of noise
Most of the bytes are passing-test chatter. The pipeline keeps only failure-shaped lines.
Why it happens
Test runners print everything: setup, passing tests, warnings, then failures. Agents read from the top and run out of room before the signal. The failure information is a tiny fraction of the bytes, but nothing extracts it unless you ask.
Edge cases
- Interleaved parallel output (pytest-xdist) scrambles line order; use the JUnit XML instead of the text log when workers interleave.
- The first failure can cascade into hundreds of knock-on failures; fix in dedupe order (most common cause first) and re-run before chasing the rest.
- Warnings-as-errors configs turn warning text into failures; keep a separate warnings section in the summary so they are not confused with real failures.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_pV-wwz3wv7R6Gbj6PxzB0g