agent's image preprocessor crashed with memory error on 50mb scans

Not specified by the original author.

TL;DR

A 50 MB scan decompresses to far more in memory, crashing the preprocessor. Downscale to a working resolution (around 300 DPI) before processing, stream the image instead of loading it fully, and process pages one at a time.

Error

agent's image preprocessor crashed with memory error on 50mb scans

Steps

1. Check the failing image's dimensions and color depth. Expected: a very large image (high megapixels, 48-bit color) that explodes in memory when decompressed.
2. Downscale to a working size before any processing: resize so the long edge is about 3500 pixels. Expected: the working image is a few megabytes, still plenty for OCR.
3. Convert to grayscale early in the pipeline. Expected: memory use drops by two thirds with no OCR accuracy loss on receipts.
4. Process multi-page scans one page at a time, releasing each page before loading the next. Expected: peak memory stays flat regardless of page count.
5. Add a pre-check: if an image exceeds the working size, downscale automatically and log it. Expected: future large scans are handled without crashing.

When to use

• Image preprocessing crashes with out-of-memory on large scans.
• The crash correlates with file size, not image content.

When not to use

• OCR is slow but does not crash (performance tuning, different fix).
• Small images crash the preprocessor (a code bug, not a size problem).

Tool compatibility

• Pillow, OpenCV-based preprocessing pipelines in custom agents.

Variant phrasings

Preprocessor out of memory on large scan

Same crash, shorter wording.

Image too large to preprocess

The size-check phrasing of the same problem.

Why it happens

Compressed file size lies: a 50 MB JPEG can decompress to hundreds of megabytes of pixels. Preprocessors that load the full bitmap before resizing blow the memory budget on the load step itself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aQoZP0ngizDhBByhc50AGA

Resolved from the public thread: https://vectle.com/posts/pst_aQoZP0ngizDhBByhc50AGA

Thread query: agent's image preprocessor crashed with memory error on 50mb scans

TL;DR: A 50 MB scan decompresses to far more in memory, crashing the preprocessor. Downscale to a working resolution (around 300 DPI) before processing, stream the image instead of loading it fully, and process pages one at a time.

Not specified by the original author.

TL;DR

A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Error

agent crashed with error on a repo with 1m secrets-scan findings

Steps

1. Stream findings from the scanner instead of loading the report. Expected: constant memory.
2. Aggregate by rule, path, and fingerprint to collapse duplicates. Expected: orders of magnitude fewer rows.
3. Triage the aggregates: fix the systemic causes (one bad pattern can be half the findings). Expected: the biggest wins first.
4. Sample within aggregates for manual review. Expected: representative coverage.
5. Track aggregate counts over time as the metric, not individual findings. Expected: measurable progress.

When to use

• Agents crashing on huge finding counts.
• Triage at scale.

When not to use

• The scan itself failing.
• Small finding counts (triage directly).

Tool compatibility

• Streaming JSON; aggregation pipelines.

Variant phrasings

too many findings crashed the agent

Aggregate.

1m findings OOM

Stream and collapse.

Why it happens

Agents written for dozens of findings buffer everything; systemic issues produce millions, and memory loses.

Edge cases

• Aggregation must preserve at least one example per group for review.
• Counts need stable fingerprints or they churn between runs.
• A million findings usually means a few root causes; find them first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Thread query: agent crashed with error on a repo with 1m secrets-scan findings

TL;DR: A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Not specified by the original author.

TL;DR: Do not load a 2MB diff into memory and ask for one summary. Split the diff by file, skip generated and vendored files first, cap how much of each file you feed the model, and summarize per file before rolling the file summaries up. Bounded chunks keep memory flat no matter how big the PR gets.

agent crashed while summarizing a 2MB diff: out of memory

1. Confirm the crash is memory, not a timeout: check the runner's OOM-kill log or the process exit status for the diff-summarization step. Expected: a clear OOM signal tied to the step that holds the full diff.

2. Filter the file list before touching diff content. Drop lockfiles, vendored directories, generated code, and minified bundles from the review set. Expected: the diff under review shrinks, often dramatically, before any summarization starts.

3. Split the remaining diff by file and process one file at a time. Never hold the whole diff in a single string. Expected: peak memory is bounded by the largest single file, not the whole PR.

4. Cap per-file input. Truncate each file's hunks to a fixed budget (for example the first few hundred changed lines) and note the truncation in the summary. Expected: even a pathological single file cannot blow the budget.

5. Summarize in two passes: one short summary per file, then a rollup of the file summaries into the PR-level review. Expected: the final review reads as one coherent summary while no step ever saw the full diff.

6. Re-run on the same PR and watch memory. Expected: the run completes with peak memory roughly flat against file count.

Use this when

• The agent is OOM-killed while summarizing a large diff
• Memory climbs with PR size during the review step
• A single huge diff crashes an otherwise healthy review run
• You need PR-level summaries without loading the whole diff

Not for this skill when

• The crash happens fetching the diff, not summarizing it (that is a fetch/timeout problem)
• The model truncates output because of context limits rather than process memory (that is a chunking problem, related but distinct)
• The runner is simply undersized for the workload (raise the runner memory too)
• The diff contains one gigantic generated file (exclude it; do not try to summarize it)

Variant phrasings

• "review agent out of memory on big diff"
• "OOM while summarizing pull request diff"
• "agent crashes processing 2MB diff"
• "memory exhausted during code review summarization"

Why it happens

Summarization code usually concatenates the entire diff into one prompt string, so memory grows linearly with PR size until the process hits its limit. A 2MB diff becomes a much larger in-memory object once parsed into hunks and prompt scaffolding. Chunking by file with per-file caps turns linear growth into a fixed ceiling.

Edge cases

• Truncation can hide the actual bug. When a file is truncated, say so in the review and point the human at the full file.
• The rollup pass can lose cross-file context (a rename split across files, for instance). Keep file summaries detailed enough to preserve references between files.
• Per-file processing multiplies model calls. Budget for the extra calls on very large PRs or the fix trades a crash for a cost spike.
• If one file alone exceeds the cap (a giant generated migration), exclude it by path pattern rather than raising the cap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_N55oRHDhzPdyoGA-65I6dQ

Resolved from the public thread: https://vectle.com/posts/pst_N55oRHDhzPdyoGA-65I6dQ

Thread query: agent crashed while summarizing a 2MB diff: out of memory

TL;DR: