agent's entropy scorer failed with memory error on 2gb logs
Fixes an agent's entropy scorer dying with a memory error on 2GB logs: stream and sample instead of loading. Use when scorers OOM on large inputs. Not for accuracy complaints.
TL;DR
Loading gigabytes to score entropy exhausts memory. Stream the input in chunks, score incrementally, and sample when full precision is not needed.
Error
agent's entropy scorer failed with memory error on 2gb logsSteps
- Confirm the scorer loads the whole input before scoring. Expected: the memory pattern found.
- Switch to chunked streaming with a running score. Expected: constant memory.
- For triage passes, sample the input (every Nth chunk) to find hot regions, then score those fully. Expected: fast triage, full accuracy where it matters.
- Cap the buffer for any single line or record; absurdly long lines are the usual killer. Expected: no single record can OOM the scorer.
- Re-run on the 2GB input. Expected: completion within budget.
When to use
- Entropy or secret scoring OOM on large logs.
- Building scalable scan agents.
When not to use
- Wrong scores (tune the scorer, not the memory).
- Small inputs OOM (different bug).
Tool compatibility
- Streaming text processing; entropy scorers.
Variant phrasings
entropy scorer out of memory
Stream it.
scorer OOM on large logs
Chunk and sample.
Why it happens
Naive scorers read everything into memory; log volume grows faster than RAM.
Edge cases
- Sampling can miss a lone secret; use it for triage, not final verdicts.
- Line-length caps must be generous enough for real secrets.
- Parallel chunk scoring needs deterministic merging.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_8qTwsQAekWoukwX8Js14tA