## TL;DR

Loading gigabytes to score entropy exhausts memory. Stream the input in chunks, score incrementally, and sample when full precision is not needed.

## Error

```text
agent's entropy scorer failed with memory error on 2gb logs
```

## Steps

1. Confirm the scorer loads the whole input before scoring. Expected: the memory pattern found.
2. Switch to chunked streaming with a running score. Expected: constant memory.
3. For triage passes, sample the input (every Nth chunk) to find hot regions, then score those fully. Expected: fast triage, full accuracy where it matters.
4. Cap the buffer for any single line or record; absurdly long lines are the usual killer. Expected: no single record can OOM the scorer.
5. Re-run on the 2GB input. Expected: completion within budget.

## When to use

- Entropy or secret scoring OOM on large logs.
- Building scalable scan agents.

## When not to use

- Wrong scores (tune the scorer, not the memory).
- Small inputs OOM (different bug).

## Tool compatibility

- Streaming text processing; entropy scorers.

## Variant phrasings

### entropy scorer out of memory

Stream it.

### scorer OOM on large logs

Chunk and sample.

## Why it happens

Naive scorers read everything into memory; log volume grows faster than RAM.

## Edge cases

- Sampling can miss a lone secret; use it for triage, not final verdicts.
- Line-length caps must be generous enough for real secrets.
- Parallel chunk scoring needs deterministic merging.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8qTwsQAekWoukwX8Js14tA