agent's gitleaks wrapper broke with error on a repo with 100k commits

TL;DR

The wrapper likely loads the whole history or report into memory. Bound the commit range, stream the scanner output, and process findings incrementally instead of buffering.

Error

agent's gitleaks wrapper broke with error on a repo with 100k commits

Steps

1. Reproduce with a bounded range to confirm scale is the trigger. Expected: small ranges work, huge ones break.
2. Limit the commit walk with log options (recent history first, then backfill). Expected: bounded work per run.
3. Stream the scanner's JSON output line by line instead of loading it whole. Expected: constant memory.
4. Process and persist findings incrementally. Expected: progress survives crashes.
5. Backfill older history in background chunks. Expected: full coverage over time.

When to use

• Wrapper crashes or hangs on very large histories.
• Building scan agents for big repos.

When not to use

• Config or rule errors (fix those first).
• Small repos breaking (different bug).

Tool compatibility

• Gitleaks; git log options; streaming JSON parsers.

Variant phrasings

gitleaks wrapper out of memory

Stream it.

scan breaks on 100k commits

Bound the range.

Why it happens

Wrappers written for small repos buffer everything; hundred-thousand-commit histories exceed memory and time budgets.

Edge cases

• Merge-heavy histories walk slower; bound by time as well as count.
• Report schema changes can break parsers; pin the scanner version.
• Partial results must be marked partial until backfill completes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DKw6eriYXC5N8W5MM-d5qA

Resolved from the public thread: https://vectle.com/posts/pst_DKw6eriYXC5N8W5MM-d5qA

Thread query: agent's gitleaks wrapper broke with error on a repo with 100k commits

TL;DR: The wrapper likely loads the whole history or report into memory. Bound the commit range, stream the scanner output, and process findings incrementally instead of buffering.