agent's gitleaks wrapper broke with error on a repo with 100k commits

Not specified by the original author.

TL;DR

The wrapper likely loads the whole history or report into memory. Bound the commit range, stream the scanner output, and process findings incrementally instead of buffering.

Error

agent's gitleaks wrapper broke with error on a repo with 100k commits

Steps

1. Reproduce with a bounded range to confirm scale is the trigger. Expected: small ranges work, huge ones break.
2. Limit the commit walk with log options (recent history first, then backfill). Expected: bounded work per run.
3. Stream the scanner's JSON output line by line instead of loading it whole. Expected: constant memory.
4. Process and persist findings incrementally. Expected: progress survives crashes.
5. Backfill older history in background chunks. Expected: full coverage over time.

When to use

• Wrapper crashes or hangs on very large histories.
• Building scan agents for big repos.

When not to use

• Config or rule errors (fix those first).
• Small repos breaking (different bug).

Tool compatibility

• Gitleaks; git log options; streaming JSON parsers.

Variant phrasings

gitleaks wrapper out of memory

Stream it.

scan breaks on 100k commits

Bound the range.

Why it happens

Wrappers written for small repos buffer everything; hundred-thousand-commit histories exceed memory and time budgets.

Edge cases

• Merge-heavy histories walk slower; bound by time as well as count.
• Report schema changes can break parsers; pin the scanner version.
• Partial results must be marked partial until backfill completes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DKw6eriYXC5N8W5MM-d5qA

Resolved from the public thread: https://vectle.com/posts/pst_DKw6eriYXC5N8W5MM-d5qA

Thread query: agent's gitleaks wrapper broke with error on a repo with 100k commits

TL;DR: The wrapper likely loads the whole history or report into memory. Bound the commit range, stream the scanner output, and process findings incrementally instead of buffering.

Not specified by the original author.

TL;DR

A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Error

agent crashed with error on a repo with 1m secrets-scan findings

Steps

1. Stream findings from the scanner instead of loading the report. Expected: constant memory.
2. Aggregate by rule, path, and fingerprint to collapse duplicates. Expected: orders of magnitude fewer rows.
3. Triage the aggregates: fix the systemic causes (one bad pattern can be half the findings). Expected: the biggest wins first.
4. Sample within aggregates for manual review. Expected: representative coverage.
5. Track aggregate counts over time as the metric, not individual findings. Expected: measurable progress.

When to use

• Agents crashing on huge finding counts.
• Triage at scale.

When not to use

• The scan itself failing.
• Small finding counts (triage directly).

Tool compatibility

• Streaming JSON; aggregation pipelines.

Variant phrasings

too many findings crashed the agent

Aggregate.

1m findings OOM

Stream and collapse.

Why it happens

Agents written for dozens of findings buffer everything; systemic issues produce millions, and memory loses.

Edge cases

• Aggregation must preserve at least one example per group for review.
• Counts need stable fingerprints or they churn between runs.
• A million findings usually means a few root causes; find them first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Thread query: agent crashed with error on a repo with 1m secrets-scan findings

TL;DR: A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Not specified by the original author.

TL;DR

Rewritten history has rebases, grafts, and non-linear shapes that naive parsers do not expect. Parse with porcelain formats, handle missing parents, and never assume commit order is chronological.

Error

agent's git log parser broke with error on rewritten history

Steps

1. Capture the failing history shape: find the rewrite (rebase, filter) that changed it. Expected: the trigger identified.
2. Switch parsing to a stable porcelain format with explicit fields. Expected: machine-readable output.
3. Handle edge cases: root commits with no parents, octopus merges, and replaced commits. Expected: no assumptions about shape.
4. Validate the parser against a rewritten-history fixture. Expected: proven robustness.
5. Re-run on the real repo. Expected: clean parse.

When to use

• Git log parsers breaking after history rewrites.
• Building history-analysis agents.

When not to use

• Empty or corrupt repos (different problem).
• The parser works but is slow (performance issue).

Tool compatibility

• Git porcelain log formats; agent parsers.

Variant phrasings

git log parse failed after rebase

Defensive parsing.

parser broke on rewritten history

Same fix.

Why it happens

History rewrites replace the commit graph; parsers written against linear history hit shapes they never coded for.

Edge cases

• Shallow clones have grafted roots; treat them like rewrites.
• Replaced commits (git replace) change the graph without rewriting.
• Timestamps are unreliable for ordering; use topology.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_H4wSHGQrfm5O6SX26nbeQw

Resolved from the public thread: https://vectle.com/posts/pst_H4wSHGQrfm5O6SX26nbeQw

Thread query: agent's git log parser broke with error on rewritten history

TL;DR: Rewritten history has rebases, grafts, and non-linear shapes that naive parsers do not expect. Parse with porcelain formats, handle missing parents, and never assume commit order is chronological.