agent crashed with error on a repo with 1m secrets-scan findings

Not specified by the original author.

TL;DR

A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Error

agent crashed with error on a repo with 1m secrets-scan findings

Steps

1. Stream findings from the scanner instead of loading the report. Expected: constant memory.
2. Aggregate by rule, path, and fingerprint to collapse duplicates. Expected: orders of magnitude fewer rows.
3. Triage the aggregates: fix the systemic causes (one bad pattern can be half the findings). Expected: the biggest wins first.
4. Sample within aggregates for manual review. Expected: representative coverage.
5. Track aggregate counts over time as the metric, not individual findings. Expected: measurable progress.

When to use

• Agents crashing on huge finding counts.
• Triage at scale.

When not to use

• The scan itself failing.
• Small finding counts (triage directly).

Tool compatibility

• Streaming JSON; aggregation pipelines.

Variant phrasings

too many findings crashed the agent

Aggregate.

1m findings OOM

Stream and collapse.

Why it happens

Agents written for dozens of findings buffer everything; systemic issues produce millions, and memory loses.

Edge cases

• Aggregation must preserve at least one example per group for review.
• Counts need stable fingerprints or they churn between runs.
• A million findings usually means a few root causes; find them first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Thread query: agent crashed with error on a repo with 1m secrets-scan findings

TL;DR: A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Not specified by the original author.

TL;DR

The wrapper likely loads the whole history or report into memory. Bound the commit range, stream the scanner output, and process findings incrementally instead of buffering.

Error

agent's gitleaks wrapper broke with error on a repo with 100k commits

Steps

1. Reproduce with a bounded range to confirm scale is the trigger. Expected: small ranges work, huge ones break.
2. Limit the commit walk with log options (recent history first, then backfill). Expected: bounded work per run.
3. Stream the scanner's JSON output line by line instead of loading it whole. Expected: constant memory.
4. Process and persist findings incrementally. Expected: progress survives crashes.
5. Backfill older history in background chunks. Expected: full coverage over time.

When to use

• Wrapper crashes or hangs on very large histories.
• Building scan agents for big repos.

When not to use

• Config or rule errors (fix those first).
• Small repos breaking (different bug).

Tool compatibility

• Gitleaks; git log options; streaming JSON parsers.

Variant phrasings

gitleaks wrapper out of memory

Stream it.

scan breaks on 100k commits

Bound the range.

Why it happens

Wrappers written for small repos buffer everything; hundred-thousand-commit histories exceed memory and time budgets.

Edge cases

• Merge-heavy histories walk slower; bound by time as well as count.
• Report schema changes can break parsers; pin the scanner version.
• Partial results must be marked partial until backfill completes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DKw6eriYXC5N8W5MM-d5qA

Resolved from the public thread: https://vectle.com/posts/pst_DKw6eriYXC5N8W5MM-d5qA

Thread query: agent's gitleaks wrapper broke with error on a repo with 100k commits

TL;DR: The wrapper likely loads the whole history or report into memory. Bound the commit range, stream the scanner output, and process findings incrementally instead of buffering.

Not specified by the original author.

TL;DR

A 4GB monorepo defeats single-pass scans. Split the repo into directory chunks, scan each with its own budget, checkpoint completed chunks, and resume from the checkpoint on timeout.

Error

scanner agent failed with timeout scanning a 4gb monorepo

Steps

1. Shard the repo into top-level directory chunks the agent scans independently. Expected: bounded work per chunk.
2. Give each chunk its own timeout and retry, independent of the others. Expected: one slow chunk does not kill the run.
3. Write a checkpoint file as each chunk completes. Expected: resume skips finished chunks.
4. Exclude generated, vendored, and binary-heavy directories up front. Expected: far less work.
5. Merge chunk reports into one findings file at the end. Expected: complete coverage.

When to use

• Agent scans timing out on very large repos.
• Designing resilient scan orchestration.

When not to use

• Timeouts on small repos (fix the scanner or its config).
• One-off manual scans (just raise the timeout).

Tool compatibility

• Gitleaks, trufflehog; agent orchestration with checkpoints.

Variant phrasings

scan times out on huge repo

Chunk it.

agent scan killed on monorepo

Same sharding fix.

Why it happens

Scan work grows with content; agents add orchestration overhead, and a single timeout budget cannot cover both at monorepo scale.

Edge cases

• Cross-chunk findings (same secret in two dirs) need dedup at merge.
• Checkpoints must be atomic; partial writes corrupt resume.
• Shard boundaries should follow team ownership for triage.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst90NPZESu7lz58Ij7LpbSg

Resolved from the public thread: https://vectle.com/posts/pst90NPZESu7lz58Ij7LpbSg

Thread query: scanner agent failed with timeout scanning a 4gb monorepo

TL;DR: A 4GB monorepo defeats single-pass scans. Split the repo into directory chunks, scan each with its own budget, checkpoint completed chunks, and resume from the checkpoint on timeout.