agent crashed with error on a repo with 1m secrets-scan findings

TL;DR

A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Error

agent crashed with error on a repo with 1m secrets-scan findings

Steps

1. Stream findings from the scanner instead of loading the report. Expected: constant memory.
2. Aggregate by rule, path, and fingerprint to collapse duplicates. Expected: orders of magnitude fewer rows.
3. Triage the aggregates: fix the systemic causes (one bad pattern can be half the findings). Expected: the biggest wins first.
4. Sample within aggregates for manual review. Expected: representative coverage.
5. Track aggregate counts over time as the metric, not individual findings. Expected: measurable progress.

When to use

• Agents crashing on huge finding counts.
• Triage at scale.

When not to use

• The scan itself failing.
• Small finding counts (triage directly).

Tool compatibility

• Streaming JSON; aggregation pipelines.

Variant phrasings

too many findings crashed the agent

Aggregate.

1m findings OOM

Stream and collapse.

Why it happens

Agents written for dozens of findings buffer everything; systemic issues produce millions, and memory loses.

Edge cases

• Aggregation must preserve at least one example per group for review.
• Counts need stable fingerprints or they churn between runs.
• A million findings usually means a few root causes; find them first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Thread query: agent crashed with error on a repo with 1m secrets-scan findings

TL;DR: A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.