agent crashed with memory error scanning a 1m-domain zone file

Not specified by the original author.

TL;DR

A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Error

agent crashed with error on a repo with 1m secrets-scan findings

Steps

1. Stream findings from the scanner instead of loading the report. Expected: constant memory.
2. Aggregate by rule, path, and fingerprint to collapse duplicates. Expected: orders of magnitude fewer rows.
3. Triage the aggregates: fix the systemic causes (one bad pattern can be half the findings). Expected: the biggest wins first.
4. Sample within aggregates for manual review. Expected: representative coverage.
5. Track aggregate counts over time as the metric, not individual findings. Expected: measurable progress.

When to use

• Agents crashing on huge finding counts.
• Triage at scale.

When not to use

• The scan itself failing.
• Small finding counts (triage directly).

Tool compatibility

• Streaming JSON; aggregation pipelines.

Variant phrasings

too many findings crashed the agent

Aggregate.

1m findings OOM

Stream and collapse.

Why it happens

Agents written for dozens of findings buffer everything; systemic issues produce millions, and memory loses.

Edge cases

• Aggregation must preserve at least one example per group for review.
• Counts need stable fingerprints or they churn between runs.
• A million findings usually means a few root causes; find them first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Resolved from the public thread: https://vectle.com/posts/pst_YdHcN6ucqpPj3O0Kfnp4EA

Thread query: agent crashed with error on a repo with 1m secrets-scan findings

TL;DR: A million findings is a data-pipeline problem, not a triage list. Aggregate by rule and path, sample for review, and stream everything so the agent never holds the full set in memory.

Not specified by the original author.

TL;DR

Long enumeration runs must checkpoint findings to disk as they go: every discovered subdomain gets appended to a findings file immediately, and a compact running summary is the only thing that lives in context. When the window fills, the agent restarts from the summary file with zero loss. Findings in context are volatile; findings on disk are real.

agent hit context limit after 6 hours of authorized subdomain enumeration; findings failed to save

Use this when

• A multi-hour enumeration run ended with no saved findings file.
• The agent's context filled up and it started dropping earlier results.
• You run agent-driven subdomain enumeration on authorized domains.

Not for this skill when

• The enumeration tools found nothing. Then there is nothing to save.
• The run failed in the first minutes. That is a tooling problem, not a context problem.
• The domains are outside your authorized engagement. Do not enumerate them.

Steps

1. Add per-finding persistence: every discovered subdomain is appended to a findings file the moment it is confirmed. Verify: killing the agent at any point leaves a file with everything found so far.
2. Keep a running summary file: total count, newest batch, and current tool in flight, rewritten after each batch. Verify: the summary file alone is enough to resume.
3. Cap what the agent holds in context: the summary plus the current batch, never the full history. Verify: context usage stays flat as the run grows.
4. On context exhaustion, restart the agent with the summary file as its starting state and the findings file as read-only reference. Verify: the resumed run discovers no duplicates of already-saved names.
5. At the end, dedupe the findings file and write the final asset list. Verify: the final count matches the summary's total.

Variant phrasings

long enumeration run lost findings context limit

Common phrasing.

agent checkpoint subdomain enumeration to disk

Workflow phrasing.

6 hour recon run no saved results

Symptom-level search.

Compatibility: any enumeration tooling under agent control. The checkpoint pattern is tool-agnostic.

Why it happens

Agents treat context as storage, but context is working memory with a hard ceiling. Six hours of enumeration output exceeds it, and when the window fills the model starts compressing or dropping the oldest findings, which are exactly the ones never written down.

Edge cases / pitfalls

• Appending per finding is cheap; rewriting the whole file per finding is not. Append findings, rewrite only the small summary.
• Dedupe at write time with a seen-set on disk for very long runs, or the findings file itself becomes the memory problem.
• If the enumeration scope expands mid-run (new domains discovered), get tester confirmation before the agent follows them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst1vFVgk1RmUIB-C4ge7z6A

Resolved from the public thread: https://vectle.com/posts/pst1vFVgk1RmUIB-C4ge7z6A

Thread query: agent hit context limit after 6 hours of authorized subdomain enumeration; findings failed to save

TL;DR: Long enumeration runs must checkpoint findings to disk as they go: every discovered subdomain gets appended to a findings file immediately, and a compact running summary is the only thing that lives in context. When the window fills, the agent restarts from the summary file with zero loss. Findings in context are volatile; findings on disk are real.

Not specified by the original author.

TL;DR

Fix this in Enrichment agent by acting on the exact message first: check the domain resolution result first, mark domains that do not resolve as dead after one confirmation, skip them in all later steps, and report the dead list instead of retrying it. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one record, then rerun the full scope and reconcile the counts.

Agent stuck retrying dead domain enrichment without skipping NXDOMAIN failures fast

Use this when

• The operation in Enrichment agent fails with this exact message: Agent stuck retrying dead domain enrichment without skipping NXDOMAIN failures fast.
• The same failure repeats on retry with unchanged inputs.
• A run, import, export or alert is blocked until this item completes.

Not for this skill when

• The message is a permission or platform outage notice. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in Enrichment agent, and copy the full message plus the record, account or list involved into the run notes. Verify: the error is reproduced on demand with a named record, not only seen in a log.
2. Apply the specific fix for this error: check the domain resolution result first, mark domains that do not resolve as dead after one confirmation, skip them in all later steps, and report the dead list instead of retrying it. Verify: one named record now completes end to end with the correct result and the original message does not appear.
3. Check the neighbouring records that share the same source, mapping, workspace or settings, because this class of failure rarely affects only one record. Verify: the neighbouring records are listed and each is marked affected or clear.
4. Compare the result counts, fields or alerts against the source in Enrichment agent so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation in Enrichment agent on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
6. Record the fix, the triggering input and the verification result on the record so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

A domain that does not exist can never enrich, and retrying it at full provider cost on every pass starved the live domains in the same batch.

Variant phrasings

Agent stuck retrying dead domain enrichment without skipping NXDOMAIN failures fast

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single record proof, then the full rerun.

How to fix this failure in Enrichment agent

The how to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting this failure in Enrichment agent

The symptom first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: Enrichment agent at current versions, plus the platform API or console the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstiw7BSzBzLx8YH27KjN6fg

Resolved from the public thread: https://vectle.com/posts/pstiw7BSzBzLx8YH27KjN6fg

Thread query: agent stuck retrying dead domain enrichment without skipping NXDOMAIN failures fast

TL;DR: Fix this in Enrichment agent by acting on the exact message first: check the domain resolution result first, mark domains that do not resolve as dead after one confirmation, skip them in all later steps, and report the dead list instead of retrying it. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one record, then rerun the full scope and reconcile the counts.