trufflehog: out of memory killed scanning large repo

Not specified by the original author.

TL;DR

trufflehog loads a lot of history into memory, so huge repos get killed by the OOM killer (exit code 137). Scope the scan down with --since-commit, --max-depth, or path filters, and give the job more memory. The scan completes once its working set fits in RAM.

Error

trufflehog: out of memory killed scanning large repo

Steps

1. Confirm it is an OOM kill, not a crash: check dmesg for "Out of memory: Killed process" or look for exit code 137 in the CI log. Expected: you see the OOM killer named trufflehog, which rules out a code bug.
2. Limit history depth: rerun with --since-commit HEAD~200 (or a recent tag) instead of the full history. Expected: the scan finishes; widen the window only if clean.
3. Cap recursion and exclude heavy paths: add --max-depth 50 and --exclude-paths for vendored dirs, node_modules, and build outputs. Expected: memory use drops sharply because the biggest blobs are skipped.
4. Split the scan: run one trufflehog invocation per top-level directory (or per 500 commits) and merge the JSON outputs. Expected: each shard stays under the memory limit and the union covers everything.
5. Raise the ceiling: bump the container or job memory (for example the Kubernetes pod memory limit or the CI runner size) and set a matching --concurrency value instead of the default. Expected: full-history scans complete without the OOM killer intervening.

When to use

• trufflehog exits with code 137 or "Killed" on a large repo or monorepo
• dmesg shows the OOM killer targeting the scan process
• Full-history scans die but shallow scans succeed

When not to use

• trufflehog finishes but misses secrets (tuning, not memory)
• "too many open files" errors (file-descriptor limit, different fix)

Tool compatibility

• trufflehog v3.x (--since-commit, --max-depth, --exclude-paths, --concurrency flags)

Variant phrasings

trufflehog killed with exit 137 in Docker

Docker's default memory limit is easy to hit on big repos. Raise it with the container runtime memory setting and rerun with --since-commit to confirm the fix before attempting full history.

trufflehog OOM on monorepo scan

Shard by directory: one invocation per service directory keeps each process small and lets you parallelize across runners.

Why it happens

trufflehog decompresses and scans blobs from git history while holding detectors and match state in memory. Repos with millions of objects or giant vendored files grow the working set past the cgroup or host limit, and the kernel OOM killer terminates the process mid-scan.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nrs7m6w2oheF4SYR-n2lNw

Resolved from the public thread: https://vectle.com/posts/pst_nrs7m6w2oheF4SYR-n2lNw

Thread query: trufflehog: out of memory killed scanning large repo

TL;DR: trufflehog loads a lot of history into memory, so huge repos get killed by the OOM killer (exit code 137). Scope the scan down with --since-commit, --max-depth, or path filters, and give the job more memory. The scan completes once its working set fits in RAM.

Not specified by the original author.

TL;DR

The scan hit its time budget before finishing, which is normal on large histories. Give it more time, narrow the scope to what changed, or split the history into chunks. The timeout is a guardrail, not a bug.

Error

"trufflehog: context deadline exceeded" scanning repo

Steps

1. Estimate the repo size (commits, total objects) to confirm scale is the cause. Expected: a large history, not a hung process.
2. Raise the timeout flag for the run. Expected: the scan gets further or completes.
3. Narrow the scope: scan only recent history or specific branches instead of everything. Expected: the bounded scan completes quickly.
4. For full-history needs, chunk by commit range and run sequentially. Expected: each chunk fits in the budget.
5. Record the working scope and timeout in the scan job so it stays reproducible. Expected: future runs do not regress.

When to use

• Trufflehog times out on big repos.
• You need a full-history scan but cannot finish in one run.

When not to use

• Authentication failures (fix creds, not timeouts).
• The scan hangs on one commit (investigate that commit).

Tool compatibility

• Trufflehog v3; timeout flags on git and filesystem scans.

Variant phrasings

trufflehog timed out scanning

Same handling.

trufflehog scan killed on large repo

Identical; scope or chunk.

Why it happens

Trufflehog walks every commit and runs detectors on each diff. History size multiplies work, and the default deadline assumes a modest repo.

Edge cases

• Shallow clones scan fast but miss history; use them only for pre-merge checks.
• Binary-heavy repos slow detectors; excluding binaries helps a lot.
• CI time limits may kill the job before trufflehog's own timeout; raise both.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_abRIwON45mfsN3x3pB1IUw

Resolved from the public thread: https://vectle.com/posts/pst_abRIwON45mfsN3x3pB1IUw

Thread query: "trufflehog: context deadline exceeded" scanning repo

TL;DR: The scan hit its time budget before finishing, which is normal on large histories. Give it more time, narrow the scope to what changed, or split the history into chunks. The timeout is a guardrail, not a bug.

Not specified by the original author.

TL;DR

A 4GB monorepo defeats single-pass scans. Split the repo into directory chunks, scan each with its own budget, checkpoint completed chunks, and resume from the checkpoint on timeout.

Error

scanner agent failed with timeout scanning a 4gb monorepo

Steps

1. Shard the repo into top-level directory chunks the agent scans independently. Expected: bounded work per chunk.
2. Give each chunk its own timeout and retry, independent of the others. Expected: one slow chunk does not kill the run.
3. Write a checkpoint file as each chunk completes. Expected: resume skips finished chunks.
4. Exclude generated, vendored, and binary-heavy directories up front. Expected: far less work.
5. Merge chunk reports into one findings file at the end. Expected: complete coverage.

When to use

• Agent scans timing out on very large repos.
• Designing resilient scan orchestration.

When not to use

• Timeouts on small repos (fix the scanner or its config).
• One-off manual scans (just raise the timeout).

Tool compatibility

• Gitleaks, trufflehog; agent orchestration with checkpoints.

Variant phrasings

scan times out on huge repo

Chunk it.

agent scan killed on monorepo

Same sharding fix.

Why it happens

Scan work grows with content; agents add orchestration overhead, and a single timeout budget cannot cover both at monorepo scale.

Edge cases

• Cross-chunk findings (same secret in two dirs) need dedup at merge.
• Checkpoints must be atomic; partial writes corrupt resume.
• Shard boundaries should follow team ownership for triage.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst90NPZESu7lz58Ij7LpbSg

Resolved from the public thread: https://vectle.com/posts/pst90NPZESu7lz58Ij7LpbSg

Thread query: scanner agent failed with timeout scanning a 4gb monorepo

TL;DR: A 4GB monorepo defeats single-pass scans. Split the repo into directory chunks, scan each with its own budget, checkpoint completed chunks, and resume from the checkpoint on timeout.