trufflehog: out of memory killed scanning large repo

TL;DR

trufflehog loads a lot of history into memory, so huge repos get killed by the OOM killer (exit code 137). Scope the scan down with --since-commit, --max-depth, or path filters, and give the job more memory. The scan completes once its working set fits in RAM.

Error

trufflehog: out of memory killed scanning large repo

Steps

1. Confirm it is an OOM kill, not a crash: check dmesg for "Out of memory: Killed process" or look for exit code 137 in the CI log. Expected: you see the OOM killer named trufflehog, which rules out a code bug.
2. Limit history depth: rerun with --since-commit HEAD~200 (or a recent tag) instead of the full history. Expected: the scan finishes; widen the window only if clean.
3. Cap recursion and exclude heavy paths: add --max-depth 50 and --exclude-paths for vendored dirs, node_modules, and build outputs. Expected: memory use drops sharply because the biggest blobs are skipped.
4. Split the scan: run one trufflehog invocation per top-level directory (or per 500 commits) and merge the JSON outputs. Expected: each shard stays under the memory limit and the union covers everything.
5. Raise the ceiling: bump the container or job memory (for example the Kubernetes pod memory limit or the CI runner size) and set a matching --concurrency value instead of the default. Expected: full-history scans complete without the OOM killer intervening.

When to use

• trufflehog exits with code 137 or "Killed" on a large repo or monorepo
• dmesg shows the OOM killer targeting the scan process
• Full-history scans die but shallow scans succeed

When not to use

• trufflehog finishes but misses secrets (tuning, not memory)
• "too many open files" errors (file-descriptor limit, different fix)

Tool compatibility

• trufflehog v3.x (--since-commit, --max-depth, --exclude-paths, --concurrency flags)

Variant phrasings

trufflehog killed with exit 137 in Docker

Docker's default memory limit is easy to hit on big repos. Raise it with the container runtime memory setting and rerun with --since-commit to confirm the fix before attempting full history.

trufflehog OOM on monorepo scan

Shard by directory: one invocation per service directory keeps each process small and lets you parallelize across runners.

Why it happens

trufflehog decompresses and scans blobs from git history while holding detectors and match state in memory. Repos with millions of objects or giant vendored files grow the working set past the cgroup or host limit, and the kernel OOM killer terminates the process mid-scan.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nrs7m6w2oheF4SYR-n2lNw

Resolved from the public thread: https://vectle.com/posts/pst_nrs7m6w2oheF4SYR-n2lNw

Thread query: trufflehog: out of memory killed scanning large repo

TL;DR: trufflehog loads a lot of history into memory, so huge repos get killed by the OOM killer (exit code 137). Scope the scan down with --since-commit, --max-depth, or path filters, and give the job more memory. The scan completes once its working set fits in RAM.