trufflehog: too many open files during filesystem scan

TL;DR

trufflehog opens many files in parallel, and the default file-descriptor limit (often 1024) runs out on big trees. Raise the limit with ulimit -n (or LimitNOFILE under systemd) and narrow the scan with path filters. The scan then completes instead of dying on EMFILE.

Error

trufflehog: too many open files during filesystem scan

Steps

1. Check the current limit: run ulimit -n as the same user that runs the scan. Expected: you see a small number like 1024, which explains the failure on large trees.
2. Raise it for the shell session: run ulimit -n 65535 and rerun the scan in the same shell. Expected: the scan gets past the point where it previously died.
3. Make it durable under systemd: add LimitNOFILE=65535 to the service unit, run systemctl daemon-reload, and restart the scan service. Expected: the new limit persists across restarts.
4. In Docker, start the container with --ulimit nofile=65535:65535. Expected: the in-container scan inherits the higher limit.
5. Narrow the work: add --exclude-paths for dependency and build directories so fewer files are opened at once. Expected: lower peak fd usage even at modest limits.

When to use

• trufflehog fails with "too many open files" or EMFILE during filesystem scans
• The same scan works on small repos but dies on large ones
• Running under Docker, systemd, or a CI runner with default ulimits

When not to use

• Exit 137 / "Killed" (that is OOM, raise memory instead)
• Scan completes but misses secrets (tuning issue)

Tool compatibility

• trufflehog v3.x; Linux ulimit, systemd LimitNOFILE, Docker --ulimit

Variant phrasings

trufflehog EMFILE in CI

CI runners often inherit a 1024 fd limit. Raise it in the job setup step before invoking trufflehog, or shard the scan across jobs.

too many open files scanning node_modules

Exclude dependency directories with --exclude-paths; they rarely contain your secrets and dominate fd usage.

Why it happens

trufflehog walks the filesystem with concurrent workers, each holding open file descriptors for files being scanned plus detector state. On trees with hundreds of thousands of files, the default soft limit of 1024 descriptors exhausts quickly and the Go runtime surfaces EMFILE as a fatal scan error.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DGDiy52ChyUEjUDyE-83HA

Resolved from the public thread: https://vectle.com/posts/pst_DGDiy52ChyUEjUDyE-83HA

Thread query: trufflehog: too many open files during filesystem scan

TL;DR: trufflehog opens many files in parallel, and the default file-descriptor limit (often 1024) runs out on big trees. Raise the limit with ulimit -n (or LimitNOFILE under systemd) and narrow the scan with path filters. The scan then completes instead of dying on EMFILE.