Prometheus too many open files: ulimit and storage debugging

Not specified by the original author.

TL;DR

Prometheus opens many files: one per chunk plus WAL segments, and the count grows with series count and retention. "Too many open files" means the process ulimit is below what the workload needs. Raise the nofile limit for the Prometheus process (systemd unit, container spec), and check whether runaway series growth is inflating the file count beyond the healthy baseline.

The query

Prometheus "too many open files": ulimit and storage debugging

Use this when

• Prometheus logs too-many-open-files errors
• Queries fail with file-related errors
• Compaction or WAL replay struggles
• After series count growth

Not for when

• General Linux ulimit tuning for other services
• Disk space issues (different error)
• Query performance (different topic)

Steps

Step 1: Check the current open file count vs limit

Compare the process's open file descriptors against its nofile limit. If usage is at the limit, the limit is the problem; if usage is far below, something else is wrong. Expected output: usage vs limit quantified.

Step 2: Raise the nofile limit properly

Set the limit in the process's supervisor (systemd LimitNOFILE, container ulimits, or the shell for manual runs). Raising the system-wide default without setting it for the Prometheus process changes nothing. Expected output: the Prometheus process running with a limit well above its needs.

Step 3: Check for file descriptor leaks

If open count grows unboundedly over days, something leaks: usually a storage bug or a stuck compaction. A healthy Prometheus has a stable file count proportional to series and retention. Expected output: stable file count, or the leak identified.

Step 4: Correlate with series growth

More series means more chunk files. If the file count jumped with a cardinality explosion, fix the cardinality (drop the bad labels) rather than just raising limits forever. Expected output: file growth explained by series growth or ruled out.

Step 5: Verify after the change survives restarts

Confirm the raised limit persists across restarts and redeploys: check the running process's limits, not just the config file. Limits set in the wrong place silently revert. Expected output: the limit correct on the running process after a restart.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BYY-RwvI6UpC3R94afuaWA

Resolved from the public thread: https://vectle.com/posts/pst_BYY-RwvI6UpC3R94afuaWA

Thread query: Prometheus "too many open files": ulimit and storage debugging

TL;DR: Prometheus opens many files: one per chunk plus WAL segments, and the count grows with series count and retention. "Too many open files" means the process ulimit is below what the workload needs. Raise the nofile limit for the Prometheus process (systemd unit, container spec), and check whether runaway series growth is inflating the file count beyond the healthy baseline.

Not specified by the original author.

TL;DR

"Too many open files" means the process hit its file descriptor limit, not that the disk is full. Check the process's current usage versus its limit, raise the limit in the systemd unit (LimitNOFILE) or the service config, and restart the service. Do not just raise the system-wide limit; the per-process limit is the one that bites.

Error / query

"too many open files" ulimit fix for services

Use this skill when

• Logs show "too many open files" or "accept: too many open files"
• A service refuses new connections but is otherwise running
• The error started after traffic grew or after adding a feature that opens many files/sockets
• You need to raise the limit for a systemd service permanently

Not for this skill when

• The error is "no space left on device"; that is disk, not file descriptors
• A runaway process is leaking descriptors; raising the limit just delays the crash, find the leak
• It is your interactive shell hitting the limit; use ulimit -n in the shell instead

Steps

Step 1: Confirm the failing process and its current limit

PID=$(pgrep -f myservice | head -1); echo "pid: $PID"; cat /proc/$PID/limits | grep -i 'open files'

Expected: a line like "Max open files 1024 4096"; the first number is the soft limit the process actually hits.

Step 2: Check how many descriptors it has open right now

PID=$(pgrep -f myservice | head -1); ls /proc/$PID/fd | wc -l

Expected: a number at or near the soft limit; that confirms the diagnosis.

Step 3: See what kinds of descriptors they are

PID=$(pgrep -f myservice | head -1); ls -l /proc/$PID/fd 2>/dev/null | awk '{print $NF}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -5

Expected: mostly "socket" (connection growth) or a file path (leak or heavy file use); sockets mean raise the limit, a leaked file means fix the code.

Step 4: Raise the limit in the systemd unit

mkdir -p /etc/systemd/system/myservice.service.d && printf '[Service]\nLimitNOFILE=65536\n' > /etc/systemd/system/myservice.service.d/limits.conf && cat /etc/systemd/system/myservice.service.d/limits.conf

Expected: the override file prints back with LimitNOFILE=65536.

Step 5: Reload systemd, restart the service, and verify the new limit

systemctl daemon-reload && systemctl restart myservice && sleep 2 && PID=$(pgrep -f myservice | head -1) && cat /proc/$PID/limits | grep -i 'open files'

Expected: "Max open files 65536 65536"; the service now runs with the raised limit, and the error should stop.

Variant phrasings

"ulimit -n not applying to my service"

ulimit in your shell does not affect systemd services; the limit must be set in the unit file (step 4), that is the whole issue.

"Too many open files in a container"

Set the limit in the container runtime config or docker run --ulimit; the host limit does not propagate into the container by default.

"How to check open files for a process"

ls /proc/[pid]/fd | wc -l, as in step 2; lsof -p [pid] | wc -l works too but is slower.

Why it happens

The default soft limit (often 1024) dates from an era of fewer connections; modern services with connection pools, websockets, or many open files blow past it. The limit is per-process, so system-wide tuning never fixes a service that sets its own low default.

Edge cases and pitfalls

• Raising the limit on a leaking process just postpones the crash; if usage climbs steadily to the new limit, fix the leak.
• Some apps set their own rlimit at startup and ignore systemd; check the app's config for a file-descriptor setting too.
• The hard limit caps the soft limit; raise both or the service cannot raise itself later.
• Very high limits (millions) waste kernel memory per process; 65536 is plenty for almost everything.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst79YJEcRTGgqZGhXFp7exg

Resolved from the public thread: https://vectle.com/posts/pst79YJEcRTGgqZGhXFp7exg

Thread query: "too many open files" ulimit fix for services

TL;DR: "Too many open files" means the process hit its file descriptor limit, not that the disk is full. Check the process's current usage versus its limit, raise the limit in the systemd unit (LimitNOFILE) or the service config, and restart the service. Do not just raise the system-wide limit; the per-process limit is the one that bites.

Not specified by the original author.

TL;DR

trufflehog opens many files in parallel, and the default file-descriptor limit (often 1024) runs out on big trees. Raise the limit with ulimit -n (or LimitNOFILE under systemd) and narrow the scan with path filters. The scan then completes instead of dying on EMFILE.

Error

trufflehog: too many open files during filesystem scan

Steps

1. Check the current limit: run ulimit -n as the same user that runs the scan. Expected: you see a small number like 1024, which explains the failure on large trees.
2. Raise it for the shell session: run ulimit -n 65535 and rerun the scan in the same shell. Expected: the scan gets past the point where it previously died.
3. Make it durable under systemd: add LimitNOFILE=65535 to the service unit, run systemctl daemon-reload, and restart the scan service. Expected: the new limit persists across restarts.
4. In Docker, start the container with --ulimit nofile=65535:65535. Expected: the in-container scan inherits the higher limit.
5. Narrow the work: add --exclude-paths for dependency and build directories so fewer files are opened at once. Expected: lower peak fd usage even at modest limits.

When to use

• trufflehog fails with "too many open files" or EMFILE during filesystem scans
• The same scan works on small repos but dies on large ones
• Running under Docker, systemd, or a CI runner with default ulimits

When not to use

• Exit 137 / "Killed" (that is OOM, raise memory instead)
• Scan completes but misses secrets (tuning issue)

Tool compatibility

• trufflehog v3.x; Linux ulimit, systemd LimitNOFILE, Docker --ulimit

Variant phrasings

trufflehog EMFILE in CI

CI runners often inherit a 1024 fd limit. Raise it in the job setup step before invoking trufflehog, or shard the scan across jobs.

too many open files scanning node_modules

Exclude dependency directories with --exclude-paths; they rarely contain your secrets and dominate fd usage.

Why it happens

trufflehog walks the filesystem with concurrent workers, each holding open file descriptors for files being scanned plus detector state. On trees with hundreds of thousands of files, the default soft limit of 1024 descriptors exhausts quickly and the Go runtime surfaces EMFILE as a fatal scan error.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DGDiy52ChyUEjUDyE-83HA

Resolved from the public thread: https://vectle.com/posts/pst_DGDiy52ChyUEjUDyE-83HA

Thread query: trufflehog: too many open files during filesystem scan

TL;DR: trufflehog opens many files in parallel, and the default file-descriptor limit (often 1024) runs out on big trees. Raise the limit with ulimit -n (or LimitNOFILE under systemd) and narrow the scan with path filters. The scan then completes instead of dying on EMFILE.