too many open files" ulimit fix for services

TL;DR

"Too many open files" means the process hit its file descriptor limit, not that the disk is full. Check the process's current usage versus its limit, raise the limit in the systemd unit (LimitNOFILE) or the service config, and restart the service. Do not just raise the system-wide limit; the per-process limit is the one that bites.

Error / query

"too many open files" ulimit fix for services

Use this skill when

• Logs show "too many open files" or "accept: too many open files"
• A service refuses new connections but is otherwise running
• The error started after traffic grew or after adding a feature that opens many files/sockets
• You need to raise the limit for a systemd service permanently

Not for this skill when

• The error is "no space left on device"; that is disk, not file descriptors
• A runaway process is leaking descriptors; raising the limit just delays the crash, find the leak
• It is your interactive shell hitting the limit; use ulimit -n in the shell instead

Steps

Step 1: Confirm the failing process and its current limit

PID=$(pgrep -f myservice | head -1); echo "pid: $PID"; cat /proc/$PID/limits | grep -i 'open files'

Expected: a line like "Max open files 1024 4096"; the first number is the soft limit the process actually hits.

Step 2: Check how many descriptors it has open right now

PID=$(pgrep -f myservice | head -1); ls /proc/$PID/fd | wc -l

Expected: a number at or near the soft limit; that confirms the diagnosis.

Step 3: See what kinds of descriptors they are

PID=$(pgrep -f myservice | head -1); ls -l /proc/$PID/fd 2>/dev/null | awk '{print $NF}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -5

Expected: mostly "socket" (connection growth) or a file path (leak or heavy file use); sockets mean raise the limit, a leaked file means fix the code.

Step 4: Raise the limit in the systemd unit

mkdir -p /etc/systemd/system/myservice.service.d && printf '[Service]\nLimitNOFILE=65536\n' > /etc/systemd/system/myservice.service.d/limits.conf && cat /etc/systemd/system/myservice.service.d/limits.conf

Expected: the override file prints back with LimitNOFILE=65536.

Step 5: Reload systemd, restart the service, and verify the new limit

systemctl daemon-reload && systemctl restart myservice && sleep 2 && PID=$(pgrep -f myservice | head -1) && cat /proc/$PID/limits | grep -i 'open files'

Expected: "Max open files 65536 65536"; the service now runs with the raised limit, and the error should stop.

Variant phrasings

"ulimit -n not applying to my service"

ulimit in your shell does not affect systemd services; the limit must be set in the unit file (step 4), that is the whole issue.

"Too many open files in a container"

Set the limit in the container runtime config or docker run --ulimit; the host limit does not propagate into the container by default.

"How to check open files for a process"

ls /proc/[pid]/fd | wc -l, as in step 2; lsof -p [pid] | wc -l works too but is slower.

Why it happens

The default soft limit (often 1024) dates from an era of fewer connections; modern services with connection pools, websockets, or many open files blow past it. The limit is per-process, so system-wide tuning never fixes a service that sets its own low default.

Edge cases and pitfalls

• Raising the limit on a leaking process just postpones the crash; if usage climbs steadily to the new limit, fix the leak.
• Some apps set their own rlimit at startup and ignore systemd; check the app's config for a file-descriptor setting too.
• The hard limit caps the soft limit; raise both or the service cannot raise itself later.
• Very high limits (millions) waste kernel memory per process; 65536 is plenty for almost everything.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst79YJEcRTGgqZGhXFp7exg