A service runs for three weeks and then starts returning 500s. The log has one line worth reading:
OSError: [Errno 24] Too many open files The reflex is to hunt for a leak. Sometimes that is the right answer. Often it is not. The process is simply holding as many descriptors as it is allowed to hold, and nobody ever checked what that number was.
The error is precise about which limit was hit, but vague about why. Linux has at least four different ceilings that produce the same family of symptoms: the per-process soft limit, the per-process hard limit, the kernel's ceiling on that hard limit, and the system-wide file limit. Add select() and inotify and you have two more traps that produce equally confusing failures.
Here is how the numbers actually fit together, how to reproduce the failure on purpose, and how to raise the right one without papering over a real leak.
Two errors, two scopes
The first useful split is the errno:
| errno | Name | Scope | Counter to inspect |
|---|---|---|---|
| 24 | EMFILE | Per process | /proc/PID/limits |
| 23 | ENFILE | System-wide | /proc/sys/fs/file-nr |
EMFILE means this process tried to allocate a descriptor number at or above its RLIMIT_NOFILE. ENFILE means the kernel ran out of file handles globally, regardless of what any single process is allowed.
Most "too many open files" incidents are EMFILE. ENFILE is rarer and usually points at a system-wide flood or a very low fs.file-max, not at one badly behaved service.
A descriptor is a handle, not a file
It is tempting to read "open files" as "files on disk". That is wrong, and it makes the counts confusing.
A file descriptor is an index into a per-process table. Each entry points at an open file description in the kernel. The same description can be referenced by several descriptors after dup(), and a descriptor can refer to almost anything:
- regular files and directories
- TCP and Unix sockets
- pipes and FIFOs
- character and block devices
- epoll instances, eventfd, signalfd, timerfd
- inotify instances, pidfd, memfd
- files passed in from another process over a Unix socket
ls -l /proc/$PID/fd The listing is the actual table. A healthy web server might show a few log files, a listening socket, hundreds of socket:[...] entries, and a couple of anon_inode:[eventpoll] entries. All of them compete for the same numeric limit.
Two consequences follow.
First, fork() copies the descriptor table but shares the underlying open file descriptions. Closing a descriptor in the child does not close the parent's copy. A forking service can therefore have many processes each holding a descriptor for the same socket.
Second, since Linux 4.5, RLIMIT_NOFILE also caps how many descriptors an unprivileged process may have "in flight" to other processes via SCM_RIGHTS on Unix sockets. That is an edge case, but it explains why a descriptor-passing daemon can hit EMFILE without a table that looks full.
The soft limit is the one that bites
Every process has two values for RLIMIT_NOFILE:
- soft limit — the value enforced right now
- hard limit — the ceiling the soft limit can be raised to
ulimit -Sn # soft ulimit -Hn # hard An unprivileged process can lower either limit and can raise the soft limit up to the hard limit. Raising the hard limit requires CAP_SYS_RESOURCE. The kernel also refuses to set the hard limit above /proc/sys/fs/nr_open, which defaults to 1048576 and is enforced for privileged processes too (setrlimit(2), proc_sys_fs(5)).
This is where systemd's default explains a lot of surprise:
DefaultLimitNOFILE=1024:524288 That is the documented manager default in systemd-system.conf(5): a soft limit of 1024 and a hard limit of 524288 for processes started by units. A service that never overrides it starts with 1024 usable descriptors, even though ulimit -Hn in a login shell might show half a million.
The application is allowed to raise its own soft limit to 524288. Some applications do that at startup; many do not, and they are the ones that fall over at roughly a thousand connections.
One more detail from the same man page: PID 1 bumps RLIMIT_NOFILE for itself, then reverts the limit for its children. So this:
cat /proc/1/limits | grep 'open files' tells you what the manager allows itself, not what your service gets. Always inspect the service process.
PID=$(systemctl show -p MainPID --value myapp.service) cat /proc/$PID/limits | grep -i 'open files' The output has two numbers:
Max open files 1024 524288 files The first is the soft limit that is actually enforced. The second is the hard ceiling.
Reproduce EMFILE in ten lines
Guessing is unnecessary. Lower the limit in a subshell and open descriptors until the kernel says no.
# /tmp/fd_lab.py import errno, os fds = [] try: while True: fds.append(os.open(os.devnull, os.O_RDONLY)) except OSError as e: print(f"opened {len(fds)} descriptors; errno {e.errno} {errno.errorcode[e.errno]}") print("last fd:", fds[-1] if fds else None) finally: for fd in fds: os.close(fd) bash -c 'ulimit -n 64; python3 /tmp/fd_lab.py' On Debian 13 with systemd 257, the run printed:
opened 59 descriptors; errno 24 EMFILE last fd: 63 The exact number of successful opens varies with interpreter startup and inherited descriptors. The last descriptor is the interesting part: 63 is one less than the limit of 64. The kernel will not hand out descriptor number 64.
That is the precise meaning of RLIMIT_NOFILE: it is one greater than the highest descriptor number the process may hold. It is not a byte budget, a disk quota, or a count of files on disk.
Because Linux allocates the lowest free descriptor number, a process fails when every slot below the limit is occupied, regardless of what those slots point at. Holes left by close() are reused immediately, so the count of open descriptors is normally close to the limit when EMFILE appears. A leak is one reason the table fills up; a low limit is another.
Read the process before changing anything
Four commands answer most of the question.
PID=$(systemctl show -p MainPID --value myapp.service) # 1. What is the process actually allowed? cat /proc/$PID/limits | grep -i 'open files' # 2. How many descriptors are open? ls /proc/$PID/fd | wc -l # 3. What kinds of objects are they? ls -l /proc/$PID/fd 2>/dev/null \ | awk '{print $NF}' \ | sed 's/\[.*//' \ | sort | uniq -c | sort -rn | head # 4. Is it still growing? for i in 1 2 3; do ls /proc/$PID/fd | wc -l; sleep 10; done The third command collapses socket:[12345] to socket: and leaves paths intact, which is usually enough to distinguish a connection pile-up from a file-handle leak:
412 socket: 18 /var/lib/myapp/data.db 4 anon_inode:[eventpoll] If the count is flat across the three samples and the process is at its soft limit, you have a sizing problem. If it climbs monotonically and never comes down, you have a leak — raise the limit if you need breathing room, but fix the leak.
Also check the system-wide counter:
cat /proc/sys/fs/file-nr It returns three numbers: allocated file handles, free handles, and the maximum. Since Linux 2.6 the middle number is always zero because the kernel deallocates handles instead of caching them. If the first number is close to the third, you are in ENFILE territory and the per-process limit is not your problem.
Size by connections, not by superstition
Copying LimitNOFILE=65535 from a blog post is not sizing. A useful starting point is:
soft limit >= idle descriptors + (peak concurrent connections x descriptors per connection) + headroom For a network daemon, one accepted connection is normally one descriptor. A reverse proxy usually needs two per proxied request: one for the client and one for the upstream. HTTP/2 multiplexes many requests over one connection, so streams do not multiply descriptors the way requests do.
nginx makes the relationship explicit. worker_connections is the maximum number of simultaneous connections per worker, and the documentation notes that it includes connections to proxied servers, not just clients. It also notes that the real number cannot exceed the process's open-file limit. That is why the two directives belong together:
worker_processes auto; events { worker_connections 8192; } worker_rlimit_nofile 16384; worker_rlimit_nofile changes RLIMIT_NOFILE for worker processes without restarting the master. If it is lower than worker_connections, the descriptor limit wins.
Databases deserve a separate note. PostgreSQL uses one backend process per client connection. Each backend has its own descriptor table and its own inherited limit, so raising LimitNOFILE on the unit raises the ceiling for every backend, but the limit is not a single pool shared by the service. The same per-process arithmetic applies to a pool of application workers: ten workers at 1024 each can collectively hold far more than 1024 descriptors.
That is also why there is no per-cgroup descriptor limit to reach for. cgroup v2 has controllers for CPU, memory, I/O, PIDs, and a few other resources, but no file-descriptor controller. systemd's own documentation points out that file descriptors are accounted like any other form of memory and suggests MemoryMax= for overall service memory, rather than lowering the hard descriptor limit. A runaway descriptor count can therefore count against the service's memory budget as well as producing EMFILE.
The fix, in a drop-in
For a system service, do not edit the vendor unit. Add a drop-in:
sudo systemctl edit myapp.service In the editor:
[Service] LimitNOFILE=65536:524288 Apply and restart:
sudo systemctl daemon-reload sudo systemctl restart myapp.service The soft:hard form matters. A single value sets both limits to the same number:
LimitNOFILE=65536 That works, but it removes the application's ability to raise its own soft limit later. Keeping the hard limit at 524288 preserves that option, and matches systemd's own default shape.
Verify against the running process, not the unit file:
PID=$(systemctl show -p MainPID --value myapp.service) grep -i 'open files' /proc/$PID/limits For a temporary change without a restart, prlimit can adjust a live process (same UID or CAP_SYS_RESOURCE required):
sudo prlimit --pid $PID --nofile=65536:524288 That is a tourniquet, not a fix. It disappears on restart, and an application that has already entered an error path may not recover just because the limit changed.
For user services, put the override in the user unit with systemctl --user edit. Remember that a user unit cannot raise its limit above the hard limit the user manager started with. To change that ceiling, set DefaultLimitNOFILE= in /etc/systemd/user.conf and restart the user manager or session; the manager reads that file at startup.
In containers, the OCI runtime sets the limit for the container process. Docker and Podman both accept the same flag:
podman run --ulimit nofile=65536:524288 myimage If the runtime does not set one, the container inherits whatever the runtime process had. Check with ulimit -n inside the container instead of assuming the host value applies.
The select() ceiling
Raising the limit above 1024 can make an old application worse, not better.
select() stores descriptors in fd_set, a fixed-size bitmask. On glibc, FD_SETSIZE is 1024. The man page is blunt about it: select() can monitor only descriptor numbers below FD_SETSIZE, calls that "an unreasonably low limit for many modern applications", and says modern applications should use poll(2) or epoll(7).
The failure mode is nasty. An application that raises its soft limit and then passes descriptor 1500 to FD_SET() is writing outside the bitmask. You do not get a clean EMFILE; you get memory corruption, a missed event, or a crash that looks unrelated.
If you are about to raise a limit past 1024, check what the process actually uses while it is handling traffic:
sudo strace -f -e trace=select,pselect6,poll,ppoll,epoll_wait -p $PID If select or pselect6 is in the output, the safe move is to keep the descriptor limit at or below 1024 or fix the application to use poll/epoll. poll() and epoll do not have the FD_SETSIZE limitation.
The other "too many files": inotify
File-watching tools — editors, sync clients, dev servers, backup agents — can fail with a message that sounds like a descriptor problem but is not:
ENOSPC: No space left on device, watch ... Inotify has its own per-user limits (inotify(7), inotify_add_watch(2)):
-
fs.inotify.max_user_watches— total watches per real UID -
fs.inotify.max_user_instances— inotify instances per real UID -
fs.inotify.max_queued_events— events queued per instance
inotify_add_watch() returns ENOSPC when the watch limit is reached. Each inotify instance also consumes one file descriptor, so a process can hit both limits, but they are separate counters.
Inspect them:
cat /proc/sys/fs/inotify/max_user_watches cat /proc/sys/fs/inotify/max_user_instances # Count inotify watches held by one process awk '/^inotify / {n++} END {print n+0}' /proc/$PID/fdinfo/* 2>/dev/null Raise the watch limit temporarily:
sudo sysctl -w fs.inotify.max_user_watches=524288 Persist it:
echo 'fs.inotify.max_user_watches = 524288' \ | sudo tee /etc/sysctl.d/99-inotify.conf sudo sysctl --system The per-UID detail is the trap. Two services running as the same user share one watch budget. Giving a chatty file-watching service its own user is often better than raising the global number again.
A short checklist
- Read the errno: 24 is per-process, 23 is system-wide.
- Read
/proc/PID/limits, not/proc/1/limits. - Count
/proc/PID/fdand break it down by type. - Sample the count over time to separate a leak from a sizing problem.
- Check whether the app uses
select()before raising past 1024. - Set
LimitNOFILE=soft:hard, not a single value, when you want headroom. - Restart and verify the running process, not just the unit file.
- If the error is
ENOSPCfrom a file watcher, look at inotify, not descriptors.
The one-line summary: "too many open files" is a statement about a number, and the first job is to find out which number. Once you know whether it is the soft limit, the hard limit, the system-wide maximum, or a completely different subsystem wearing the same error message, the fix is usually boring — which is exactly what you want from an infrastructure problem.
References
-
getrlimit(2)— man7.org -
setrlimit(2)— man7.org -
proc_sys_fs(5)— man7.org -
systemd.exec(5)— man7.org -
systemd-system.conf(5)— man7.org -
systemd-user.conf(5)— man7.org -
inotify(7)— man7.org -
inotify_add_watch(2)— man7.org -
select(2)— man7.org - nginx core module,
worker_connectionsandworker_rlimit_nofile— nginx.org - Podman
--ulimit— docs.podman.io
