EMFILE on Production: A Field Engineer's Guide to File Descriptor Limits
It hit at 2 AM. Not with a dramatic stack trace — just a sudden, silent drop in request throughput. Nginx was still serving pages, but new connections weren't being accepted. The error log told the story in one line: accept4() failed (24: Too many open files).
If you've ever seen this on a production server, you know the feeling: the panic of a midnight page, the half-remembered commands, the desperate hunt for which layer of limits is choking your process.
I've seen this error across dozens of servers — Node.js APIs, Python scrapers, Java microservices, Go web servers. Each time, the fix was the same: understand the three layers of Linux file descriptor limits, and tune them correctly. Let me walk you through what EMFILE really means, how to diagnose it fast, and the exact fixes that work in production.
What "File" Really Means in Linux
Here's the first thing most engineers miss: in Linux, a "file" is anything that can be read from or written to. This includes:
- Regular files (logs, configs, data)
- Network sockets (TCP connections, HTTP requests)
- Pipes and FIFOs
- Device files (
/dev/null, TTYs) - Inotify watches
- epoll instances
So when your web server handles 10,000 concurrent connections, it's opening 10,000 "files." When a log rotation script opens old and new log files simultaneously, that's two more. The numbers add up fast.
The kernel assigns each open resource a file descriptor — a small non-negative integer. The first three (0, 1, 2) are reserved for stdin, stdout, and stderr. Everything else starts at 3 and counts up.
By default, most Linux distributions set the per-process soft limit to 1024. For a desktop app, that's plenty. For a server under load? It's a trap waiting to spring.
The Three Layers of Limits
Think of file descriptor limits as a system of nested buckets. All three must be high enough for your workload.
Layer 1: Per-Process Soft Limit
This is what ulimit -n shows you. It's the active ceiling for a single process.
$ ulimit -n 1024 A non-root user can raise this limit on their own, but only up to the hard limit. Once raised, it applies to all child processes spawned from that shell.
The catch: changing this limit in a login shell does nothing for a daemon that started before your change. The limit is copied into the process at fork time. You must restart the service.
Layer 2: Per-Process Hard Limit
This is the ceiling that only root can raise. It's set in /etc/security/limits.conf and applies to new sessions.
$ ulimit -Hn 1024 If soft and hard are equal, no user can raise the limit further without root. The common production setup sets both to 65535.
Layer 3: System-Wide Limit
Even if every process can open 65535 files, the kernel has an overall cap on total file descriptors across all processes. Check it:
$ cat /proc/sys/fs/file-max 823008 This is rarely the bottleneck unless you're running thousands of high-concurrency services on a single box. For most setups, 2 million (2097152) is a safe target.
$ cat /proc/sys/fs/file-nr 18432 0 823008 The three numbers: currently allocated FDs, unused, and the max. If the first number is approaching the third, you need to raise file-max.
Diagnosing EMFILE in Production
When the alert fires, you need answers fast. Here's the diagnostic workflow:
Step 1: Confirm Which Process Is Affected
# Find the PID of your service pgrep -f "node|nginx|python|java" | head -5 # Check its current limits cat /proc/<PID>/limits | grep -A1 "open files" You'll see output like:
Max open files 1024 1024 files Both soft and hard are 1024 — that's your smoking gun.
Step 2: Count Current Open File Descriptors
ls /proc/<PID>/fd | wc -l If this is near 1024, the process is genuinely at its limit. If it's well below (say, 300), something else is going on — possibly a misconfigured service that's not actually hitting the limit, or a different error masquerading as EMFILE.
Step 3: Identify What's Consuming FDs
# See what types of files are open ls -l /proc/<PID>/fd | head -30 # Count by type ls -l /proc/<PID>/fd | awk '{print $9}' | sed 's|/dev/tcp/||' | sort | uniq -c | sort -rn | head -20 Common patterns:
- Hundreds of
/dev/tcp/*entries → network connections (normal for a busy server) - Many log files open → logging misconfiguration
- Thousands of
/tmp/*→ temp file leak -
eventfd,timerfd→ modern I/O event loops (epoll/kqueue style)
Step 4: Check for FD Leaks
If the FD count climbs steadily over 30 minutes without corresponding traffic growth, you likely have a leak:
# Snap twice, 30 seconds apart sleep 30 && ls /proc/<PID>/fd | wc -l A steady climb = leak. You'll need to identify the code path that opens but doesn't close. Common culprits: unclosed database connections, file handles in loops without finally/defer, socket connections in retry logic.
The Fix: Raising Limits Correctly
Quick Fix: Current Session
# Raise soft limit to 65535 (must be <= hard limit) ulimit -n 65535 # Verify ulimit -n This works immediately for the current shell and its children. But it resets on logout. Don't use this as your permanent solution — it's a bridge while you configure the proper fix.
Permanent Fix: /etc/security/limits.conf
Edit as root:
sudo nano /etc/security/limits.conf Add at the end:
* soft nofile 65535 * hard nofile 65535 root soft nofile 65535 root hard nofile 65535 Or, to set per-user limits:
deploy soft nofile 65535 deploy hard nofile 65535 Critical: Users must log out and back in for this to take effect. Service managers reading this file (like systemd with PAM) will pick it up on next start.
The Systemd Trap (Where Most People Get Stuck)
Here's the gotcha that cost me two hours once: systemd ignores /etc/security/limits.conf by default. If your service runs as a systemd unit, the limits you set in limits.conf won't apply.
Instead, create an override:
sudo systemctl edit nginx This opens an editor with an empty drop-in file. Add:
[Service] LimitNOFILE=65535 Save and exit, then:
sudo systemctl daemon-reload sudo systemctl restart nginx Verify the running process picked it up:
PID=$(pgrep -f nginx | head -1) cat /proc/$PID/limits | grep "open files" Expected output:
Max open files 65535 65535 files You can also preview what systemd will apply without restarting:
systemctl show -p LimitNOFILE nginx System-Wide Kernel Limit
If you're running many high-FD processes (a VPS with multiple Node.js services, for example), raise the kernel ceiling:
# Check current value cat /proc/sys/fs/file-max # Temporarily raise (resets on reboot) sudo sysctl -w fs.file-max=2097152 # Permanently add to sysctl config echo "fs.file-max = 2097152" | sudo tee -a /etc/sysctl.conf sudo sysctl -p 2 million FDs system-wide handles ~30 processes at 65535 each with room to spare.
The Verification Mindset
Here's the golden rule: never trust a config file. Trust /proc.
Config files are promises. /proc is reality. After every change:
# 1. Find the running process PID=$(pgrep -f "your-service") # 2. Check actual limits cat /proc/$PID/limits | grep "open files" # 3. Check actual usage ls /proc/$PID/fd | wc -l # 4. Cross-reference echo "Usage: $(ls /proc/$PID/fd | wc -l) / Limit: $(cat /proc/$PID/limits | grep 'open files' | awk '{print $NF}')" If usage is climbing toward the limit even after your fix, you haven't solved the problem — you've only delayed the next crash. Go back to Step 4 and hunt for the leak.
Special Cases
Alpine Linux / musl
Alpine doesn't use PAM. Editing /etc/security/limits.conf has zero effect. Instead, set limits in your entrypoint script:
# Dockerfile RUN apk add --no-cache util-linux # ... ENTRYPOINT ["sh", "-c", "ulimit -n 65535 && exec your-app"] Or use a sidecar/init container that calls prlimit.
macOS / launchd
macOS uses launchd, not systemd. GUI apps and launchd services ignore ulimit. Set limits via a plist:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>limit.maxfiles</string> <key>ProgramArguments</key> <array> <string>launchctl</string> <string>limit</string> <string>maxfiles</string> <string>65536</string> <string>200000</string> </array> <key>RunAtLoad</key> <true/> </dict> </plist> Save to /Library/LaunchDaemons/limit.maxfiles.plist and load it. This trips up almost everyone who develops locally on Mac and deploys to Linux.
Containers and cgroups
Docker containers inherit limits from the host by default. To set custom limits:
docker run --ulimit nofile=65535:65535 your-image Or in Kubernetes, set it in the pod spec:
securityContext: sysctls: - name: fs.file-max value: "2097152" Note: cgroups v2 may introduce a pids.max cap. If "too many open files" coincides with fork failures, check the PID cgroup, not just nofile.
Prevention: Monitoring That Actually Works
Raising limits is reactive. Prevention is better. Set up monitoring that catches FD exhaustion before it becomes an outage:
# One-liner for your monitoring agent watch -n 60 'echo "PID: $(pgrep -f your-service) | Used: $(ls /proc/$(pgrep -f your-service)/fd | wc -l) / Limit: $(cat /proc/$(pgrep -f your-service)/limits | grep open | awk "{print \$(NF-1)}")"' Or integrate with Prometheus:
# node_exporter collects fd_usage automatically # Alert when usage > 80% of limit groups: - name: filedescriptors rules: - alert: HighFileDescriptorUsage expr: fd_used / fd_limit > 0.8 for: 5m labels: severity: warning Set alerts at 70% (warning) and 85% (critical). A slow leak gives you days of advance notice; a sudden spike gives you minutes.
The Bottom Line
EMFILE is one of those errors that feels arcane until you understand the three layers. Once you do, it's straightforward:
- Diagnose: Check
/proc/<PID>/limitsand count/proc/<PID>/fd - Fix: Raise limits at the right layer (systemd override, not just
limits.conf) - Verify: Trust
/proc, not config files - Prevent: Monitor the ratio of used-to-limit before it hits 100%
The next time your phone buzzes at 2 AM with an EMFILE alert, you'll know exactly where to look — and more importantly, you'll know how to explain to your team that this wasn't a mystery, just a configuration that hadn't caught up with the workload.
More ops notes and server reliability guides: jiunotes · ops notes
