Originally published on Code Beneath.
UDP packets vanish. Your application never sees them. The sender gets no error. Your monitoring shows traffic arrived on the interface, yet your receive loop pulls nothing from the socket. The culprit is often a UDP receive buffer overflow, where the OS kernel drops incoming datagrams because your application cannot consume them fast enough. Understanding how the kernel manages the UDP receive buffer, how to detect when it overflows, and how to tune SO_RCVBUF will save you hours of debugging and silent data loss. This article shows exactly what happens, with runnable code that demonstrates the problem and strace output that proves packets are being dropped at the OS level.
How UDP Receive Buffering Works
When a UDP packet arrives at a network interface, the kernel does not deliver it directly to your application. Instead, the packet lands in a socket receive buffer, a fixed-size ring buffer managed by the kernel. Your application calls recvfrom() (or recv()) to extract datagrams from this buffer. If the buffer fills faster than your code drains it, the kernel drops new arrivals silently. This is fundamentally different from TCP, where the sender is notified via congestion control; UDP is fire-and-forget, so the sender has no way to know a packet was discarded.
The size of this kernel buffer is controlled by the SO_RCVBUF socket option. On Linux, the default is typically 131,072 bytes (128 KB), but it can be much smaller on constrained systems or overridden per-socket. The actual allocated size is doubled by the kernel (for bookkeeping), so setting SO_RCVBUF to 1 MB means the kernel allocates 2 MB. If you receive many small packets or process them slowly, the buffer exhausts quickly.
The kernel tracks buffer usage via the sk_rmem_alloc counter (visible in kernel source). When a new datagram arrives and sk_rmem_alloc plus the new packet size would exceed the socket's receive buffer limit, the kernel increments a drop counter and discards the packet. No exception is thrown in your application; it simply never appears in recvfrom().
Detecting Packet Loss: The netstat Proof
Before you write code, you need to confirm packets are actually being dropped. Linux provides netstat and ss commands that expose UDP statistics at the system level. Run this on a machine receiving UDP traffic:
netstat -us Look for lines like:
Udp: Forwarding: 0 16000 packets received 0 packets to unknown port received 15800 packet receive errors The "packet receive errors" counter includes all drops: buffer exhaustion, checksum failures, and other kernel-level rejections. If this number grows while your application is receiving traffic, packets are being dropped. You can also use ss with per-socket granularity:
ss -uan This shows all UDP sockets, their receive buffer size, and (on newer kernels with RxQ/TxQ) the current queue depth. If you see RxQ climbing toward the buffer size, you have a bottleneck.
A Runnable Example: UDP Server with Intentional Buffer Starvation
Here is a complete, working UDP server in Go that deliberately leaves the receive buffer small and processes packets slowly, causing drops:
package main import ( "fmt" "net" "os" "syscall" "time" ) func main() { addr := net.UDPAddr{ Port: 9999, IP: net.ParseIP("127.0.0.1"), } conn, err := net.ListenUDP("udp", &addr) if err != nil { fmt.Fprintf(os.Stderr, "listen error: %v\n", err) os.Exit(1) } defer conn.Close() // Get the underlying socket file descriptor file, err := conn.File() if err != nil { fmt.Fprintf(os.Stderr, "file error: %v\n", err) os.Exit(1) } defer file.Close() fd := int(file.Fd()) // Set SO_RCVBUF to 16 KB, forcing buffer exhaustion with many small packets smallBuffer := 16 * 1024 err = syscall.SetsockoptInt(fd, syscall.SOL_SOCKET, syscall.SO_RCVBUF, smallBuffer) if err != nil { fmt.Fprintf(os.Stderr, "setsockopt error: %v\n", err) os.Exit(1) } // Verify the setting actualBuffer, err := syscall.GetsockoptInt(fd, syscall.SOL_SOCKET, syscall.SO_RCVBUF) if err == nil { fmt.Printf("SO_RCVBUF set to: %d bytes\n", actualBuffer) } fmt.Println("UDP server listening on 127.0.0.1:9999") fmt.Println("Processing packets with 10ms delay to cause buffer overflow...") buffer := make([]byte, 2048) for { n, remoteAddr, err := conn.ReadFromUDP(buffer) if err != nil { fmt.Fprintf(os.Stderr, "read error: %v\n", err) continue } fmt.Printf("Received %d bytes from %s: %s\n", n, remoteAddr, buffer[:n]) // Simulate slow processing: hold the receive loop for 10ms time.Sleep(10 * time.Millisecond) } } To generate load and trigger drops, run this Python client in another terminal:
import socket import time sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) server_addr = ("127.0.0.1", 9999) for i in range(100): message = f"Packet {i}".encode() sock.sendto(message, server_addr) # No delay; send as fast as possible if i % 10 == 0: print(f"Sent {i} packets") sock.close() print("All packets sent") Run the server, then run the client. The server will not print all 100 packets; many are dropped silently by the kernel because the 16 KB buffer fills while the application sleeps.
Using strace to See Kernel-Level Drops
To prove the kernel is dropping packets, use strace to watch the actual system calls and kernel statistics. Start the server with strace attached:
strace -e trace=recvfrom,setsockopt -f ./udp_server 2>&1 | head -40 Then run the client. You will see recvfrom() syscalls returning with packet sizes, but fewer calls than packets sent. The gap is your drop count. For more precise statistics, check /proc/net/udp while the server is running:
cat /proc/net/udp | grep 270F The columns are: sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode. The rx_queue column (index 4) shows how many bytes are buffered but not yet read. If this stays high (near your SO_RCVBUF limit), you have congestion. Unfortunately, /proc/net/udp does not directly expose drop counts; for that, you rely on netstat -us or monitoring /proc/net/snmp with the UdpInErrors counter.
A cleaner modern approach is to use the ss command with JSON output:
ss -uan --json | jq '.UDP[] | select(.local_address == "127.0.0.1")' On systems with BPF tracing (bcc tools), you can capture drops in real time:
sudo python3 /usr/share/bcc/tools/udplife.py This attaches eBPF probes to the kernel's UDP stack and prints every datagram, including drops, with full source/dest details.
Tuning SO_RCVBUF: Best Practices
Once you confirm drops, the first lever is SO_RCVBUF. How large should you set it? This depends on your packet rate, packet size, and how long your application can afford latency. A rough formula:
SO_RCVBUF = (packets_per_second) * (average_packet_size) * (max_latency_seconds) * 2 For example, if you receive 10,000 packets per second of 500 bytes each, and you can tolerate 100 ms of buffering before drops, you need:
10,000 * 500 * 0.1 * 2 = 1,000,000 bytes = 1 MB Set SO_RCVBUF early, before calling bind() or connect():
fd, _ := syscall.Socket(syscall.AF_INET, syscall.SOCK_DGRAM, 0) syscall.SetsockoptInt(fd, syscall.SOL_SOCKET, syscall.SO_RCVBUF, 10*1024*1024) // Then bind or connect On Linux, there is also a system-wide limit: /proc/sys/net/core/rmem_max. SO_RCVBUF cannot exceed this value. Check and raise it if needed:
cat /proc/sys/net/core/rmem_max # Typically 212992 (208 KB) by default # Raise it with: sudo sysctl -w net.core.rmem_max=134217728 # 128 MB Similarly, the default starting value is controlled by /proc/sys/net/core/rmem_default. For high-traffic UDP services, set this in /etc/sysctl.conf and persist it across reboots.
The ENOBUFS Edge Case
There is a subtler scenario where you might see ENOBUFS (No buffer space available) returned directly from recvfrom(). This is distinct from silent drops. ENOBUFS indicates a temporary kernel memory allocation failure, not buffer overflow on the socket. It can occur if:
- The system is under extreme memory pressure.
- The kernel's internal UDP buffer pool is exhausted (rare on modern systems).
- You are running on a system with strict memory limits (containers, VMs).
If your application receives ENOBUFS, it should not treat it as a fatal error. Typically, you sleep briefly and retry the read, giving the kernel time to free memory. However, ENOBUFS is not the same as UDP receive buffer overflow; it is a system-wide resource constraint, not a per-socket tuning issue.
Comparison: UDP vs. TCP Buffer Behavior
| Aspect | UDP | TCP |
|---|---|---|
| Receive buffer overflow behavior | Packets dropped silently by kernel; sender unaware | Sender blocked or retransmits due to congestion control |
| SO_RCVBUF tunability | Critical; directly impacts packet loss | Important but less critical due to flow control |
| System limits | rmem_max limits per-socket buffer | tcp_rmem limits per-socket buffer |
| Visibility of drops | Requires netstat -us or /proc/net/snmp inspection | Observable via TCP retransmit counters and tcpdump |
| Application-level recovery | Impossible; packet already lost at OS | Built-in retransmission at transport layer |
Real-World Monitoring in Production
In production, do not rely on manual netstat runs. Automate monitoring with a script that periodically checks /proc/net/snmp and alerts on rising drop counts:
#!/bin/bash # monitor_udp_drops.sh prev_drops=0 while true; do current_drops=$(awk '/^Udp:/{getline; print $4}' /proc/net/snmp) if (( current_drops > prev_drops )); then new_drops=$((current_drops - prev_drops)) echo "WARNING: $new_drops UDP packets dropped in last interval" | \ logger -t udp_monitor fi prev_drops=$current_drops sleep 10 done Alternatively, use Prometheus node_exporter, which exports UDP metrics. Scrape the node_network_receive_dropped_total metric and set an alert if it increases.
For application-level insight, you can use syscall tracing tools like bpftrace to correlate drops with application state. Track how many bytes are in the socket receive queue when drops occur:
sudo bpftrace -e 'tracepoint:skb:kfree_skb { printf("drop: %s\n", str(args->reason)); }' Interaction with Datagram Size and Fragmentation
The interaction between UDP receive buffer size and datagram size matters. If you set SO_RCVBUF to 64 KB but receive 1.5 KB packets, you can buffer around 42 packets. However, if you receive 8 KB packets, you fit only 8. More subtly, if packets are fragmented at the IP layer (which happens when they exceed MTU, typically 1500 bytes), the kernel reassembles them before placing them in the UDP receive buffer. This reassembly itself uses kernel memory. For large datagrams, especially those that fragment across multiple IP packets, you need a larger buffer to account for the per-packet overhead. See UDP datagram size and fragmentation for more on this.
Common Mistakes
1. Setting SO_RCVBUF after bind(): On some systems, the kernel may ignore the setting if the socket is already bound. Always set socket options immediately after socket creation.
2. Forgetting the system-wide limit: You set SO_RCVBUF to 256 MB, but /proc/sys/net/core/rmem_max is only 212 KB. The kernel caps your request silently. Always check and raise rmem_max first.
3. Assuming SO_RCVBUF alone solves the problem: If your application's recvfrom() loop is blocked on other I/O or CPU-bound work, a large buffer just delays the inevitable overflow. You must also optimize packet processing speed and consider multi-threading or async I/O.
4. Confusing SO_RCVBUF with SO_REUSEADDR: SO_RCVBUF is for the receive buffer. SO_REUSEADDR allows rapid socket rebind. They are orthogonal; you need both for resilient UDP services.
5. Ignoring netstat output changes over time: A single netstat run is a snapshot. If packet loss is intermittent (e.g., load spikes), you need continuous monitoring to catch it.
FAQ
Can I detect dropped packets from inside my application?
Not reliably. You can count the packets you receive and compare against an external counter, but the kernel does not expose the drop count to your process. You must query /proc/net/snmp or use netstat -us from outside the application. One workaround: have the sender include a sequence number in each datagram, and your receiver logs gaps. This reveals packet loss but does not tell you if it was due to buffer overflow or network loss.
How large should SO_RCVBUF really be?
There is no one-size-fits-all answer. Start with 10x the amount of data you expect to arrive in one round-trip time (RTT). For local networks with 1 ms RTT and 10 Gbps throughput, that is roughly 1.25 MB. For 10 ms RTT on the same link, 12.5 MB. Monitor drops with netstat and increase SO_RCVBUF until drops stop. Diminishing returns set in quickly; 1 GB buffers on a gigabit link are overkill.
Do I need to tune rmem_default, or only rmem_max?
rmem_default is the starting size for a new UDP socket if you do not set SO_RCVBUF. rmem_max is the absolute ceiling. For a service that receives high-volume UDP, set rmem_default high (e.g., 8 MB) so default sockets have breathing room, and set rmem_max even higher (e.g., 256 MB) to allow per-socket tuning. If you only raise rmem_max but leave rmem_default low, sockets created without explicit SO_RCVBUF tuning will still drop packets.
What about SO_RCVBUFFORCE on Linux?
SO_RCVBUFFORCE bypasses the rmem_max limit but requires CAP_NET_ADMIN capability (usually root). Use it with extreme caution; setting an excessively large buffer can starve other processes or destabilize the system. Modern services running in containers or with least-privilege assumptions should avoid it. Instead, ask the operator to raise rmem_max via sysctl if needed.
Why does the kernel double SO_RCVBUF internally?
The kernel uses the extra space for sk_buff headers, socket structure overhead, and book-keeping. The doubled size is what is actually allocated from kernel memory. When you call getsockopt(SO_RCVBUF), you get the value you set, not the true allocation. This is by design to keep the API simple, but it means a 1 MB SO_RCVBUF consumes 2 MB of kernel RAM.
Can packet loss happen even if my SO_RCVBUF is large?
Yes. If your application is blocked on recvfrom() but the kernel cannot deliver packets faster than they arrive, or if there are network-level drops (bad checksums, NIC buffer overflow, network congestion), you will lose packets regardless of SO_RCVBUF size. SO_RCVBUF only protects against application-level processing slowness, not network problems. Use tcpdump on the interface to verify packets are actually reaching your machine.