eBPF Ring Buffer Backpressure in Go Services: Kernel-to-User Data Loss, Perf Event Semantics, and the Sampling Tradeoff
eBPF-based observability tooling—network tracing, syscall auditing, latency histograms—sits inside production Go services today not as a curiosity but as a load-bearing instrumentation layer. The appeal is real: zero application recompilation, nanosecond-accurate timestamps, kernel-side aggregation. The failure mode is subtle and almost never surfaced in vendor documentation: the ring buffer fills, the kernel drops events silently, and your dashboards look completely healthy while you are blind to entire request classes.
This article walks through the mechanics of that failure, compares the two primary kernel-to-user transport mechanisms, and proposes a concrete adaptive sampling strategy implementable in a Go consumer.
Two Transports, Two Failure Modes
Perf Event Array (BPF_MAP_TYPE_PERF_EVENT_ARRAY)
The older mechanism. Each CPU core writes to its own per-CPU ring buffer via bpf_perf_event_output. The Go side—typically cilium/ebpf or libbpf-go—opens an epoll file descriptor per CPU and drains records in a polling loop.
The critical property: when the per-CPU buffer is full, the kernel drops the event and increments a lost-samples counter accessible only by reading the perf event header's PERF_RECORD_LOST records inline with data records. If your consumer loop is not reading that header type explicitly, you never see the loss count. The buffer is sized at map creation time in pages (must be a power of two). Default tooling often uses 8–64 pages (32–256 KB per CPU). On a 32-core host tracing every syscall, that fills in under a millisecond under load.
Ring Buffer (BPF_MAP_TYPE_RINGBUF)
Introduced in kernel 5.8. A single shared ring buffer across all CPUs, avoiding the per-CPU fragmentation problem. The eBPF program reserves space atomically with bpf_ringbuf_reserve, writes, then commits. If reservation fails (buffer full), your eBPF program receives a NULL pointer—and must explicitly handle that path or silently skip the event.
The Go consumer uses a single file descriptor with epoll. Reads are lock-free via a pair of producer/consumer position counters in a shared memory region. Throughput is higher and memory utilization is better, but the single buffer means a slow Go consumer blocks all CPU cores from committing new records, not just one.
The failure modes differ:
- Perf array: per-CPU isolation means one hot core hurts only its own queue, but loss counters are buried in stream.
- Ring buffer: global contention, but the fd-based notification model makes watermark-driven wakeup straightforward.
Where Go Consumer Latency Causes Drops
A Go eBPF consumer is structurally a tight loop:
rd, err := ringbuf.NewReader(objs.Events) if err != nil { log.Fatal(err) } defer rd.Close() for { record, err := rd.Read() if errors.Is(err, ringbuf.ErrClosed) { return } if err != nil { continue } processEvent(record.RawSample) } processEvent is where latency accumulates. If it does anything synchronous—a MongoDB write, a JSON marshal with allocation, a channel send to a blocked goroutine—the Read call is delayed. During that delay, the kernel continues writing. When the buffer reaches capacity, bpf_ringbuf_reserve returns NULL in your eBPF C program and the event is dropped. The kernel does expose a lost counter in the ring buffer's metadata page, but cilium/ebpf's Reader does not surface it directly; you must memory-map the consumer page yourself and read offset 24 bytes in (the lost u64 field).
The runtime scheduler compounds this. A goroutine blocked on a channel send while another goroutine holds the P can delay the consumer loop by an entire scheduler quantum (~10ms in the worst case with a saturated GOMAXPROCS). Unlike a C consumer that owns its thread, a Go consumer is subject to preemption. Pinning the consumer goroutine to an OS thread with runtime.LockOSThread() reduces jitter but does not eliminate GC stop-the-world pauses.
Adaptive Sampling: The Correct Mitigation
Buffer resizing is the blunt instrument. Doubling the ring buffer from 64 pages to 512 pages absorbs spikes but adds 1.75 MB of locked kernel memory per buffer instance and does not help if the consumer is structurally slow. The better approach is kernel-side adaptive sampling driven by a feedback signal from the user-space consumer.
The architecture:
- A BPF map of type
BPF_MAP_TYPE_ARRAYholds a single u32sample_rate. The eBPF program reads it on each event and emits only whenbpf_get_prandom_u32() % sample_rate == 0. - The Go consumer tracks the fill ratio of the ring buffer by reading the producer and consumer position counters from the mmap'd metadata page.
- A control goroutine adjusts
sample_rateup (more dropping) when fill ratio crosses a high-water mark, and down (more recording) when it drops below a low-water mark.
type SamplingController struct { sampleRateMap *ebpf.Map metaPage []byte // mmap of ring buffer consumer page mu sync.Mutex current uint32 } func (sc *SamplingController) Adjust() { prod := atomic.LoadUint64((*uint64)(unsafe.Pointer(&sc.metaPage[0]))) cons := atomic.LoadUint64((*uint64)(unsafe.Pointer(&sc.metaPage[8]))) // ring size is known at construction; derive fill ratio inflight := prod - cons ratio := float64(inflight) / float64(sc.ringSize) sc.mu.Lock() defer sc.mu.Unlock() switch { case ratio > 0.75 && sc.current < 64: sc.current = min(sc.current*2, 64) case ratio < 0.25 && sc.current > 1: sc.current = max(sc.current/2, 1) } val := sc.current _ = sc.sampleRateMap.Put(uint32(0), val) } Run Adjust on a 500ms ticker. The convergence is coarse but sufficient: a sudden 10× traffic spike raises sample_rate to 8 within 1.5 seconds (three ticks), dropping 87.5% of events, which is far preferable to dropping 100% while the buffer saturates and losing the lost-count signal itself.
Operational Consequences
Sampling breaks rate-based alerting. If you emit events for every inbound request and drive a Prometheus counter from the Go consumer, sampling invalidates the counter without correction. The fix: maintain a total_events_seen counter in the eBPF program (a per-CPU array, summed in user space) and expose it alongside total_events_emitted. Compute effective sample rate as their ratio and apply it as a multiplier in your recording rules.
The lost counter is not a replacement for structured drop attribution. The ring buffer's metadata-page lost count tells you how many reservations failed but not which event types were dropped. Under mixed workload tracing—syscalls, network events, uprobe hits—you cannot diagnose which probe is the hot path from the aggregate. Instrument per-probe drop counters in a second BPF map keyed by probe ID.
Memory-mapping the metadata page is not documented in cilium/ebpf as a stable API. The page layout matches kernel bpf_ringbuf internals. Pin a kernel version test in CI that validates the offsets have not shifted. Alternatively, use BPF_OBJ_GET_INFO_BY_FD to retrieve ring_buffer_max_entries and correlate with the fd's seek position—noisier but kernel-API-stable.
Decision Framework
Choose your transport and mitigation path based on these axes:
| Condition | Recommendation |
|---|---|
| Kernel < 5.8 | Perf event array; read LOST records explicitly |
| Single hot CPU core (e.g., network RX softirq affinity) | Perf array preferred; ring buffer will serialize all cores |
| Mixed multi-probe workload, kernel ≥ 5.8 | Ring buffer; lower total memory, single wakeup fd |
| Go consumer does synchronous I/O in hot path | Decouple: channel → worker pool; never block Read loop |
| Consumer latency p99 > 5ms | Add adaptive sampling controller; do not just enlarge buffer |
| Need event-type-level drop attribution | Per-probe drop counter map; aggregate in user space |
| Rate-based alerting downstream | Emit total-seen counter from eBPF; apply ratio correction in Prometheus rules |
The ring buffer is the right default for Go services on modern kernels—its throughput and memory profile are better, and the notification model suits Go's epoll integration. But treating it as a lossless channel is the architectural mistake. Build the sampling controller, expose the drop signal, and correct derived metrics at the recording layer. Observability infrastructure that can silently lose data under its own load is not infrastructure—it is a liability dressed as insight.

