Backend
CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale
arXiv cs.DC0 views
arXiv:2611.22476v1 Announce Type: new Abstract: Event streams are rarely uniform: a few dominant keys can generate orders of magnitude more events than the rest, overloading the workers that receive them and capping throughput. We present CheckCo, a checkpoint-coordinated load-balancing algorithm built on a dynamic oracle that estimates per-key frequencies from the live stream, with no prior statistics. Workers count events per key locally; at each checkpoint barrier, the counts are merged into a partitioning function that divides each key into a number of splits roughly proportional to its estimated frequency and packs the splits onto workers, and this function routes the next checkpoint window. We derive approximation bounds on the resulting load imbalance from classical scheduling results and bound its total key fan-out. On a production pipeline processing over 100 billion events per day, the most loaded worker receives only 9.8% more events than the average over a day (imbalance ratio 1.10), against 8.03 for key-based routing in simulation; in simulation, CheckCo also needs less than half the fan-out of earlier partitioners that reach comparable balance. Shifts in the key distribution are reflected in routing from the next checkpoint on, and the load stays balanced through job restarts.
Source:https://arxiv.org/abs/2611.22476PreviousXM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory PoolsNextByzantine Resilience of Robot Gathering under Round-Robin Scheduling
Related posts
glasp v0.5.3 Released: config init / Safer File Sync / Signature Verification on Install
0dev.to GitHub
Public Claude Code Settings Files With Secrets, One Week After Apiiro's Report
0dev.to GitHub

Two SARIMA models, one clear winner: picking with AIC value
0dev.to Challenge

Kling Image 4.0 Review: 8K Photos, Long Text and Image Sets, Tested
0dev.to Design
Running Out of Room? 7 Smart Home Improvements That Create More Living Space Without Moving
0dev.to Design
5 California App Development Companies and the Technical Questions to As
0dev.to Discuss