AI & ML
The Cost of Synchronization in GPU Scale-Up Domains
arXiv cs.DC0 views
arXiv:2608.22503v2 Announce Type: replace Abstract: GPU scale-up domains are the building blocks of machine learning infrastructure. Architects provision the size and bandwidth in these domains assuming all GPUs reach a collective operation at the same time. However, we find that this assumption does not hold in practice. Across 300,000 collectives from training four language models on A100, H100, and H200 domains, GPUs reach the same collective hundreds to thousands of microseconds apart, despite running identical kernels. Since a collective cannot complete until every GPU contributes its data, GPUs that arrive early wait for the slowest one. We call this waiting time the synchronization tax, and it consumes nearly 40% of tensor-parallel communication time in an 8-GPU domain. To find its source, we develop an algorithm that recovers the chain of operations that delays each GPU before a collective. Our algorithm attributes 78% of the tax to variation in the runtimes of matrix-multiplication (GEMM) kernels across GPUs. Since the tax depends on the slowest of many GPUs, our analytical model shows that it grows at least logarithmically with domain size. Thus, a 72-GPU domain requires 3.3x more per-GPU bandwidth than today's simulators predict to keep communication below 12% of step time, and the optimal scale-up fabric should partition a switch's capacity among fewer GPUs with more bandwidth each.
Source:https://arxiv.org/abs/2608.22503PreviousProgrammable Probabilistic Computer with 1,000,000 p-bitsNextA Flat KV Cache Storage Architecture for LLM Serving
Related posts
CaMBRAIN: Real-time, Continuous EEG Inference with Causal State Space Models
0arXiv cs.HC
How Far Does Online RL Generalize in Mobile GUI Agents? A Controlled Study with AndroidWorld-Generalization
0arXiv cs.HC
From Consensus-Based Optimization to Evolution Strategies: Proof of Global Convergence
0arXiv cs.NA
Unconditionally $L^2$-stable explicit--implicit--null filtered Runge--Kutta methods for general nonlinear degenerate parabolic equations
0arXiv cs.NA
CS-PIKAN: Curvilinear Separable Physics-Informed Kolmogorov-Arnold Networks with Autograd-Based Metric Evaluation and Certified Error Bounds for PDEs on Complex Domains
0arXiv cs.NA
Fourier--Galerkin Methods for Subwavelength Resonances in two-dimensional Acoustic Metamaterials
0arXiv cs.NA