Open Source
Triple Overlap in Pipelined Krylov Methods on Multi-GPU Systems
arXiv cs.DC0 views
arXiv:2610.25058v2 Announce Type: replace Abstract: We introduce a novel triple-overlap implementation of the pipelined preconditioned conjugate gradient (Pipe-PCG) and pipelined stabilized biconjugate gradient (Pipe-BiCGStab) solvers, methods widely used in scientific applications. By leveraging a collective communication library, our approach extends conventional computation-communication overlap to concurrently execute collective operations, irregular P2P communication, and compute kernels, enabling computation-communication-communication overlap across multiple GPUs. We evaluate the design on the MareNostrum 5 (H100) and Leonardo (A100) supercomputers, comparing against PETSc and the state-of-the-art two-stream NCCL implementation using SuiteSparse matrices and three massive instances scaled up to 512 GPUs. The three-stream design outperforms the two-stream baseline on every instance, with the margin widening from 2-6% at 4 GPUs to 18-47% at 64 GPUs. At 64 GPUs, the geometric-mean speedup over the two-stream baseline is 1.33x on MareNostrum 5 and 1.23x on Leonardo for Pipe-PCG, and reaches 1.30x for Pipe-BiCGStab. In large-scale experiments, we observed a speedup up to 35% in Pipe-PCG and up to 33% in Pipe-BiCGStab compared to the two-stream design. For PETSc, the margin is even wider, up to 3.27x speedup in both Pipe-PCG and Pipe-BiCGStab. Related code for the proposed method is available online: https://github.com/ParCoreLab/aCG
Source:https://arxiv.org/abs/2610.25058PreviousTripleKV: Predict, Prefetch, Protect -- KV Cache Management for AI AgentsNextProperties of Concurrent Algorithms that Allow Asynchrony
Related posts
Are You Buying a Company or a model_id?
0dev.to Discuss
How to Test a Resistor, Diode, LED and Capacitor with a Multimeter (Before You Blame Your Code)
0dev.to Beginners
Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations
0arXiv eess.SP
Spectral Vision Transformer for Efficient Tokenization with Limited Data
0arXiv eess.SP
Gaussian Quasi-Likelihood Estimation for Nonsynchronously Observed L\'evy processes
0arXiv stat.TH
Rhythm-TTS: High-Fidelity Shortcut Latent Modeling and Probabilistic Rhythm Generation for Zero-Shot Text-to-Speech
0arXiv cs.SD