Backend
Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication
arXiv cs.DC0 views
arXiv:2611.18075v1 Announce Type: new Abstract: GPU communication libraries such as NCCL use Streaming Multiprocessors (SMs) for data movement and synchronization, slowing overlapped computation at a cost that grows with interconnect bandwidth. We present Zero-SM NCCL, which communicates within and across scale-up domains without launching any SM kernel. It synchronizes through CUDA stream memory operations, moves data asynchronously with copy engines within a scale-up domain, and drives network RDMA from a stream-ordered CPU proxy. It supports data-movement collectives such as AllGather and AlltoAll, plus one-sided ncclPutSignal and ncclWaitSignal APIs. On GB200 NVL72 and B200 NVL8 systems, Zero-SM collectives reach up to 1.36x higher AllGather bandwidth than NCCL's legacy SM-based kernels and cut the median slowdown of overlapped GEMMs from 33% to 4%, at a higher latency per operation. In Megatron-LM training, they accelerate FSDP by up to 3.5%, and one-sided puts accelerate pipeline parallelism by up to 15%. Zero-SM communication is not a universal win, however. Through an end-to-end study across models and parallelism strategies, we show that it pays off for large transfers in GPU-bound workloads and for deep pipelines with large bubbles, but loses on small messages and host-bound workloads. We distill these findings into guidelines for enabling zero-SM communication in NCCL and building it into other libraries.
Source:https://arxiv.org/abs/2611.18075PreviousThe Exact Stabilization Time of Self-Stabilizing Coloring: Mean, Variance, and DistributionNextMulti-GPU Tridiagonal Partition Algorithm
Related posts
How to redesign a website without losing Google rankings
0dev.to UX
What a Good Sandboarding Booking Website Needs in Dubai
0dev.to UX
Why GitHub Actions warns about Node 20 when your project uses Node 24
0dev.to GitHub
165 "completed" tasks. 21 left a clickable link. 11 of those links are already dead.
0dev.to GitHub
How to Push to GitHub Without Git: The API Workflow Behind My Solo Shipping Habit
0dev.to GitHub
How to get YouTube channel transcripts in bulk
0dev.to Data