Backend
HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs
arXiv cs.DC0 views
arXiv:2601.22585v2 Announce Type: replace Abstract: The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.
Source:https://arxiv.org/abs/2601.22585PreviousTIDE: Reclaiming Lost Performance In On-Policy Reinforcement LearningNextSyclKittens: A Tile Programming Model for Programmers and Coding Agents on Intel GPUs
Related posts
![Robot Middleware and ROS2 — Ch.06: ROS2 Security [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3oowmzpb4qkk5km0di1k.png)
Robot Middleware and ROS2 — Ch.06: ROS2 Security [EN]
0dev.to Programming
![Robot Middleware and ROS2 — Ch.07: ROS2 Lifecycle Management [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx3a51weo0z1ob95yn3x2.png)
Robot Middleware and ROS2 — Ch.07: ROS2 Lifecycle Management [EN]
0dev.to Programming
![Robot Middleware and ROS2 — Ch.08: ROS2 Deployment and DevOps [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9z4fmoxmqqngiro7boaa.png)
Robot Middleware and ROS2 — Ch.08: ROS2 Deployment and DevOps [EN]
0dev.to Programming
![Robot Middleware and ROS2 — Ch.09: ROS2 for AMR [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmrvn70kciyfra1s6m77r.png)
Robot Middleware and ROS2 — Ch.09: ROS2 for AMR [EN]
0dev.to Programming
![Robot Middleware and ROS2 — Ch.10: ROS2 for Physical AI [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fboyfq92kdiuwiwz8xsoa.png)
Robot Middleware and ROS2 — Ch.10: ROS2 for Physical AI [EN]
0dev.to Programming
![Robot Middleware and ROS2 — Ch.12: ROS2 Case Studies [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw7pygqy790h9vw6mz16j.png)
Robot Middleware and ROS2 — Ch.12: ROS2 Case Studies [EN]
0dev.to Programming