AI & ML
ISER-MoE: Dynamic In-Switch Replication of Hot Experts for Load Balancing in Expert-Parallel
arXiv cs.DC0 views
arXiv:2611.09008v1 Announce Type: new Abstract: Expert-parallel mixture-of-experts (MoE) layers run at the pace of the most loaded GPU, and a few hot experts carry most of the excess load. Replicating them at every MoE layer balances the load only if their weights arrive within the layer, and broadcasting from the owner is the bottleneck: point-to-point copies overload it, and baseline broadcasts pay latency or bandwidth overheads. We present ISER-MoE (In-Switch Expert Replication for MoE), built in TensorRT-LLM, which multicasts hot experts through the NVLink switch to fixed GPU groups, so each byte leaves its owner once. A GPU scheduler assigns hot experts to groups per layer, and a multi-port broadcast spreads the injection over the group's ports, both overlapped with the shared expert. In trace replay, the scheduler removed 98-99% of the excess load; in prefill serving of DeepSeek-V4-Pro, throughput rose 17.4% on a math benchmark and 7.2% on agentic prompts.
Source:https://arxiv.org/abs/2611.09008PreviousCapital Efficiency, Isolation, Risk, and Ruin in Generative AI Inference Serving InfrastructureNextDARIC: A Position-Independent KV Cache Reuse System with Distribution-Aware Recomputation
Related posts
Our Route Planner Sent Crane Lorries to the Middle of Each Postcode and Some Rural Postcodes Are Miles Wide
0dev.to Data
How much of GPT-6 Astra's weight-editing demo is the harness?
0dev.to Machine Learning
How Pet Groomers Can Track Appointments, Pet Notes and Rebookings
0dev.to Tutorial
Symmetrical Components in Plain English: Fortescue Without the Fog
0dev.to Tutorial
When P2P Pays You and the Account Freezes Anyway
0dev.to Career
Your Company Will Pay for the Course if You Agree to Pay It Back When You Leave
0dev.to Career