AI & ML
ISER-MoE: Dynamic In-Switch Replication of Hot Experts for Load Balancing in Expert-Parallel
arXiv cs.DC0 views
arXiv:2611.09008v1 Announce Type: new Abstract: Expert-parallel mixture-of-experts (MoE) layers run at the pace of the most loaded GPU, and a few hot experts carry most of the excess load. Replicating them at every MoE layer balances the load only if their weights arrive within the layer, and broadcasting from the owner is the bottleneck: point-to-point copies overload it, and baseline broadcasts pay latency or bandwidth overheads. We present ISER-MoE (In-Switch Expert Replication for MoE), built in TensorRT-LLM, which multicasts hot experts through the NVLink switch to fixed GPU groups, so each byte leaves its owner once. A GPU scheduler assigns hot experts to groups per layer, and a multi-port broadcast spreads the injection over the group's ports, both overlapped with the shared expert. In trace replay, the scheduler removed 98-99% of the excess load; in prefill serving of DeepSeek-V4-Pro, throughput rose 17.4% on a math benchmark and 7.2% on agentic prompts.
Source:https://arxiv.org/abs/2611.09008PreviousCapital Efficiency, Isolation, Risk, and Ruin in Generative AI Inference Serving InfrastructureNextDARIC: A Position-Independent KV Cache Reuse System with Distribution-Aware Recomputation
Related posts
Release Radar, 03–09.11: 27 flagged releases with verbatim quotes
0dev.to Python
chmod 755 Is Not 'Read, Write, Execute': 5 Permission Traps
0dev.to DevOps
Building a Production-Oriented MLOps Prototype with Python and FastAPI
0dev.to Docker

A profile of Alice Huang, a suspected Chinese spy who founded Megaspeed in 2023 and spent years in Jensen Huang's orbit; Megaspeed's Nvidia imports top $4.6B (Mark Anderson/Bloomberg)
0Techmeme

Meta says it is globally expanding Instagram's AI age estimation tech to apply Teen Account protections to teens aged 13 to 17 who misrepresented their age (Jess Weatherbed/The Verge)
0Techmeme

Alphabet's Wing plans to invest up to $125M over two years to build a 15,000-square-foot facility in Fort Worth, Texas, to produce tens of thousands of drones (Mark R. Long/Wall Street Journal)
0Techmeme