AI & ML
ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
arXiv cs.DC0 views
arXiv:2607.00466v3 Announce Type: replace Abstract: Equal load does not mean equal decode latency in prefill-decode (PD) disaggregated mixture-of-experts (MoE) serving. In bandwidth-bound decode, more distinct experts mean more weight loads and higher latency. We present ELDR, a lossless, expert-locality-aware decode router. Prefill signatures predict decode expert usage. Balanced $K$-means fits per-decoder centroids with background refitting. Locality-band routing selects the least-loaded worker among close matches. A block-granular cache keeps signatures exact under prefix caching. We evaluate ELDR in vLLM on up to 40 GPUs. Across three MoE models on 12 public task benchmarks and multilingual WildChat, it reduces median time-per-output-token (TPOT) by 5.4-14.5% on average across request rates and up to 17.8% over the best of four load-balancing baselines, preserving model outputs and token-to-expert assignments.
Source：https://arxiv.org/abs/2607.00466PreviousOnline Graph Exploration under T-Interval ConnectivityNextGreedy-Like Defective Coloring: Distributed Algorithms and Applications
Related posts
DeepRitzSplit: A Variational Neural Operator Framework for Gradient Flows with Application to Dendritic Growth
0arXiv cs.NA
Quantitative Universal Approximation for Noisy Quantum Neural Networks
0arXiv cs.NA
Vertex interpolation by bivariate $C^1$ cubic splines: counterexamples, geometric stability, and indecomposable defects
0arXiv cs.NA
Adaptive refinement based on local approximation power of hierarchical spline spaces
0arXiv cs.NA
Discrete hypocoercive estimates for discontinuous Galerkin methods: application to the Vlasov-Poisson-Fokker-Planck system
0arXiv cs.NA
Novel technique based on L\'eja Points Approximation for Log-determinant Estimation of Large matrices
0arXiv cs.NA