AI & ML
JigSLO: Solving the Resource Puzzle for Efficient Multi-LLM Serving
arXiv cs.DC0 views
arXiv:2509.06261v3 Announce Type: replace Abstract: GPU spatial sharing co-locates instances with complementary demands, such as compute-bound prefill and memory-bound decode, promising cost-efficient multi-LLM serving. Prior systems partition compute and coordinate shared memory capacity, but do not model bandwidth contention among co-located instances of heterogeneous LLMs. This contention makes each instance's latency depend on the memory-access patterns of its co-runners, even when its own resource allocation remains unchanged. Drawing on extensive measurements, we find that the latency impact of bandwidth contention can be captured by a time-shared memory-access model with contended and uncontended periods, so the overall latency can be analytically estimated as a time-weighted average of the latency estimates for these periods. We propose JigSLO, a contention-aware multi-LLM serving system that manages memory resources and schedules each model's prefill and decode phases as separate instances, treating their co-location as a placement decision. Its contention-aware predictor estimates per-instance latency under any co-location from standalone profiles alone. Guided by these predictions, its placement algorithm jointly selects each instance's tensor parallelism, compute share, and batch configuration, pairing complementary memory-access patterns to maximize throughput within each model's latency targets. JigSLO's shared resource manager complements this on the capacity side, sharing weights, pooling KV cache, and transferring it between prefill and decode instances. Across three datasets covering chat, code generation, and long-context summarization, JigSLO outperforms state-of-the-art baselines by $2.3\times$ in SLO attainment and $1.2\times$ in throughput on average, matching their best attainment with half the GPUs on long-context workloads.
Source:https://arxiv.org/abs/2509.06261PreviousSpecSearch: Towards Hardware-Aware Efficient Speculative DecodingNextAngelfish: Leader, DAG, or Anywhere in Between
Related posts
AI-Powered Sashiko Introduces Experimental Patch Series Design Assessment
0Phoronix

Preparing for a post-quantum future
0TechRadar

75+ best Black Friday deals 2026 in the UK — GTA 6, Kindles, Switch 2, Shark vacuums, LG OLED TVs, and more
0TechRadar

Why no 40fps mode for GTA 6? — Rockstar says the ‘complexity of simulating and rendering our world’ requires it to ‘carefully balance how we spend every millisecond of both the central processor and t
0TechRadar

'Enterprises need to connect AI to the right data': Report finds over half of firms have abandoned AI projects over unclear goals
0TechRadar

$3.26 million in SpaceX AI data center revenue will be pumped back into the City of Memphis
0Tom's Hardware