AI & ML
The GPU Needs an Operating System
arXiv cs.DC0 views
arXiv:2611.08720v1 Announce Type: new Abstract: AI chips sit mostly idle while serving models, because replies must be fast and each step waits on memory. On two large open models, the math units were busy only about 6-11% of the time. We argue the GPU needs an operating system that keeps urgent work on schedule and fills idle capacity with other useful work. A feature that ships today recovers a quarter of a GPU with work designed to share, and a tenth with real jobs, at 5% slower decode. Our research system does better: one resident kernel runs a whole model faster than vLLM's best for a single request, and shares the GPU at under 2% slowdown. Scaling beyond one hand-built model needs compilers and a few new chip features, which we specify.
Source：https://arxiv.org/abs/2611.08720PreviousResServe: Accurate and Efficient FP4 LLM Inference with Residue CompensationNextReKV: Overlapping Quantized KV Transfer and Suffix Recomputation
Related posts
A Straight Line Is Not a Camera Pose: Auditing Short-Clip Pose Benchmarks
0arXiv cs.RO
Mechanisms of Misgeneralization in Physical Sequence Modeling
0arXiv cs.RO
Behavior Graphs: Unsupervised Graph Structures for Interpreting Robot Deployment Data
0arXiv cs.RO
RoboFQE: Rollout-Free Evaluation of Visual Robot Policies from Offline Trajectories
0arXiv cs.RO
Improving Imitation Learning Efficiency for Manipulation through Geometric Prior Pretraining
0arXiv cs.RO
MATE: Asynchronous Multi-Rate Diffusion Policy Composition
0arXiv cs.RO