AI & ML
GPUSparse: Fast and Exact Learned Sparse Retrieval on a GPU-Resident Inverted Index
arXiv cs.DC0 views
arXiv:2606.26441v2 Announce Type: replace-cross Abstract: Learned sparse models such as SPLADE match the quality of dense bi-encoders and keep the exact-match behavior of lexical retrieval. They are served on CPUs, using inverted-index traversal algorithms such as WAND. These algorithms are fast because they prune: they skip documents that cannot enter the top-k. Pruning makes the result approximate, and its control flow is sequential and datadependent, which suits a CPU and not a GPU. We argue that a GPU should not run a faster traversal. It should remove traversal altogether. A GPU has enough memory bandwidth to score every posting, so exactness need not be traded for speed. We present GPUSparse, which keeps the inverted index in GPU memory and scores queries with one fused scatter-add kernel. Nothing is pruned. GPUSparse searches all 8.8M MS MARCO passages in 1.27 ms per query, at 787 queries per second on one H100. Its results are exact: Recall@10, @100 and @1000 against an exhaustive FP32 reference are all 1.000, and quality at full scale matches Pyserini's exact SPLADE retrieval to three decimals. Exactness costs hardware rather than latency. Under core pinning, one H100 is worth about seven CPU cores. GPUSparse is 5.6x faster than one core and 2.8x slower than a 32-core socket. A CPU socket is therefore cheaper for approximate serving. The GPU is the better choice when exactness is required, where GPUSparse is 3.0x faster than exact CPU block-max pruning. We also decompose GPU latency. Traversal takes 55 to 80% of it and selection 18 to 38%, and the split moves with scale, which reconciles three conflicting published reports.
Source:https://arxiv.org/abs/2606.26441PreviousTileMaxSim: Roofline Regimes and Compression Limits for Fused GPU MaxSim ScoringNextHow Much Reconstruction Does Quantum Machine Learning Need? Late Fusion of Uncoupled Quantum Subcircuits
Related posts
Startup Founder Interview: How Novera Is Turning Enterprise Infrastructure Into an Autonomous System
0HackerNoon
Startup Founder Interview: The Startup Using AI and Data to Improve Adoption and Foster Care
0HackerNoon

TurnWise Outperforms Streaming Video RL Baselines Across Three Benchmarks
0HackerNoon
TokenPulse: Building the Leadership Team Behind Cost-Efficient AI Inference
0HackerNoon
From Raw Sensor Signals to Regulatory-Grade Data: How Air Quality Monitoring Networks Are Engineered
0HackerNoon

Researchers Decrypt Hidden Chain-of-Thought From OpenAI, Anthropic and Google APIs
0HackerNoon