Backend
Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs
arXiv cs.DC0 views
arXiv:2608.01651v2 Announce Type: replace Abstract: Emerging LLMs increasingly interleave full attention with recurrent linear attention to reduce long-context serving costs. Tree speculative decoding could further accelerate these hybrid models, but recurrent linear-attention layers carry parent-dependent states. Existing engines therefore verify proposal trees sequentially and snapshot the full state per node, causing latency and memory to grow with tree and batch sizes. We present Twig, a kernel--runtime co-design for tree speculative decoding of hybrid LLMs. Twig derives a tree-structured factorization of the linear-attention recurrence and realizes it with a resource-efficient GPU kernel that verifies all nodes in parallel, accelerating linear-attention tree verification by 3.4--7.7$\times$. It losslessly represents speculative state updates as token-level factors and reconstructs only the accepted state, reducing transient state memory by 82--99$\times$ and freeing GPU memory for KV caches. We integrate Twig into SGLang, a widely deployed production LLM serving engine, and further optimize its runtime through batch-wide verification budgeting. Across four models, two GPU platforms, and diverse datasets, Twig achieves up to $4.72\times$ the offline decode throughput of autoregressive decoding and up to $2.03\times$ that of the strongest tree-speculative baseline. Under online agent workloads, Twig reduces TTFT and TPOT by up to $67.6\%$ and $49.9\%$, respectively, over the strongest tree-speculative baseline.
Source:https://arxiv.org/abs/2608.01651PreviousWave-Based Dispatch for Circuit Cutting in Hybrid HPC-Quantum SystemsNextLantern: Finding Committable Transactions via Back-Propagation on DAGs
Related posts

How Hindsight memory badges stopped us repeating bad fixes
0dev.to
AMD opens the managed agent runtime behind its own Instinct GPU clusters
0dev.to
My friends kept pitching games nobody built. So I gave our group chat three AI teammates.
0dev.to
Phone photos are weak evidence: EXIF is editable in 30 seconds
0dev.to
How to Blur Faces in Photos Online (Free, No Signup)
0dev.to
Using the browser's speech recognizer as a pronunciation judge
0dev.to