Backend
Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs
arXiv cs.DC0 views
arXiv:2608.01651v2 Announce Type: replace Abstract: Emerging LLMs increasingly interleave full attention with recurrent linear attention to reduce long-context serving costs. Tree speculative decoding could further accelerate these hybrid models, but recurrent linear-attention layers carry parent-dependent states. Existing engines therefore verify proposal trees sequentially and snapshot the full state per node, causing latency and memory to grow with tree and batch sizes. We present Twig, a kernel--runtime co-design for tree speculative decoding of hybrid LLMs. Twig derives a tree-structured factorization of the linear-attention recurrence and realizes it with a resource-efficient GPU kernel that verifies all nodes in parallel, accelerating linear-attention tree verification by 3.4--7.7$\times$. It losslessly represents speculative state updates as token-level factors and reconstructs only the accepted state, reducing transient state memory by 82--99$\times$ and freeing GPU memory for KV caches. We integrate Twig into SGLang, a widely deployed production LLM serving engine, and further optimize its runtime through batch-wide verification budgeting. Across four models, two GPU platforms, and diverse datasets, Twig achieves up to $4.72\times$ the offline decode throughput of autoregressive decoding and up to $2.03\times$ that of the strongest tree-speculative baseline. Under online agent workloads, Twig reduces TTFT and TPOT by up to $67.6\%$ and $49.9\%$, respectively, over the strongest tree-speculative baseline.
Source:https://arxiv.org/abs/2608.01651PreviousWave-Based Dispatch for Circuit Cutting in Hybrid HPC-Quantum SystemsNextLantern: Finding Committable Transactions via Back-Propagation on DAGs
Related posts

Building PAKKA: An AI-Powered Debt Accountability App That Remembers Promises
0dev.to
MongoDB X.509 Authentication Depends on Exact Subject Mapping
0dev.to
YouTube recommends 3840x2160. Top size guides still say 1280x720
0dev.to
We Built a CV Scanner That Compares Skills Against Real Jobs. Then Production Exposed a Bottleneck.
0dev.to

Amazon Leadership Principles Interview Questions: the stories behind the 16 principles
0dev.to
Block .env and config-file probes in Express and Fastify
0dev.to