Backend
Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs
arXiv cs.DC0 views
arXiv:2608.01651v2 Announce Type: replace Abstract: Emerging LLMs increasingly interleave full attention with recurrent linear attention to reduce long-context serving costs. Tree speculative decoding could further accelerate these hybrid models, but recurrent linear-attention layers carry parent-dependent states. Existing engines therefore verify proposal trees sequentially and snapshot the full state per node, causing latency and memory to grow with tree and batch sizes. We present Twig, a kernel--runtime co-design for tree speculative decoding of hybrid LLMs. Twig derives a tree-structured factorization of the linear-attention recurrence and realizes it with a resource-efficient GPU kernel that verifies all nodes in parallel, accelerating linear-attention tree verification by 3.4--7.7$\times$. It losslessly represents speculative state updates as token-level factors and reconstructs only the accepted state, reducing transient state memory by 82--99$\times$ and freeing GPU memory for KV caches. We integrate Twig into SGLang, a widely deployed production LLM serving engine, and further optimize its runtime through batch-wide verification budgeting. Across four models, two GPU platforms, and diverse datasets, Twig achieves up to $4.72\times$ the offline decode throughput of autoregressive decoding and up to $2.03\times$ that of the strongest tree-speculative baseline. Under online agent workloads, Twig reduces TTFT and TPOT by up to $67.6\%$ and $49.9\%$, respectively, over the strongest tree-speculative baseline.
Source:https://arxiv.org/abs/2608.01651PreviousWave-Based Dispatch for Circuit Cutting in Hybrid HPC-Quantum SystemsNextLantern: Finding Committable Transactions via Back-Propagation on DAGs
Related posts

We rewrote zg in Rust, cutting indexing time by 65% and peak memory by 53%
0Rust Users Forum

Michiu (now at v0.2.2!): An experimental GUI library for Windows focused on DirectComposition, wgpu, and DOD
0Rust Users Forum
Mocking a 410 Gone for a Retired Resource vs a Plain 404
0dev.to
Three MCP Python SDK Advisories, and the One an Upgrade Will Not Fix
0dev.to
We Benchmarked GPT-Image-2.6 Native <svg> Output: Path Counts, Cost & Real Edge Cases
0dev.to
When an MCP Tool Needs a Missing Value: Elicitation vs Failing the Call
0dev.to