AI & ML
HYPIC: Position-Independent Caching for Hybrid-Attention LLM Serving
arXiv cs.DC0 views
arXiv:2607.01299v3 Announce Type: replace Abstract: In retrieval-augmented generation (RAG) and agentic LLM serving, prompts are assembled from multiple independent segments into long contexts, making prefill dominate the computation cost. To reduce this cost, position-independent caching (PIC) admits KV reuse for non-contiguous segments across requests, while hybrid-attention models reduce computational complexity using linear attention. However, existing PIC methods break down on hybrid-attention models because their per-token KV-cache reuse primitives do not transfer to per-request recurrent states. In this work, we present HYPIC, the first system to accelerate hybrid-attention LLM serving with position-independent caching. For linear-attention layers, we identify the segment-cumulative transition operator as the missing algebraic primitive and cache it for each segment, enabling near-exact and constant-time state composition. For the remaining full-attention layers, we construct a small seam window to propagate hidden states through the linear-attention layers, so that cross-segment attention can be repaired by recomputation. Finally, HYPIC introduces segment parallelism, which parallelizes cache-miss prefill across instances, turning long cold requests into an accelerable workload. Evaluated across four hybrid-attention models and five workloads, HYPIC reduces time-to-first-token (TTFT) by $3.25\times$ on average and improves QPS by $1.66\times$ over Prefix Cache, while preserving task quality with a $1.71$-point gap from Full Recompute.
Source:https://arxiv.org/abs/2607.01299PreviousEkka: Automated Diagnosis of Silent Errors in LLM InferenceNextBreaking Fault Lines: Unifying TEE-Assisted BFT Consensus in Partially Trusted Worlds
Related posts
Scalable Source Selection for Machine Learning Tasks
0arXiv cs.DB
AvalancheBench: Evaluating Enterprise Agent-Driven Semantic Query Processing Through Latent World Recovery
0arXiv cs.DB
ConStruM: A Structure-Guided LLM Framework for Context-Aware Schema Matching
0arXiv cs.DB
From Certified Pairs to a Certified Hub: Reasoner-Checked Crosswalks for the Information Exchange Standard
0arXiv cs.DB
A Comparative Study of Importance Attribution to Columns in Deep Learning over Relational Databases
0arXiv cs.DB
Community Search under Member-Specific Tolerances
0arXiv cs.DB