LoopVQ compresses attention caches to sub‑bit precision, turning the KV cache memory wall into a speed lever [1]. By encoding multiple loop steps jointly and fine‑tuning the codebooks, the method fits each element into less than one bit while preserving the model’s functional behavior. The implementation plugs directly into SGLang’s serving stack, requiring no retraining of the underlying transformer.
Prior to LoopVQ, KV cache quantization methods for standard LLMs used vector‑quantization techniques but did not address the redundancy specific to looped transformers; looped transformers still suffered a linear cache growth that dominated memory usage [1]. The authors observe that “the KV cache … becomes the main memory bottleneck” because each repeated layer stores its own keys and values, and that the cache exhibits high redundancy along the loop axis. Existing vector‑quantization schemes therefore left most of the latency on the table.
At 0.5 bits per element LoopVQ halves decode latency for long contexts, reaching “about half of the BF16 latency at 32K tokens” [1]. This reduction comes with negligible loss in downstream reasoning and retrieval performance, matching full‑precision results on chain‑of‑thought benchmarks despite the aggressive compression. The latency gain scales with context length because cache reads dominate the decode pipeline.
The throughput gains are even more striking: LoopVQ “raises the peak serving throughput by up to 4.3× and 6.1× on the 1.4B and 2.6B Ouro models compared to the BF16 baseline” [1]. For typical batch sizes this translates into roughly a two‑fold increase in overall request‑per‑second capacity, while keeping the same hardware footprint. The advantage widens as generated sequences grow longer and BF16 batches shrink.
The paper evaluates LoopVQ primarily on the 1.4B and 2.6B Ouro models (and also on Huginn models) and relies on a post‑training codebook fine‑tuning step; applying LoopVQ to other architectures may require bespoke calibration. Moreover, the 0.25‑bit regime only retains “much of the BF16 accuracy” when combined with TaSQ, indicating that ultra‑low precision still incurs a quality trade‑off. These constraints suggest that broader adoption will depend on tooling that automates codebook adaptation.
If serving pipelines continue to benchmark against raw BF16 latency, they will vastly under‑report real‑world performance potential. The field should replace the BF16 baseline with a sub‑bit KV cache configuration when reporting long‑context decoding speeds, because LoopVQ demonstrates that a simple cache‑level compression can double throughput without changing model parameters.
