Backend
ReKV: Overlapping Quantized KV Transfer and Suffix Recomputation
arXiv cs.DC0 views
arXiv:2611.08770v1 Announce Type: new Abstract: Prefill--decode disaggregation moves a prompt's key--value (KV) cache between workers before the first token can be generated. Reducing this handoff to either cache compression or recomputation misses a useful choice: which layers to transmit, at what precision, and which layers to execute again on the decode worker. ReKV transfers a quantized prefix KV cache and one BF16 boundary state, then reconstructs a contiguous suffix while the prefix is in flight. In a model-shaped synthetic benchmark on cross-node H100s, the Q8 configuration meeting separately evaluated quality criteria reduces 8K handoff latency by $2.36\times$, $2.84\times$, and $3.65\times$ over BF16 transfer for Qwen2.5-7B, Qwen2.5-3B, and Qwen3-8B. It is also faster than the best measured transfer-only or recompute-only action at those points. A separate real-weight BF16 recomputation experiment reduces prefill-to-token-return time by $1.16$--$1.63\times$. The preferred action changes with prompt length and link bandwidth.
Source:https://arxiv.org/abs/2611.08770PreviousThe GPU Needs an Operating SystemNextAdaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement Learning
Related posts
Building a Private AI Control Plane for PS5 Porting: What We Learned After SQLite Locked Us Out at 2 AM
0dev.to
Monolith vs Microservices: A Founder's Guide
0dev.to
Sentinel Dev Diary: Read First, Run Second
0dev.to
A website for the café down the street, prepared before they ask
0dev.to

Where Temporary Files Come From and Why They Can Leak Office Data
0dev.to

How to Accept USDC Payments on EVM: A Reorg-Safe Stablecoin Payments Detector in Node.js
0dev.to