Backend
Inside the Trap: A Renewal Law for Prefix-Cache Collapse in LLM Agent Serving
arXiv cs.DC0 views
arXiv:2611.08902v1 Announce Type: new Abstract: Prefix caching lets an LLM agent's follow-up turns reuse cached KV, but a burst can tip an engine into a trap in which every turn misses and the backlog does not drain, at a load it served before. In registered paired tests on two engines, two GPUs and two models (passed at 9 of 10 rates), a 200-session burst left the engine at zero hits to the end in all 30 runs, while every matched no-burst run stayed good and a session cap prevented the trap in all 30 capped runs. We analyse an idealization of the trapped state exactly: for a fluid FIFO server on which every turn misses, we prove that along FIFO orbits its cycles obey a linear renewal recursion. As is classical, the trap drains if and only if arrivals are below the all-miss capacity; above it, a Perron root fixes the dynamics: every trapped state converges to a known self-similar regime in which the gap between turns grows linearly and the turns served are young and short. For our workload the backlog grows at 0.6 to 0.8 of the excess load and sessions complete faster than the all-miss capacity, a youth premium with turn-count and cost-growth factors, so in the model a trap's measured throughput overstates the recovery threshold. Retrospectively, with nothing fitted to trap data, the theory matches 57 stuck runs: the mean served prompt within 3% in 50 runs, completions above the all-miss capacity in every run, and the pooled gap growth. Blind in outcome, 4 runs trapped by a tool slowdown met 4 of 5 pre-set scores, cut traps persisted, and with a 20-step budget, trapped runs completed below the all-miss capacity, as predicted. Of two exits, refusing arrivals sheds sessions; a session cap restores reuse: in registered tests it rescued 9 of 9 established traps at the same load without shedding, and the engine stayed good after release. We do not prove that the engine converges to the model.
Source:https://arxiv.org/abs/2611.08902PreviousAdaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement LearningNextMore with Less: Efficient Parallel Scientific Code Translation via an Agent-Skill Workflow
Related posts

When "Not Found" Doesn't Mean "Never Happened"
0dev.to
Blameless Postmortems: A Leader’s Guide to Learning From Incidents
0dev.to

The Model Predicts Itself
0dev.to
Designing Clean REST APIs: Common Anti-Patterns and How to Fix Them
0dev.to

Dengan osFoundry Operator, agen menjalankan aplikasi dan setiap aksinya tercatat
0dev.to
Hold-Out Self-Validation Cannot Certify the Accuracy of Photogrammetric Reconstructions
0arXiv cs.RO