vLLM Prefix Caching: One Timestamp Dropped My Hit Rate From 89% to 3%

On Tuesday my support bot's time to first token roughly tripled, and nobody had touched the model, the GPU, or vLLM.

The only change in the deploy was one helpful line at the top of the system prompt:

Current date and time: 2026-10-27T14:03:22Z 

That line cost me almost all of my vLLM prefix caching. The server's stats line went from a prefix cache hit rate around 89% to around 3%. Every request was now prefilling a 2,400-token system prompt from scratch, because the timestamp at token 8 made every request look brand new to the cache.

This post explains the mechanism: how vLLM prefix caching decides what counts as "the same prefix", why one changed token near the top throws away everything after it, and how I order prompts now.

TL;DR

• vLLM prefix caching (Automatic Prefix Caching, APC) reuses KV cache blocks only for an exact token prefix. Matching content further down the prompt doesn't count.
• The prompt is split into fixed-size blocks (16 tokens by default). Each block's hash includes the hash of the block before it, so one changed token invalidates that block and every block after it.
• A timestamp, request ID, user name, or shuffled tool list near the top of the prompt drops your hit rate to roughly zero, even if 99% of the text is identical.
• Fix: put static content first (instructions, tool schemas, few-shot examples), volatile content last, and coarsen anything time-based to the granularity you actually need.
• Check it with a tokenizer diff: find the first divergent token between two real requests and divide by the block size. That number is how many blocks can be reused.

What does vLLM prefix caching actually cache?

vLLM prefix caching stores the KV cache blocks computed during prefill and reuses them when a new request starts with the exact same tokens. It doesn't cache text, embeddings or responses. It caches the attention keys and values for full blocks of tokens.

vLLM's PagedAttention already stores KV cache in fixed-size blocks, like pages in virtual memory. Prefix caching gives each full block an identity so that a later request can point at an existing physical block instead of computing it again.

On my setup (Qwen2.5-7B-Instruct, one GPU, recent vLLM where prefix caching is on by default), the relevant knobs look like this:

vllm serve Qwen/Qwen2.5-7B-Instruct \ --enable-prefix-caching \ --block-size 16 

The flag is explicit here so the intent is obvious to the next person. The block size is the number that matters for everything below.

Why does one changed token break the whole prefix cache?

Because block hashes are chained. A block's identity is not "these 16 tokens". It is "these 16 tokens, following exactly this history".

Simplified, the hashing works like this:

BLOCK = 16 def block_hashes(token_ids, extra_keys=None): parent = None hashes = [] for start in range(0, len(token_ids), BLOCK): block = tuple(token_ids[start:start + BLOCK]) if len(block) < BLOCK: break # partial blocks are never cached h = hash((parent, block, extra_keys)) hashes.append(h) parent = h # the next block inherits this block's identity return hashes 

extra_keys covers things like a LoRA adapter ID or multimodal inputs, so two requests with identical text but different adapters don't share KV.

This isn't a lazy implementation choice. It falls out of how attention works. The key and value vectors at position 500 depend on every token at positions 0 through 499, because each layer's hidden state at position 500 has attended to all of them. If token 8 changes, the KV for token 500 is genuinely different math. Reusing it would give you wrong outputs, not just slow ones.

So the chain is the correctness guarantee. And it means the cache only ever matches from the start.

How many blocks did my timestamp actually cost?

All of them. My system prompt was about 2,400 tokens: instructions, eight tool schemas as JSON, and four few-shot examples. With a block size of 16, that's 150 full blocks.

At the top, block 0 differs between any two requests that land in different seconds. Block 0's hash changes, which changes block 1's parent, which changes block 1's hash, and so on down the chain. 150 blocks recomputed per request.

The 3% residual hit rate I saw was requests that happened to arrive within the same second, plus multi-turn conversations re-reading their own earlier turns.

How do I find what's breaking my prefix cache?

Diff the token IDs of two real requests, not the strings. Strings hide the problem: chat templates inject headers, whitespace normalizes differently, and JSON serialization can reorder keys before you ever see the text.

This is the script I keep around now:

from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct") BLOCK = 16 def ids(messages, tools=None): return tok.apply_chat_template( messages, tools=tools, tokenize=True, add_generation_prompt=True ) def first_divergence(a, b): for i, (x, y) in enumerate(zip(a, b)): if x != y: return i return min(len(a), len(b)) a = ids(request_1["messages"], request_1.get("tools")) b = ids(request_2["messages"], request_2.get("tools")) i = first_divergence(a, b) print(f"diverge at token {i}") print(f"reusable blocks: {i // BLOCK} of {len(a) // BLOCK}") print("context:", repr(tok.decode(a[max(0, i - 10):i + 10]))) 

Grab two logged requests from different users a few minutes apart and run it. If the divergence index is small, the context line shows you the exact culprit.

When I ran it after fixing the timestamp, it caught a second problem I hadn't noticed.

What else silently breaks prefix caching?

Anything that varies per request and sits before your large static content. The ones I've hit or seen in other people's code:

1. Timestamps with seconds. The classic. Most prompts need the date, not the second.
2. User name or account tier in the first line. "You are helping Dana, a Pro customer" means every user has a private cache. Fine for a chatbot with long conversations, terrible for a high-volume classifier.
3. Tool schemas built from a set or a dict merged in a different order. This was my second bug. Tools were assembled from two plugin registries, and the merge order depended on which registry finished loading first. Same tools, different token sequence, about half the time.
4. Retrieved RAG passages placed before the instructions. Retrieval output changes every request. If it comes first, your long instruction block never gets reused.
5. Per-request IDs or trace tags pasted into the prompt "for debugging".
6. Random few-shot sampling. Rotating examples per request is a valid technique, but it costs you the cache from the first rotated example onward.

None of these throw errors. Responses stay correct. You just pay full prefill on every request and wonder why latency climbs under load.

How should I order a prompt for vLLM prefix caching?

Order it from most stable to least stable. The cache rewards you for every token of shared prefix and gives you nothing for shared content after the first difference.

The layout I use now:

STATIC_SYSTEM = load("instructions.md") # never changes per deploy TOOLS = sorted(all_tools, key=lambda t: t["name"]) # deterministic order FEWSHOT = load("examples.md") # fixed set def build_messages(user_msg, user, retrieved): system = ( STATIC_SYSTEM + "\n\n" + FEWSHOT + f"\n\nToday's date: {date.today().isoformat()}" # day, not second ) context = "\n\n".join(p.text for p in retrieved) return [ {"role": "system", "content": system}, {"role": "user", "content": f"Customer: {user.name} ({user.tier})\n\n" f"Context:\n{context}\n\nQuestion: {user_msg}"}, ], TOOLS 

Three changes did the work:

• Date moved to the end of the system prompt, coarsened to the day. The whole system block now matches for everyone, all day. At midnight the last block or two change, and that's it.
• Tools sorted by name. Deterministic serialization, same token IDs every time.
• User identity and retrieval moved into the user turn. They still reach the model; they just stop poisoning the shared prefix.

After the deploy, the hit rate went back to the high 80s and TTFT returned to where it had been before Tuesday.

Does this apply to hosted APIs too?

Yes, the same prefix rule applies to hosted prompt caching. Anthropic's prompt caching matches an exact prefix up to the cache_control breakpoints you set, and OpenAI's automatic caching also matches on the prompt prefix. Neither can reuse anything after the first difference, for the same attention reason as vLLM.

The difference is that hosted APIs cache less finely than vLLM's 16-token blocks and add minimum lengths, so a volatile line near the top costs you the same way, just billed per token instead of shown in TTFT.

Things to check before you blame the cache

Not every low hit rate means a prompt-order bug:

• Eviction under memory pressure. Cached blocks live in free KV memory and get evicted when running requests need the space. A high-concurrency spike can lower hit rate even with perfect prompts.
• Different LoRA adapters don't share blocks, by design.
• Short prompts. Anything under one full block never gets cached at all.

If the token-diff script shows a late divergence and the hit rate is still low, look at memory and concurrency next.

So why did one timestamp kill my vLLM prefix caching?

vLLM prefix caching only reuses KV cache blocks that match an exact token prefix, and each block's hash includes the hash of the block before it. A timestamp near the top of the system prompt changed block 0 on every request, which changed the identity of all 150 blocks after it, so the server recomputed the full 2,400-token prefix on every call and the hit rate fell from about 89% to about 3%. The fix is to order prompts from most stable to least stable: static instructions, sorted tool schemas and fixed examples first, then a day-level date, with user identity and retrieved context in the user turn. Then verify by diffing the token IDs of two real requests and checking where they first diverge.

Written by the developer behind Preterview, an interview prep platform.