6 Things That Quietly Break Your LLM Prompt Cache in .NET

I had a support bot that sent the same 1,300-token system prompt on every request, and I assumed the provider's prompt cache was doing its job. OpenAI and Azure OpenAI cache long prompt prefixes automatically, and for the models I priced, cached input costs a tenth of normal input. Free money, as long as the cache actually hits.

Mine didn't. When I finally measured it, only 7% of my input tokens were cached. The cause was one friendly line at the top of the system prompt: Today is 2026-11-13 09:14:07 UTC. You are helping Acme Robotics.

That line fixed nothing and broke two things at once. It's also not the only way to lose the cache without noticing. Below are the six cache breakers I now check for in every .NET LLM app, how each one shows up, and the shape of the fix.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete, tested project behind this post (a .NET 10 LLM cost router with Clean Architecture and 267 passing tests: prompt-cache linting, a small-model-first cascade and per-tenant dollar budgets, running offline against a simulator) plus a step-by-step PDF guide is inside. Join Tech Skill Builder

First, the one rule behind all six

Provider prompt caching reuses an exact prefix, matched from the first token, and only once that prefix is long enough (I plan with about 1,024 tokens, counted in 128-token steps, which is how OpenAI documents it for the models I used). Anything that changes a single byte early in the prompt makes everything after it a miss.

So every breaker below is a variation of "something early in the prompt isn't the same as last time".

1. A timestamp (or GUID, or request id) in the system prompt

The classic. "Today is ..." with seconds in it gives you a brand-new prefix every second. Request ids and GUIDs do the same thing per request.

How it shows up: cached tokens near zero, even though "the prompt never changes".

Fix: keep the date if the model needs it, just move it. Everything that changes goes into the last user message, inside a small context block. The stable part (instructions plus reference material) goes first, byte-identical every time.

public ComposedPrompt Compose(TenantInfo tenant, IReadOnlyList<PromptMessage> history, string question, AnswerFormat format, DateTimeOffset now) { // 1. stable system text first: same bytes for every tenant, every request // 2. history, append-only // 3. last user message: <context>customer, plan, date</context> + question // full implementation (format rules, fingerprint, token counts) in the complete project } 

I also lint the stable part at startup. A regex check for ISO dates, clock times, GUIDs, long numbers and phrases like "today is" catches the well-meant "let's add the date" pull request before it ships.

2. Tenant or user text inside the shared prefix

"You are helping Acme Robotics." looks harmless. But now Acme's prefix and Globex's prefix differ from the first sentence. At best you get one cache per tenant instead of one for everyone. At worst it's a smell that customer data is mixing into what should be shared instructions.

How it shows up: the hit rate looks fine for your biggest tenant and terrible for everyone else.

Fix: same as #1. The customer name and plan belong in the last message. In my linter, any tenant display name found inside the stable prefix is an error, not a warning.

3. A prefix that's too short to be cached at all

This one is sneaky because the prompt is perfectly stable. It's just under the provider's minimum cacheable length, so caching never applies. Worse: a prompt sitting just above the minimum can drop under it after one small edit, and nobody notices.

How it shows up: zero cached tokens on a prompt that never changes.

Fix: know your prefix size in tokens (count it with the same tokenizer family the model uses), and warn when it's under the minimum or within one 128-token step of it. Sometimes the right fix is putting genuinely useful stable material in there, like a support handbook. Sometimes it's accepting that this prompt won't cache.

Here's the shape of the check, with all five rules as codes:

// PC001 dynamic value in the prefix PC002 prefix below the minimum // PC003 tenant text in the prefix PC004 stable messages not first // PC005 prefix just above the minimum (info) public static IReadOnlyList<LintFinding> Lint(string prefixText, long prefixTokens, int minCacheableTokens, IEnumerable<string> tenantMarkers, bool stableFirst = true) { // regexes for dates, times, GUIDs, long ids, "today is"; length checks; tenant markers // full implementation in the complete project } 

Run against my original "naive" prompt, it flagged PC001 and PC003. Exactly the two problems in that one line.

4. Rewriting earlier turns of the conversation

Multi-turn chat caches beautifully when history is append-only: the whole previous request is a prefix of the next one. It falls apart when the app "tidies up" earlier messages, re-summarises history every turn, or injects something new near the top. Same goes for putting anything in front of the stable system messages.

How it shows up: good cache hits on turn one, then mysteriously worse as the conversation gets longer, which is exactly when caching should help the most.

Fix: stable messages first, then history exactly as it was sent, then the new message. If you must compact history, accept that the next request is a cold start and do it rarely.

5. Switching models mid-conversation

This is the one that surprised me. The prompt cache is per model. OpenAI's cache diagnostics even list model_changed as a miss reason. So if a cheap model answered turn three and the big model answered turn four, turn five on the cheap model starts cold.

That matters the moment you add any kind of routing, like a small-model-first cascade. A "cheaper" model with a cold cache can cost more than staying on the model that already has the whole conversation warm.

How it shows up: routing saves less than the price list says it should.

Fix: make the router cache-aware. Predict cached tokens per candidate model before the call, compare expected costs, and stay on the warm model when that's cheaper. The prediction is only for routing; billing always uses what the provider reports.

public static long Predict(string modelId, long inputTokens, long prefixTokens, ConversationRouteState? conversation, DateTimeOffset? prefixLastSeenOnModel, DateTimeOffset now, CacheRules rules) { // same model as the last turn, within the TTL -> previous prompt is warm // same prefix seen on this model recently -> the stable prefix is warm // otherwise 0 // full implementation in the complete project } 

6. No cache key, or one key doing all the work

prompt_cache_key helps the provider route requests with the same prefix to the same cache. Without it, hits can be patchier than they need to be. With a single key for all traffic, you can overload it: OpenAI suggests roughly 15 requests per minute per prefix and key combination.

Two .NET-specific notes from building this:

• The OpenAI .NET SDK version I used (2.14) has no typed property for prompt_cache_key. I set it through the SDK's JSON patch API, which is marked experimental (SCME0001), and I verified in a test that the field actually appears in the request body.
• Don't put the raw tenant id in the key. I send a hash, and I don't use string.GetHashCode() for it, because .NET randomises that per process and the key has to survive restarts.

public static string? For(CacheKeyMode mode, string appKey, string prefixFingerprint, string tenantId, int shards) { // None -> don't send a key // Sharded -> app : short prefix fingerprint : shard (spreads busy prefixes) // PerTenant -> app : short prefix fingerprint : stable hash of the tenant id // full implementation in the complete project } 

Measure it, don't assume it

None of the six is visible unless you read the number the provider sends back. Microsoft.Extensions.AI 10.11 maps prompt_tokens_details.cached_tokens to UsageDetails.CachedInputTokenCount, so it's one property away:

var response = await chat.GetResponseAsync(messages, options, ct); long? cached = response.Usage?.CachedInputTokenCount; // what the cache actually saved // pricing cached vs uncached input, usage records, metrics: in the complete project 

Two more cheap guards I added: a SHA-256 fingerprint of the stable prefix on every usage record (so a prompt edit lines up with the hit-rate drop in the same chart), and a small monitor that alerts when more than two different prefix fingerprints show up within one minute. That's what catches breaker #1 when it sneaks back in.

What fixing it was worth

Same 102 requests (34 support questions across 3 tenants), all sent to the large model, only the prompt layout changed:

That's the "nearly 4x" (3.86x, to be exact). Honest caveat: these come from my offline simulator running under the real OpenAI SDK and Microsoft.Extensions.AI, with a prefix cache modelled on OpenAI's documented rules and list prices for gpt-5.4. They're not a benchmark of a live model. I didn't model the cache-write pricing newer OpenAI models have either. Measure your own traffic before trusting any number, mine included.

The checklist

• [ ] No dates, times, GUIDs or request ids in the stable prefix.
• [ ] No tenant or user text in the shared prefix.
• [ ] The prefix is comfortably above the provider's minimum cacheable length.
• [ ] Stable messages come first; history is append-only.
• [ ] Routing knows the cache is per model and prices "stay warm" vs "switch".
• [ ] A prompt_cache_key is sent, hashed, and sharded if one prefix gets busy.
• [ ] Cached tokens are read from the response and logged with a prefix fingerprint.

Get the complete version

This post covers one slice of the project: keeping the prompt cache warm. What I left out is the part that turns it into a cost router. That means the small-model-first cascade with a deterministic answer verifier, hard per-tenant dollar budgets that reserve the worst case before every call and settle the real cost after, cost pricing in whole micro-dollars, the full linter and composer, the MEAI gateway with the cache key on the wire, and the tests that prove all of it.

The complete, working, tested project (.NET 10, Clean Architecture, 267 passing tests: 229 unit + 38 integration, runs offline against a simulator, no API key needed) plus a step-by-step PDF guide is in Tech Skill Builder. It's $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10: https://elitesolutions1.gumroad.com/l/TechSkillBuilder

Have you checked your own cached-token numbers lately? I'm curious which of these six people hit most often, or if there's a seventh I haven't run into yet. Tell me in the comments.