Prompt caching with real numbers, and the bug that breaks it

You turn on prompt caching. The docs say cached tokens cost a tenth of the normal price, so you expect the input bill to drop by most of that.

A month later the bill is higher than before.

Nothing errored. No warning in the logs. The API accepted every request and answered every one correctly. It just never read the cache once, and on some providers a cache that is written and never read costs more than no cache at all.

This post prices one workload three ways: no caching, caching that works, and caching that is silently broken. Then it covers what each provider charges, when caching loses money, and how to check in five minutes that yours is hitting.

I ran into this from the other side. I was building a tool that shrinks an agent's tool output, and the obvious design was to go back and trim old results a few steps later, once you know what the agent ended up needing. It took me a while to see what that does to the bill: every trim rewrites the start of the conversation, so the cache I was counting on would never get read. I ended up deciding each result once and replaying it byte for byte after that, and I've read the cache fields on every response since.

TL;DR

• On a docs assistant with a 10,000-token fixed prefix, caching takes a request from $0.024 to $0.005 on Claude Sonnet 5.5. That's $2,400 a month down to $500 at 100,000 requests.
• Put a timestamp in the system prompt and the same setup costs $2,900: every request pays the 1.25× write price and nothing is ever read back.
• The 5-minute cache pays off from the first reuse. The 1-hour cache needs two. Log the cache fields in the usage object, because a miss never raises an error.

What is prompt caching?

Prompt caching is a discount on the part of a request you already sent. The provider keeps the processed form of the start of your prompt, and a later request that starts with exactly the same content reads it back instead of processing it again.

Two details decide everything else.

It's a prefix. The cache covers the prompt from its first token up to some point. It can't skip over a changed part and resume after it. One different token early in the prompt, and everything after it is a miss.

It's exact. The match is byte for byte. "Almost the same prompt" is a different prompt.

So the layout rule is simple: stable content first (tool definitions, instructions, reference documents), changing content last (the user's question, fresh search results, today's date).

It also doesn't change the answer. The model produces the same output it would have produced without the cache. Only the price and the time to first token move.

How much does prompt caching save?

Take a concrete case. Say you run a docs assistant: users ask questions about your product and the model answers from instructions and reference pages you send along. These numbers are an illustration; swap in your own.

• Instructions, tool definitions and reference pages: 10,000 tokens, identical on every request
• The user's question: 500 tokens
• The answer: 300 tokens
• Volume: 100,000 requests a month

On Claude Sonnet 5.5, input costs $2 per million tokens, output $10, a 5-minute cache write $2.50 and a cache read $0.10 (Anthropic's pricing page, checked November 9, 2026).

The middle row is the promise: 79% off the whole bill, because the prefix was 83% of it.

The last row is the part people skip. Writing to the cache costs 25% more than a normal request. That premium is fine when you pay it once and read the prefix a thousand times. It is not fine when you pay it on every request.

The cached part of the bill has more moving pieces than this (two cache lifetimes, different read prices per model). I wrote up how Anthropic bills cached tokens with the full table if you need a model other than Sonnet.

What does each provider charge?

The idea is the same everywhere; the rules aren't. One mid-range model per provider, USD per million tokens:

Sources: the pricing and caching pages of Anthropic, OpenAI, Google and DeepSeek, checked between October 14 and November 9, 2026. DeepSeek's prices are its peak-hour prices; off-peak is half.

Three things to take from the table.

"Automatic" doesn't mean "working". On OpenAI, Gemini and DeepSeek you don't opt in, so you also never see a setting that says it's on. The only proof is the cached-token count in the response.

The minimum is silent. A prompt shorter than the minimum runs uncached with no error. A 2,000-token system prompt caches on Claude Sonnet 5.5 and GPT-6.1 Sol but not on Gemini 3.8 Flash. Older Claude models need 1,024 to 4,096 tokens.

A write premium turns a broken cache into a cost. Where writes cost the same as normal input, a miss costs you nothing extra. Where they cost 1.25× or 2×, a miss is a surcharge.

When does prompt caching cost more than no caching?

Caching is a bet that the prefix will be read again before it expires. You pay the write premium up front and earn the discount on each reuse.

For the cached prefix on Claude Sonnet 5.5, compared with sending it uncached every time:

The math behind one cell: with one reuse, uncached is 2 × the prefix price. Cached is 1.25 (the write) + 0.05 (the read) = 1.3. That's 35% less.

So the 5-minute cache wins from the first reuse, and the 1-hour cache from the second. Anthropic refreshes the 5-minute cache for free on every hit, so steady traffic keeps it alive as long as requests keep coming.

The losing cases are the ones with no reuse:

• A feature that gets one request every ten minutes, on a 5-minute cache. Every request is a write.
• A per-user prefix (the user's own documents first) for users who ask one question and leave.
• A user who comes back after six minutes. The next request rewrites the whole history at the write price.

The one change that breaks it

Here is the line that turns $500 a month into $2,900:

system = f"Current time: {datetime.now().isoformat()}\n\n{INSTRUCTIONS}" 

It looks harmless. The model should know what time it is. But the timestamp is the first line of the prompt and it changes every second, so no two requests share a prefix. Each one writes 10,000 tokens to the cache at $2.50 per million, and no request ever reads them.

The fix is to move the changing value after the stable part:

import anthropic client = anthropic.Anthropic() response = client.messages.create( model="claude-sonnet-5-5", max_tokens=1024, system=[{ "type": "text", "text": INSTRUCTIONS, # identical on every request "cache_control": {"type": "ephemeral"}, # cache everything up to here }], messages=[{ "role": "user", "content": f"Current time: {now}\n\n{question}", # changes, so it goes last }], ) 

The timestamp is the famous one. The same bug has quieter forms:

• A request ID or user name near the top of the system prompt.
• Tools in a different order. Build the tool list from a set or an unordered map and the prefix differs between requests. Sort tools by name.
• JSON with unstable key order, or floats formatted differently between runs. Same data, different bytes.
• Editing earlier turns. Summarizing or trimming old messages changes the prefix from that point, so every later turn is written again.
• A/B testing the system prompt. Two variants are two caches, each with half the traffic.

How do I know my cache is hitting?

Read the usage object. On Anthropic, total input is split into three fields, and each is billed at a different price:

PRICES = { # USD per 1M tokens, Claude Sonnet 5.5 "input": 2.00, "cache_write": 2.50, "cache_read": 0.10, "output": 10.00, } def request_cost(usage) -> float: return ( usage.input_tokens * PRICES["input"] # after the cache point + usage.cache_creation_input_tokens * PRICES["cache_write"] # written this request + usage.cache_read_input_tokens * PRICES["cache_read"] # read from the cache + usage.output_tokens * PRICES["output"] ) / 1_000_000 def hit_rate(usage) -> float: total = usage.input_tokens + usage.cache_creation_input_tokens + usage.cache_read_input_tokens return usage.cache_read_input_tokens / total 

Send the same request twice within a minute and look at the second response:

OpenAI and Gemini report the same thing under other names: cached_tokens in OpenAI's usage details, cached_content_token_count in Gemini's usage metadata. If that number is zero on the second identical request, the prefix isn't stable.

To find what changed, log a hash of the serialized prefix with each request. Two requests that should share a cache and have different hashes give you two strings to diff.

Does caching fix an agent that keeps getting more expensive?

Partly. An agent re-sends its whole conversation on every step, so caching the history is a large win. But every new tool result still enters the conversation at the write price, and is then read again on every later step.

Here is one tool result I measured on a demo agent of mine: a test run of 7,209 tokens, most of it passing tests and progress bars around two failures. Priced on Claude Opus 5.5 ($4 input, $5 write, $0.20 read) over a 20-step run:

The token counts are measured; the prices and step count are an illustration. Caching takes 89% off. Not sending the noise takes off almost all of the rest. (Disclosure: I work on Contextwaise, which does that trimming, so weigh this section accordingly.)

The order matters. Trim a result before it is first sent and the cache never knows. Trim it three steps later and you have edited the prefix, which is the bug from the previous section.

Is prompt caching the same as semantic caching?

No, and the names get mixed up.

Prompt caching reuses the model's work on an identical prefix and still generates a fresh answer. It can't return a wrong answer, only a cheaper one.

Semantic caching sits in your own code. It stores whole responses and returns a stored one when a new question is similar enough to an old one. That skips the model call entirely, and it can be wrong: "how do I cancel my plan" and "how do I cancel my order" are close in meaning and need different answers.

They stack. Use prompt caching by default, and add semantic caching only where questions repeat almost word for word and a stale answer is harmless.

What to do on Monday

1. Log the cached-token fields from the usage object for a day of real traffic. Compute the hit rate per endpoint.
2. For any endpoint under about 80% on its second and later requests, hash the prefix and diff two requests. Look for a date, an ID or an unordered list.
3. Reorder the prompt: tools, then fixed instructions, then reference material, then everything that changes.
4. Check the prefix is above your model's minimum length. Under it, nothing is cached and nothing tells you.
5. Count reuses before choosing a lifetime: bursts within minutes fit the 5-minute cache, two or more reads an hour fit the 1-hour cache, less than that and caching that segment loses money.

What's your cache hit rate on your busiest endpoint, and what was the thing that broke it when you first checked?