An agent loop sends the same long prefix on every call: the system prompt, the tool definitions, the reference documents pinned at the top, the conversation so far. On a long session that prefix can dwarf the actual question, and you pay input price for all of it again and again. When a provider matches a repeated prefix, it reuses the earlier work and bills those tokens at a lower cache read rate.
The ratio is what matters
Every model page in the ModelSite catalog lists the cache-read price next to the input price. The ratio between the two decides what caching is worth. If cache reads cost a few percent of the input price, a stable prefix saves far more than on a model priced at a quarter of it, so look at that ratio instead of the headline input price.
The table below is calculated live from those same prices. It compares the input cost of 1,000 requests that each carry a 100,000-token prompt: once with no cache hits, once with 90% of the prompt served from cache. Output tokens are left out, because caching only changes the input side.
Input cost of 1,000 requests × 100,000-token prompt each, at a 90% cache-hit rate, calculated from live catalog prices on 2026-11-12.
Prices move, so treat any screenshot as dated. The model pages — for example deepseek-v4.1-flash-1017 and glm-5.3 — always show the current numbers.
How to structure a prompt for cache hits
- Put the stable parts first: the system prompt, tool definitions and reference material, before anything that varies.
- Put the volatile parts last: the latest user message, timestamps, request IDs. A timestamp near the top invalidates the whole prefix on every call.
- Keep the prefix byte-identical between calls. Providers generally match on an exact prefix, so even reordering the tool list can cost you the hit.
- Send follow-up requests soon after the first. Caches expire, and the lifetime depends on the provider.
- Stay on one model for a conversation. A cache belongs to a single model.
How to check that it is working
The response reports what was served from cache. In the OpenAI response format the field is usage.prompt_tokens_details.cached_tokens; Anthropic's format uses usage.cache_read_input_tokens; DeepSeek adds prompt_cache_hit_tokens, which passes through unchanged. If the counter stays at zero on a prefix you deliberately repeat, something in that prefix changes between calls.
import os from openai import OpenAI client = OpenAI( api_key=os.environ["MODELSITE_API_KEY"], base_url="https://api.modelsite.ai/v1", ) messages = [ {"role": "system", "content": LONG_STABLE_PREFIX}, # identical every call {"role": "user", "content": "Summarize the third reference document."}, # varies every call ] resp = client.chat.completions.create(model="glm-5.3", messages=messages) details = resp.usage.prompt_tokens_details cached = details.cached_tokens if details else 0 print(f"{cached} of {resp.usage.prompt_tokens} prompt tokens came from cache") Run it twice with the same long prompt; on most providers the second call reports most of the prefix as cached. DeepSeek builds its disk cache in the background, so the hit can take another call to show up. The example uses the official OpenAI Python SDK against an OpenAI-compatible endpoint, so the same check works on any OpenAI-compatible provider that reports cache hits. Provider fine print, including cache expiry, lives in the docs: OpenAI's prompt caching guide, Anthropic's prompt caching page, and DeepSeek's context caching notes. ModelSite's Anthropic-compatible endpoint reports the same cache_read_input_tokens field.
Worth knowing
- Cached tokens are billed at the cache-read rate instead of the input rate, not on top of it.
- Not every model reports cache hits, and a few price cache writes separately. The model page shows the full breakdown.
- Prefixes below a minimum length never cache at all. The threshold is provider- and model-specific: 1,024 tokens on OpenAI (GPT-5.6 and later; earlier models vary by request settings) and 512 to 4,096 tokens on Anthropic depending on the model. A short test prefix reports zero by design.
- Cache prices and off-peak windows change. The table above was calculated from the live catalog on 2026-11-12; treat it as a snapshot, not the current price.
Originally published on the ModelSite blog on 2026-10-22.
