LLMOps Deep: Observability, Cost, Eval-Gated Deploys
Deep Part 10 built a real MCP server and attacked it five ways. This part moves the same system across the line into production. A tool that is safe in a demo can still bankrupt a team in production, and the failure mode is boring: nobody watches the bill, nobody watches the traces, and a bad prompt ships because nobody gated the deploy. We built the missing pieces ourselves this week: a traced gateway, a priced workload, and an eval gate that actually said no.
Three things to know before the numbers start. First, we had no API keys for this part, so there are no live provider calls in the measurements. The model in our gateway is a local extractive stand-in, and every number that depends on a provider comes from measured tokens plus published list prices, not from a live bill. We say so wherever it matters. Second, the gateway measures real local work: HTTP, retrieval, tokenization, and trace writes, all timed with real spans. Third, the eval gate ran against that gateway and returned HOLD, which is the correct answer and the most useful result in this part.
The setup: one gateway, three signals, zero API keys
The lab is a small FastAPI support gateway for a fictional outdoor gear store. A request arrives, retrieval picks help centre chunks, a local extractive model answers from the top chunk, and every step is wrapped in an OpenTelemetry span. Spans and costs land in a SQLite trace store. The workload is 100 support questions built from 25 real templates with varied order numbers, products, and amounts. The system prompt is a full support prompt: policies, few-shot examples, tool definitions, and product knowledge, 2,141 tokens measured with tiktoken.
The three signals are the whole job. Traces answer what happened on one request. Metrics answer what is happening across all requests. Logs answer what the code said while it happened. Most teams buy the dashboard first and the alerts never. We did the reverse: traces in SQLite, four numbers on a page (error rate, p50, p95, cost per request), and one alert rule. That is enough to run a small feature, and it is honest about what it does not cover.
What observability means when the model is a black box
A normal service tells you what it did. A model call tells you what it returned, and the reasoning stays inside weights you cannot inspect. So observability for LLM apps is not about opening the model. It is about recording everything around the call so a bad answer can be replayed: the prompt version, the retrieved chunks, the model name, the token counts, the latency, and the cost. Without those five, a bad answer is a mystery. With them, it is a bug with a trace id.
OpenTelemetry now has a shared vocabulary for this, the GenAI semantic conventions under the gen_ai namespace. The span for a chat call carries gen_ai.operation.name set to chat, gen_ai.provider.name, gen_ai.request.model, and the usage counters gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Span names follow the operation: chat plus the model, invoke_agent plus the agent name, execute_tool plus the tool name. Two details matter in practice. First, input_tokens is the total, and cached tokens are a subset of it, not an addition. Second, the conventions are still marked Development, and they moved to their own repository, open-telemetry/semantic-conventions-genai, so attribute names can still move between releases. Build dashboards on the stable few: model, tokens, duration, status.
Prompt and response bodies are opt-in by design, and that is the right default. Traces routinely contain customer names, order numbers, and addresses. We record token counts and chunk ids on every trace, and we record prompt text only on traces we explicitly sample for debugging, with PII redacted first. A trace store that holds every customer message in plain text is a breach waiting for a query.
For teams that do not want to build the store, the two names that come up are Langfuse and LangSmith. Langfuse is open source under the MIT licence for its core, reached v4 general availability in August 2026, and self-hosts with Postgres, ClickHouse, Redis, and S3 storage. Its cloud Hobby plan includes 50,000 units a month, and Core is 29 dollars a month for 100,000 units. LangSmith is LangChain's managed platform and bills per seat plus base traces. Our position is simple: a team with fewer than 10,000 requests a day should start with the SQLite approach in this part and move to Langfuse when the trace volume or the team size makes a shared UI worth paying for. Buying the platform before the volume exists is how a 29 dollar bill becomes a 299 dollar bill with nothing in it.
Measurement 1: the real cost of 100 support requests
We counted every token in the workload with tiktoken using the cl100k_base encoding, which is the GPT-4o family tokenizer. Counts for Claude and Gemini tokenizers differ by a few percent, so treat the cross-model comparison as close, not exact. The system prompt measured 2,141 tokens. The three retrieved chunks measured 132 tokens. The user question averaged 22.3 tokens, median 22, range 16 to 30. Total input per request averaged 2,202 tokens, and 100 requests used 229,528 input tokens. Replies were costed at an assumed 180 tokens each, because output tokens cannot be measured without a model call, and the assumption is stated so it can be challenged.
One number decides the whole bill: the system prompt is 93.3 percent of the input tokens. The user question is 1 percent. Every discussion about cost that starts with the user message is looking at the wrong end. The prompt is the bill.
ModelInput per 1MOutput per 1MCost per 100 requestsCost per requestMonthly at 10,000 requests a day
GPT-4o-mini$0.15$0.60$0.0452$0.000452$135.60
Claude Haiku 4.5$1.00$5.00$0.3195$0.003195$958.50
Gemini 2.5 Flash$0.30$2.50$0.1139$0.001139$341.70
Prices are published list prices verified against provider pages in September and October 2026, and they move, so check before budgeting. Three readings from the table. First, the same workload costs 7.1 times more on Haiku 4.5 than on GPT-4o-mini. Model choice is the biggest cost lever, bigger than any caching trick, and it is also a quality choice, so it has to be made with an eval, not a price list. Second, output tokens are a large share of the bill: 23.9 percent on GPT-4o-mini, 28.2 percent on Haiku 4.5, and 39.5 percent on Gemini 2.5 Flash at our assumed 180-token reply. Capping replies with max_tokens is a cost control, not just a style choice. Third, at 10,000 requests a day, the gap between the cheapest and the most expensive option is 822.90 dollars a month. That is a salary line in some teams, and it is decided by one string in a config file.
For the earlier cost work on this series, see our measured guide to cutting the LLM API bill. The numbers there were measured the same way, and the routing advice still holds: send the easy work to the cheap model first.
Measurement 2: prompt caching, priced per provider
The first draft of our system prompt measured 573 tokens. That is a real finding, not a mistake to hide: at 573 tokens, no provider would have cached it. OpenAI caches automatically only above 1,024 tokens, and Anthropic needs 1,024 tokens for Sonnet and 2,048 for Haiku. A short prompt gets zero caching benefit, and most teams never check. We grew the prompt to 2,141 tokens with few-shot examples, the full returns and delivery policies, tool definitions, and product knowledge. That is the real shape of a production support prompt, and it clears every threshold.
The providers do not cache the same way. Anthropic is explicit: mark the stable prefix with a cache_control breakpoint, pay 1.25 times the base input price to write a 5-minute cache (2.0 times for the 1-hour tier), and pay 0.10 times to read it. OpenAI is automatic above 1,024 tokens, reads cost 50 percent of the input price, and there is no write charge on GPT-4o generation models. The cache lives 5 to 10 minutes of inactivity, and it matches in 128-token blocks. Google gives 75 percent off context cache reads with a 1-hour minimum, and charges separately for holding the cache, which we did not model because the figure depends on how long the cache is held.
We priced the same 100 requests with the 2,141-token system prompt cached: one write, 99 reads, full price on the user question and retrieved context.
ModelCache writeCache readCost per 100 requests cachedSaving vs no cacheMonthly cached at 10,000 requests a day
GPT-4o-miniFree50% off input$0.029335.1%$87.90
Claude Haiku 4.51.25x input90% off input$0.129359.5%$387.90
Gemini 2.5 FlashStorage charge75% off input$0.066241.9%$198.60
The break-even math is the part most write-ups skip. On Anthropic, the write premium is 0.25 times the base price, and each read saves 0.90 times. The cache pays for itself on the first reuse: two requests cost 1.35 times the base price for the cached prefix instead of 2.00 times uncached. After 100 requests, the cached prefix costs 11.15 times base instead of 100 times. The catch is the 5-minute TTL. If requests arrive less often than every 5 minutes, the cache expires and every request pays the write premium again. Caching rewards steady traffic and punishes sporadic traffic, which is the opposite of what most teams assume.
One structural rule decides hit rates on every provider: stable content first, volatile content last. The cache key is the exact prefix, so a timestamp, a session id, or a user name near the front invalidates everything after it. Put the system prompt, tool definitions, and few-shot examples first, and the user message and retrieved chunks last. OpenAI returns cached_tokens in the usage object, and Anthropic returns cache_read_input_tokens, so hit rate is measurable. If the number is zero, the prompt order is wrong, not the feature.
Measurement 3: the harness, and the blind spot it found
The gateway wraps each request in spans using the OpenTelemetry SDK: a root span named http.post /ask with the gen_ai attributes, a retrieve span, a chat span named chat gpt-4o-mini, and a store step that writes the trace and its spans to SQLite. We fired 50 real requests through the live server, then 5 chaos requests that forced the model step to fail with a 500. The client saw all 5 failures. The trace store saw none of them.
That was the first run, and it is the most important number in this part: 50 traces stored, 0 errors recorded, real error rate 9.1 percent. The cause was structural. The store step ran after the model step, inside the same request path, so a crash in the model step skipped the write entirely. The store only ever saw successes, which is exactly the data that makes a dashboard look healthy while users get errors. Every team that stores traces after the work completes has this blind spot until it is measured.
The fix is small and it belongs in a finally block: store the trace after the root span ends, and store it on error too, with the error status and the failure reason. The second run stored 55 traces: 50 ok, 5 error, and the 9.1 percent error rate was visible in the store for the first time. The rule we took from this is simple: a trace store that cannot show failures is not observability, it is a success log.
try: with tracer.start_as_current_span("http.post /ask") as root: root.set_attribute("gen_ai.operation.name", "chat") root.set_attribute("gen_ai.request.model", "extractive-v1") retrieve(question) # span: retrieve answer(question) # span: chat gpt-4o-mini except RuntimeError as e: status = "error" finally: # store AFTER the root span ends, and on error too store_trace(trace_id, status, tokens, cost, latency, spans) Latency across the 50 successful requests, measured inside the gateway: mean 0.86 ms, p50 0.63 ms, p95 0.79 ms, max 11.71 ms. Per-span means: the root span 0.88 ms, the chat span 0.50 ms, retrieve 0.23 ms, and dependency resolution 0.11 ms. The slowest trace took 11.71 ms, and its chat span took 11.13 ms: first-request warmup, not a slow model. Token totals across the 50 requests were 5,179 input and 1,604 output, costing 0.001739 dollars at GPT-4o-mini list prices, or 0.000035 dollars per request. Be clear about what these numbers are: the model is a local stand-in, so sub-millisecond latencies describe the harness and the pipeline, not a provider. The honest reading is that the harness itself is cheap, and that the numbers to watch in production are the provider's latency and the error rate, not our local timings.
One surprise was free. FastAPI 0.162 ships built-in telemetry: once a tracer provider exists, it emits spans for routing, dependencies, the endpoint, and serialization with no instrumentation package installed. We verified that by listing the virtual environment: no instrumentation package is present, and the spans carry code.function.name for the endpoint. The lesson is that the framework already gives you the server half of the trace. The half you still have to write is the model half: tokens, cost, prompt version, and retrieval ids. Nobody else can know those.
Measurement 4: the eval gate said hold
Deep Part 8 built a full eval harness; this part uses one to make a shipping decision. The eval set is 24 support questions, each with one fact the answer must contain, taken from the help centre chunks. Two gateway versions ran against it. Version 1 retrieves the top 3 chunks by token overlap. Version 2 retrieves 5 chunks and reranks them by raw overlap count, then answers from the top 3. The gate rule was fixed before the run: promote version 2 only if its score rises, its p95 latency stays under 1,200 ms, and its cost per request stays within 10 percent of version 1.
Both versions scored 18 out of 24, with the same 6 misses. The gate returned HOLD. The rerank shuffled ranks 2 and 3 but the top chunk stayed wrong on the same six questions: return shipping, damaged items, VAT invoices, duplicate charges, starting a return, and delayed parcels. In each case the question shares few words with the chunk that holds the answer, so overlap scoring picks a confident wrong chunk. A better rerank cannot fix a top-1 selection that is wrong; the fix is to answer from the top 3 chunks instead of only the top chunk, or to add the missing keywords to the chunks. That is a retrieval fix, not a rerank fix, and the gate is what forced the diagnosis instead of a confident ship.
Cost per request was identical at 0.000036 dollars. Client-observed latency was 47.51 ms mean for version 1 and 50.47 ms for version 2, with p95 at 54.63 ms and 59.38 ms. But the work itself took about 1 ms. The gap is the second finding of this part: the synchronous SQLite write in the request path cost about 50 times the work it recorded. A trace store write belongs in a background batch export, never in the request path. OpenTelemetry ships a batch span processor for exactly this reason, and a production gateway should use it or an async exporter, not a blocking insert per request.
The canary math decides how long a deploy needs to bake. To detect a quality drop from 80 percent to 75 percent at 95 percent confidence, each arm needs about 534 requests. At a 10 percent canary on 1,000 requests a day, that is 100 canary requests a day and 5.3 days to detect the drop. At a 50 percent canary it is 1.1 days. The standard error of a proportion at n equals 100 and p equals 0.80 is 0.040, so a 5-point drop is only 1.25 standard deviations, which is noise. A 10 percent canary on a small feature cannot see a 5-point quality drop in a day. Either raise the canary share or wait a week, and write the number down before the deploy, not after the complaints.
The cost controls that actually move the bill
Five controls, in the order we would apply them. First, route small first. On our workload Haiku 4.5 costs 7.1 times GPT-4o-mini per request. Send the easy 80 percent to the cheap model and escalate the hard 20 percent. The router is a classifier with a confidence threshold, and the threshold is set by an eval, not by a guess. Second, cache the stable prefix. On our workload that saved 35.1 percent on GPT-4o-mini and 59.5 percent on Haiku 4.5, and the only work is prompt ordering and a hit-rate check. Third, cap output. Output tokens were 24 to 40 percent of our bill, and a max_tokens limit plus stop sequences is a one-line change. Fourth, set budgets that bite. A daily budget per feature, an alert at 80 percent, and a hard stop at 100 percent that degrades to a cheaper model instead of failing open. A budget that only sends an email is a suggestion. Fifth, sample traces. Keep 100 percent of errors and 1 percent of successes. Storing every trace at 10,000 requests a day is a storage bill that grows faster than the model bill, and nobody reads the middle 99 percent.
Two controls we would not reach for first: semantic caching and model fine-tuning. Semantic caching, answering from a cache when a new question is close to an old one, saves real money at high volume with repeated questions, but it adds a correctness risk (a near-duplicate question with a different answer) and a cache invalidation problem. Fine-tuning to shorten prompts is a multi-week project with its own eval burden. Both are real tools for later, and both are the wrong first move when routing and prompt caching are still on the table.
Our LLMOps checklist
Every request carries a trace id from the gateway to the model call and back, so one id answers any customer complaint. Every trace records model, prompt version, token counts, cost, latency, and the ids of retrieved chunks, and never records raw prompt text by default. Error rate, p50, p95, and cost per request are the four numbers on the page, with alerts on error rate and daily spend, not on latency alone. Every prompt version is stored with its eval score, and no version ships without beating the current one on the same eval set. The canary share and the detection window are written down before the deploy. The trace store writes asynchronously, and failures to write traces never fail the request. That is the whole checklist. It fits on one page, and every item in it was earned by a measurement in this part.
What we would not do
We would not buy a dashboard before the first alert exists. A dashboard that nobody watches is a screenshot, not observability. We would not log full prompts by default. The first time a customer pastes a password into a support chat, that password is in the trace store, and the team that has to delete it is us. We would not write traces synchronously in the request path; we measured that cost at 50 times the work. We would not run a canary without the sample-size math, and we would not call a 10 percent canary a safety net when it needs 5 days to see a 5-point drop. We would not upgrade a model on a benchmark score alone; the eval that matters is our own questions, our own chunks, and our own gate. And we would not treat a falling bill as a win without checking the eval score in the same week. A cheaper model that answers worse is not a saving, it is a refund queue.
What to do this week
Count the tokens in your own system prompt with tiktoken and check it against the 1,024 and 2,048 thresholds. If it is under 1,024, you are paying full price on every request and caching cannot help until the prompt grows. Add one span to your model call with the five attributes from this part, and store it somewhere you can query. Fire five forced failures at your own gateway and check whether the error rate in your store matches the error rate your users saw. If it does not, your store has the blind spot we found. Then write one gate rule with numbers in it, run your current prompt against it, and see whether it passes. Most teams find it does not, and that is the point: the gate is supposed to catch the next bad deploy, and the first thing it catches is usually the current one.
What is Next in This Series
Deep Part 12 closes the series with Python Deep: Asyncio Internals and FastAPI at Scale. It takes the gateway from this part and asks what happens under real concurrency: how the event loop schedules the work, where a blocking call stalls every request, and how FastAPI behaves when the model call is slow. The observability from this part becomes the measurement tool for that one, because a p95 latency problem is only visible when the traces are already there. If you are following along, keep the gateway running: the next part measures it.
Earlier in this series: Deep Part 1 on attention, Deep Part 2 on tokenization, Deep Part 3 on training and scaling laws, Deep Part 4 on inference and KV cache, Deep Part 5 on context windows and RoPE, Deep Part 6 on RAG chunking, Deep Part 7 on hybrid search and reranking, Deep Part 8 on evals, Deep Part 9 on LangGraph, and Deep Part 10 on MCP. The original series ends with Part 8 on shipping AI to production, which is where this deep part picks up.
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.