LLM Inference Cost in 2026: Why Cost per Finished Task Beats Cost per Token

Most conversations about LLM cost start and end with a price table: dollars per million input tokens, dollars per million output tokens, pick the cheapest model that passes a demo. That table is necessary, but it measures the wrong unit. Your users do not buy tokens. They want a support ticket triaged, an invoice extracted, a contract clause flagged or a pull request reviewed.

The unit that matters is cost per finished task: everything you paid, across every call, retry, tool loop and fallback, divided by the number of tasks that actually reached your quality bar. Latency works the same way. Users feel the time until the task is done, not the time to the first token of one call.

This article is for engineers and tech leads running LLM features in production, with a focus on Germany and the EU. It covers pricing mechanics, why cheaper tokens can mean pricier tasks, measurement, the main levers, self-hosting break-even and EU hosting, and ends with a checklist.

All prices below were checked on 6 November 2026 and will change. Epoch AI found that the price of reaching a fixed performance level has fallen between 9x and 900x per year, depending on the milestone [1]. Treat every price here as an example, and any routing decision based on today's price table as short-lived.

How token pricing actually works in late 2026

Input versus output

Output tokens cost several times more than input tokens. OpenAI lists gpt-6.1-sol at $2.00 per million input tokens and $10.00 per million output tokens, and gpt-6-luna at $0.10 and $0.50 [2]. Anthropic lists Claude Sonnet 5.5 at $2 and $10, and Claude Opus 5.5 at $4 and $20 [3]. In all four cases output costs 5x input.

With a 5:1 price ratio, output dominates the bill as soon as a task produces more than one output token for every five input tokens. Many benchmark sites assume a fixed mix. Artificial Analysis, for example, blends input and output prices at a 3:1 ratio [4]. That is a reasonable default for comparing models, but your workload has its own ratio, and you should compute it from your logs instead of borrowing someone else's.

Token counts also depend on language. OpenAI's rule of thumb is about 4 characters or three quarters of a word per token in English, and it notes that non-English text can cost more per character [5]. If your product serves German users, measure token counts on German prompts and German outputs, not on English test fixtures.

Some input is invisible in your code. When you pass tools to Claude models, Anthropic adds a tool-use system prompt of 286 tokens for the listed models, on top of the tool schemas themselves [3]. Every tool definition you send is input you pay for on every call.

Cached input

All three large API vendors now discount repeated prompt prefixes, but the mechanics differ.

• OpenAI caches automatically. The minimum cacheable prompt is 1,024 tokens for GPT-5.6 and later, the default cache lifetime is 30 minutes (12 hours can be requested in the Responses API), and cache writes on these models cost more than standard input [6]. Cached input for gpt-6.1-sol is listed at $0.10 against $2.00 for uncached input [2].
• Anthropic uses explicit cache_control breakpoints. A 5-minute cache write costs 1.25x the base input price, a 1-hour write costs 2x, and a cache read costs 0.1x, or 0.05x on Claude Opus 5.5 and Sonnet 5.5 [3]. Prompts below a model-specific minimum (between 1,024 and 4,096 tokens for the models listed) are processed without caching, and no error is returned [7].
• Google Gemini caches implicitly on Gemini 2.5 and newer models, and lists a 90% discount on cached tokens plus prorated storage for caching [8].

The break-even point for Anthropic-style pricing is easy to compute. With a 5-minute cache, two requests sharing a prefix cost 1.25 + 0.1 = 1.35 units of prefix price instead of 2.0, so caching pays off on the first reuse. With a 1-hour cache, two requests cost 2.0 + 0.1 = 2.1 units, which is more than 2.0, so you need at least two reuses before the longer TTL pays off.

Caches are prefix matches, so small changes break them. Anthropic documents that modifying tool definitions invalidates the entire cache, and that changing thinking settings invalidates message-level cache entries [7]. Changing the thinking budget or the effort level between requests also invalidates cache breakpoints [9][10]. OpenAI's guidance is to put stable instructions and shared reference material first and dynamic content such as timestamps or user-specific text at the end [6].

Reasoning tokens

Reasoning ("thinking") tokens are the most common source of surprise on an invoice. OpenAI states that reasoning tokens are not visible through the API but still occupy context and are billed as output tokens, reported as output_tokens_details.reasoning_tokens [11]. Anthropic bills the full internal thinking as output tokens even when you only receive a summary, and states that the billed output count does not match the visible token count [10].

Two details matter for cost control:

• OpenAI recommends reserving at least 25,000 tokens for reasoning and outputs when you start experimenting with reasoning models [11]. That is the order of magnitude a single call can consume.
• Anthropic notes that max_tokens is a hard cap per request, but in a tool-use loop each request has its own max_tokens, so it "doesn't bound the whole turn's spend" [10]. A task-level budget has to live in your orchestrator.

Batch, flex and paying for speed

If a task does not need an answer within seconds, you can usually halve its price.

• Anthropic's Batch API gives a 50% discount on input and output tokens [3].
• Gemini's Batch API runs at 50% of standard cost with a 24-hour target turnaround, and Gemini Flex inference also costs 50% of standard with a target of 1 to 15 minutes [8].
• OpenAI's Flex/Batch prices are half the standard rates, for example $1.00 input and $5.00 output for gpt-6.1-sol [2].
• Azure OpenAI Global Batch and Data Zone Batch deployments are 50% cheaper than the corresponding standard deployments, with a 24-hour target and no real-time SLA [12].

The opposite direction exists too. OpenAI renamed Priority processing to Fast mode on 30 July 2026; Fast mode for gpt-6.1-sol is listed at $4.00 input and $20.00 output, and an Ultrafast tier at $12.00 and $60.00 [2]. Gemini Priority inference costs 75% to 100% more than standard [8]. Latency is now an explicit line item.

Long context and region multipliers

Prompt length can change the per-token price. Claude Haiku 5.5 charges higher prices for any request whose prompt exceeds 100,000 tokens, counting cache reads and writes toward that length, while other Claude 4.6 and later models charge the same rate across the full 1M-token window [3].

Data residency also costs money. OpenAI charges a 10% uplift on regional processing endpoints for models released on or after 5 March 2026 [2]. Anthropic applies a 1.1x multiplier to all token categories when you request US-only inference with inference_geo [3]. If you need regional processing, add that multiplier to your cost model from day one.

Why a cheaper model per token can cost more per task

Four mechanisms turn a low token price into a high task price.

More calls. Agentic tasks are loops: the model plans, calls a tool, reads the result and continues. Every step resends a growing context. Anthropic reported that its agents typically use about 4x more tokens than chat interactions, and multi-agent systems about 15x more [13]. A model that needs six steps where another needs four pays for that context six times.

Longer reasoning. Reasoning models can spend many tokens on easy inputs. A study of o1-like models found they used on average 1,953% more tokens than conventional models to answer simple problems, and that a mitigation reduced token output by 48.6% on MATH500 while maintaining accuracy for QwQ-32B-Preview [14].

Retries. If the output fails validation (broken JSON, a missing field, an answer that fails a business rule), you pay again. When OpenAI introduced Structured Outputs, it reported that the new model scored 100% on its complex JSON schema evals while gpt-4-0613 scored less than 40% [15]. Every percentage point of format failure is a retry you pay for.

Failure handling. Some tasks fail after all retries and go to a stronger model or a human. Kapoor et al. showed that agents with similar accuracy can differ in cost by almost two orders of magnitude; on HumanEval, a simple baseline reached about 93% accuracy for about $2.45 in total, while a more complex agent architecture cost about $134.50 for 88% [16]. Accuracy and cost have to be evaluated together.

A formula for cost per finished task

Start with the cost of a single model call, using per-million-token prices:

c_call = ( N_in * p_in + N_cache_read * p_cache_read + N_cache_write * p_cache_write + N_out * p_out ) / 1,000,000 * m_region N_out includes reasoning tokens. m_region is 1.0, or 1.1 for the regional uplifts quoted above. 

An attempt is the sequence of calls needed to try the task once, plus any validation calls:

C_attempt = sum of c_call over all calls in the attempt (+ tool and validation costs) 

If an attempt succeeds with probability s and you allow at most k attempts, the expected number of attempts is a truncated geometric series:

E[attempts] = (1 - (1 - s)^k) / s P(all k attempts fail) = (1 - s)^k 

If failed tasks are escalated at cost E (a stronger model, or a person's time), the expected cost per task is:

C_task = C_attempt * (1 - (1 - s)^k) / s + (1 - s)^k * E 

If failed tasks are simply dropped, divide total spend by the number of successful tasks instead:

C_finished = C_attempt * E[attempts] / (1 - (1 - s)^k) 

Worked example (illustrative)

The numbers below are illustrative assumptions, not measurements. They show how the formula behaves. Prices are the OpenAI list prices quoted above; the token counts and success rates are invented to make the mechanics visible.

Task: classify a support ticket and extract structured fields, using a small agent loop with tools. Both models share a 6,000-token static prefix (system prompt plus tool definitions) that is read from cache on every call. Cache writes are ignored because they amortise across many tasks. Each model gets at most two attempts.

Per call, Model A costs 2,000 x $2 + 6,000 x $0.10 + 1,100 x $10, divided by one million, which is $0.0156. Four calls make $0.0624 per attempt. Model B costs $0.0025 + $0.0003 + $0.0090 = $0.0118 per call, and six calls make $0.0708 per attempt.

With k = 2, Model A needs (1 - 0.1^2) / 0.9 = 1.10 expected attempts and Model B needs (1 - 0.25^2) / 0.75 = 1.25. Expected spend per task is $0.0686 for A and $0.0885 for B. Model A finishes 99% of tasks and Model B finishes 93.75%, so the cost per finished task is $0.0693 for A and $0.0944 for B.

Model B is half the price per token and about 36% more expensive per finished task. It also sends 6.25% of tasks to the escalation path instead of 1%, so any non-zero escalation cost E widens the gap by 0.0525 x E per task. On latency, B generates about 13,500 output tokens per task in expectation against about 4,840 for A, so at equal decode speed it spends roughly 2.8x as long generating.

Change the assumptions and the answer can flip. That is the point: s, the number of calls and the reasoning length are properties of your traffic, and you can only get them from your own logs.

Measuring cost and latency per task

Call-level latency metrics

NVIDIA's benchmarking guide gives standard definitions [17]:

• Time to first token (TTFT): time to process the prompt and generate the first token. It generally includes queueing, prefill and network latency, and grows with prompt length.
• End-to-end latency: TTFT plus generation time.
• Inter-token latency (ITL) or time per output token (TPOT): the average gap between output tokens during decoding.
• Tokens per second: per user (output length divided by end-to-end latency) or per system (all output tokens across concurrent requests).

The same guide describes the core trade-off: raising concurrency increases system throughput until the GPU saturates, while per-user speed falls and requests beyond the batch limit queue, which increases TTFT [17].

Public leaderboards help with shortlisting. Artificial Analysis tests endpoints eight times a day with prompts of roughly 100, 1k and 10k tokens and reports the median over 14 days [4]. Those numbers are not your SLO: your prompts, your region and your concurrency differ.

Task-level latency

For a task, measure wall-clock time from the user's request to the validated result. That includes every sequential model call, tool execution, retries and queueing in your own services. Report p50 and p95 per task type. Do not add up per-call p95 values to estimate task p95; percentiles do not add, so measure the task directly.

For reasoning models, also track time to the first visible token. Reasoning tokens are generated before the answer, so a fast TTFT on the wire can still mean a long wait for the user.

If you self-host with vLLM, the engine exposes Prometheus histograms for vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:e2e_request_latency_seconds and vllm:request_queue_time_seconds, plus gauges for vllm:num_requests_waiting and vllm:kv_cache_usage_perc and a vllm:prefix_cache_hits counter [18]. Queue time is the number to watch when p95 TTFT drifts upward.

What to log for every task

Log one record per task, not just per call, with: task type, task ID, model per call, uncached input, cache read and cache write tokens, output and reasoning tokens, number of calls, number of attempts, final success flag, cost, TTFT of the first call and wall time. Do not log prompt text in the same pipeline. Token counts are enough for cost analysis, and keeping user content out of metrics makes GDPR reviews much simpler.

A minimal Python version, with an adapter for Anthropic's usage fields (where input_tokens excludes cache reads and writes, and output_tokens is the inclusive total that includes thinking [7][10]):

import json, statistics, time, uuid from dataclasses import dataclass, field # USD per 1M tokens. Copy from your provider's pricing page; these will go stale. PRICES = { "model-a": {"in": 2.00, "cache_read": 0.10, "cache_write": 2.50, "out": 10.00}, "model-b": {"in": 1.00, "cache_read": 0.05, "cache_write": 1.25, "out": 5.00}, } @dataclass class Call: model: str in_tok: int # uncached input cache_read_tok: int cache_write_tok: int out_tok: int # billed output, reasoning included reasoning_tok: int ttft_s: float total_s: float @classmethod def from_anthropic(cls, model, usage, ttft_s, total_s): details = usage.get("output_tokens_details") or {} return cls(model, usage["input_tokens"], usage.get("cache_read_input_tokens", 0), usage.get("cache_creation_input_tokens", 0), usage["output_tokens"], details.get("thinking_tokens", 0), ttft_s, total_s) def cost(self): p = PRICES[self.model] return (self.in_tok * p["in"] + self.cache_read_tok * p["cache_read"] + self.cache_write_tok * p["cache_write"] + self.out_tok * p["out"]) / 1e6 @dataclass class Task: kind: str id: str = field(default_factory=lambda: uuid.uuid4().hex) calls: list = field(default_factory=list) t0: float = field(default_factory=time.monotonic) def finish(self, success: bool) -> dict: inp = sum(c.in_tok + c.cache_read_tok + c.cache_write_tok for c in self.calls) rec = { "task_id": self.id, "kind": self.kind, "success": success, "calls": len(self.calls), "cost_usd": round(sum(c.cost() for c in self.calls), 6), "out_tok": sum(c.out_tok for c in self.calls), "reasoning_tok": sum(c.reasoning_tok for c in self.calls), "cache_hit_ratio": round(sum(c.cache_read_tok for c in self.calls) / inp, 3) if inp else 0.0, "wall_s": round(time.monotonic() - self.t0, 3), } print(json.dumps(rec)) # ship to your log pipeline; never log prompt text here return rec def summarize(records): done = [r for r in records if r["success"]] wall = sorted(r["wall_s"] for r in records) return { "cost_per_finished_task": sum(r["cost_usd"] for r in records) / max(len(done), 1), "success_rate": len(done) / len(records), "p50_s": statistics.median(wall), "p95_s": statistics.quantiles(wall, n=20)[-1] if len(wall) > 1 else wall[0], } 

Retries of the same task should append to the same Task, so their cost lands on the task that caused them. For other providers, write a second from_... adapter and check in the docs whether the reported input count already includes cached tokens.

The levers, roughly in order of effort

Cut tokens you do not need

Output is the expensive side, so start there.

• Bound reasoning. Anthropic recommends lowering the effort level before trying prompt-based steering, and treats max_tokens as the hard cap on thinking plus answer [10]. OpenAI exposes reasoning.effort with model-dependent values from none to max [11]. Run your eval set at two or three effort levels and pick the lowest one that holds quality. The overthinking results above suggest that easy inputs often do not need the highest setting [14].
• Constrain the format. Structured outputs with a strict schema remove a whole class of retries [15]. Budget for a one-time delay: OpenAI notes that the first request with a new schema incurs extra latency, typically under 10 seconds, after which the processed schema is cached [15].
• Ask for less. Short field values, no restated input, no unread explanations.
• Send fewer tools. Each tool definition is input on every call. Group tools per route so each route has a small, stable set, because changing tool definitions also invalidates the cache [7].

Make prompt caching work

Order prompts as static to dynamic: system instructions, tool definitions, shared reference text, then conversation history, then the new user input [6]. Keep model settings such as thinking effort stable within a conversation [10]. Track the cache hit ratio per task type (cache read tokens divided by total input tokens), and alert when it drops after a deploy, because a small template change can silently move a timestamp to the top of the prompt.

Self-hosted engines offer the same idea. vLLM's automatic prefix caching reuses the KV cache of earlier requests that share a prefix, but it only reduces prefill time, not decode time, so it helps most with long shared contexts and short answers [19]. SGLang uses RadixAttention for KV cache reuse, and its paper reports up to 6.4x higher throughput than other inference systems on workloads such as agent control, few-shot benchmarks and multi-turn chat [20].

Move asynchronous work to batch or flex

Nightly enrichment, document backfills, evaluation runs and report generation rarely need second-level latency. Moving them to batch or flex tiers halves token cost at all three major vendors and on Azure [3][8][2][12]. Build a small queue abstraction so a task can declare "needed within 24 hours" and the platform picks the tier.

Cache whole responses where it is safe

Exact-match response caching (same normalised input, same model, same prompt version) is safe and cheap. Semantic caching, which reuses answers for similar inputs via embeddings, is riskier. The GPTCache documentation itself warns about false positives on cache hits [21]. Use it only where a near-duplicate answer is acceptable, and include the similarity threshold in your evals.

Route and cascade

A router sends easy requests to a small model and hard ones to a large model. A cascade tries the small model first and escalates if a check fails. FrugalGPT reported matching the best individual model with up to 98% cost reduction using cascades and related strategies [22]. RouteLLM reported cost reductions of over 2x in some cases without lowering response quality, evaluated on MT Bench, MMLU and GSM8K [23]. Kapoor et al. cut variable cost by 53% for GPT-3.5 and 41% for Llama-3-70B on HotPotQA by optimising cost and accuracy jointly [16].

Put the cascade into the formula before you ship it. An escalated request pays for the failed small-model attempt and adds its latency, so a cascade only wins if the small model resolves enough traffic on its own.

Distil or fine-tune for narrow tasks

For a stable, high-volume task, a small specialised model can beat a large general one. In "Distilling Step-by-Step", a fine-tuned 770M-parameter T5 model outperformed few-shot prompted 540B PaLM using only 80% of the available data on a benchmark [24]. Distillation is a fixed cost that pays back with volume. In a different setting, Kapoor et al. found that a one-time optimisation cost was outweighed by inference savings after roughly 1,300 inferences [16]; compute the same payback for your own task.

Serving-engine levers if you self-host

If you run open-weight models, the serving engine decides how much work each GPU hour does.

• Continuous batching. Orca introduced iteration-level scheduling, which adds and removes requests from the batch at each decoding step, and reported 36.9x throughput over FasterTransformer at the same latency for a GPT-3 175B model [25]. TensorRT-LLM's in-flight batching applies the same idea, processing context and generation phases together to keep GPUs busy [26].
• Paged KV cache. PagedAttention targets KV cache memory that is wasted by fragmentation and redundant duplication, and vLLM reported 2x to 4x higher throughput than FasterTransformer and Orca at the same latency [27].
• Chunked prefill. vLLM splits long prefills and schedules them alongside decodes. Lower max_num_batched_tokens values (for example 2048) improve inter-token latency, while higher values improve TTFT and throughput [28]. When KV cache memory runs out, vLLM preempts requests and recomputes them later, which hurts end-to-end latency; the docs list remedies such as increasing gpu_memory_utilization or lowering max_num_seqs [28].
• Prefill/decode disaggregation. DistServe places prefill and decode on different GPUs to remove interference between them, and reported 4.48x more requests or 10.2x tighter SLOs under TTFT and TPOT constraints [29]. TensorRT-LLM offers disaggregated serving as a beta feature [26].
• Quantization. vLLM's FP8 docs claim a 2x reduction in model memory and up to 1.6x throughput, with W8A8 officially supported on Hopper and Ada Lovelace GPUs [30]. Red Hat's large evaluation of quantized Llama 3.1 models found full accuracy recovery for FP8 across its tasks, close to 99% average recovery on OpenLLM Leaderboard v2, about 1.8x average speedup for 8-bit models in server workloads, and 2.4x average single-stream speedup for 4-bit weight-only models, which it recommends for latency-critical use [31]. AWQ, a low-bit weight-only method, protects about 1% of salient weights and reports more than 3x speedup over the Hugging Face FP16 implementation [32]. TensorRT-LLM supports FP8 on H100 and later and FP4 weights on B200 [26]. Always rerun your own task eval after quantizing; the Red Hat study saw more variability on 8B models than on 70B and 405B [31].
• Speculative decoding. A small draft model proposes tokens and the large model verifies them. The original method gave 2x to 3x acceleration on T5-XXL with identical outputs and no retraining [33]. EAGLE-3 reports speedup ratios up to 6.5x, but a 1.38x throughput gain in SGLang at batch size 64 [34]. NVIDIA measured 3.55x for Llama 3.3 70B with a 1B draft on one H200, at batch size 1 [35]. The pattern is consistent: speculative decoding is a latency lever that shrinks as batches grow.
• Faster constrained decoding. SGLang's compressed finite state machine decodes several tokens at once when the schema allows only one path, and reported up to 2x lower latency and 2.5x higher throughput for JSON decoding [36].

Self-hosting versus API: the break-even

The economics of self-hosting reduce to one variable most teams underestimate: utilisation.

cost per 1M output tokens = P_gpu_hour / (T * 3600 * u / 1,000,000) break-even utilisation u* = P_gpu_hour / (T * 3600 / 1,000,000 * P_api_per_1M) P_gpu_hour price of the GPU (or node) per hour T sustained output tokens per second at your latency SLO u fraction of hours the GPU actually serves at that rate P_api_per_1M managed API price per 1M output tokens for an equivalent model 

T should be measured at the concurrency where your p95 latency still meets the SLO, not at peak throughput, because the throughput curve flattens while latency keeps rising [17].

Illustrative calculation, with assumptions. Scaleway lists a single H100 PCIe 80 GB instance in Paris at €2.52 per hour, rising to €2.73 per hour from 1 July [37]. For comparison, Lambda lists $4.59 per GPU hour for an HGX H100 effective 16 November 2026 [38]. Assume €2.73 per hour and an assumed sustained throughput of 1,500 output tokens per second for an open-weight model at your SLO (measure this; it varies widely by model, quantization, prompt length and engine).

At 100% utilisation that GPU produces 5.4 million output tokens per hour, or about €0.51 per million. At 30% utilisation the same GPU costs about €1.69 per million. A managed EU option, STACKIT AI Model Serving, lists gpt-oss-120b and Llama 3.3 70B Instruct FP8 at €0.65 per million output tokens [39]. Under these assumptions, u* is about 78%: the GPU must stay busy 78% of all hours just to match the managed price, before you pay for engineering time, on-call, monitoring, model updates and a second GPU for redundancy (which halves utilisation per GPU at the same traffic). Note also that this simple formula charges everything to output tokens; long prompts consume GPU time for prefill as well.

Self-hosting is easiest to justify when traffic is high and steady, the model is small or fine-tuned for one task, or data rules exclude third-party APIs. For spiky or low-volume traffic, run the same formula against a managed endpoint plus batch tiers before buying GPU hours.

EU and GDPR-relevant hosting options

Data residency is not GDPR compliance: you still need a data processing agreement, a lawful basis, retention rules and transfer assessments. Residency narrows where prompts are processed and stored, which simplifies reviews. Documented options include:

• OpenAI API with European data residency. Announced on 5 February 2025: eligible customers create a new Project with Europe selected as the region, requests are handled in-region with zero data retention, and existing Projects cannot be converted [40]. Regional processing carries a 10% uplift for models released on or after 5 March 2026 [2].
• Azure OpenAI Data Zone deployments. Data Zone EU processes data only within the Microsoft-defined zone, which follows the EU Data Boundary and can include EFTA countries such as Norway and Switzerland; data at rest stays in the designated Azure geography; Data Zone Batch gives 50% savings [12].
• Amazon Bedrock geographic cross-region inference. Requests to a geography-bound profile such as EU stay within that geography, but prompts and outputs can move between Regions inside it, and data stored for abuse detection is stored in the destination Region [41].
• Anthropic's first-party API. The pricing page documents inference_geo with a US-only option at 1.1x and global routing as the default [3]. If you need EU-only processing for Claude models, check the cloud-provider routes above and confirm the details with your provider.
• European providers. STACKIT AI Model Serving offers open-weight models in its EU01 region with euro token pricing, and states that data and queries are neither stored nor used to train models [39]. Scaleway rents H100 GPUs in a Paris data centre for self-hosting [37].

Put the residency multiplier into m_region in your cost formula, and keep your own logs residency-aware too: if your cost logger stores prompt text in a US-hosted observability tool, the API region no longer matters much.

Checklist: a recipe for cost per finished task

1. Define "finished" per task type. Write down the validation rule (schema valid, business checks pass, human accepted) that turns an attempt into a success.
2. Log per task, not per call. Use the fields above: tokens by category, reasoning tokens, calls, attempts, success, cost, TTFT and wall time. No prompt text in metrics.
3. Compute the baseline. Cost per finished task, success rate, p50 and p95 wall time, cache hit ratio and output-to-input token ratio for each task type.
4. Bound the loop. Set a task-level token and step budget in the orchestrator, because per-request max_tokens does not bound an agent loop [10].
5. Tune reasoning effort on your eval set. Pick the lowest level that holds quality.
6. Use strict structured outputs wherever the result is parsed by code.
7. Fix prompt layout for caching. Static first, dynamic last, stable tool sets and settings. Alert on cache hit ratio drops.
8. Move async work to batch or flex. Anything that can wait minutes or hours.
9. Add exact-match response caching for repeated inputs; use semantic caching only with measured false-hit rates.
10. Test a router or cascade against the formula, including the cost and latency of escalations.
11. Consider distillation for stable, high-volume narrow tasks, and compute the payback volume.
12. Self-host only after the break-even math, using throughput measured at your SLO and realistic utilisation, including redundancy and people.
13. Choose hosting by data rules first, then add the residency multiplier to the cost model.
14. Re-run the numbers monthly. Prices and models change fast [1]; yesterday's cheapest route may not be today's.

The price per token is an input to your cost model, not the output. Measure the task, and the right model, tier and hosting choice usually becomes clear from your own data.

Sources

All sources accessed 6 November 2026.

1. Epoch AI, LLM inference price trends (data insight). https://epoch.ai/data-insights/llm-inference-price-trends
2. OpenAI, API Pricing. https://developers.openai.com/api/docs/pricing
3. Anthropic, Claude API Pricing. https://platform.claude.com/docs/en/about-claude/pricing
4. Hugging Face blog, "Bringing the Artificial Analysis LLM Performance Leaderboard to Hugging Face". https://huggingface.co/blog/leaderboard-artificial-analysis
5. OpenAI Help Center, "What are tokens and how to count them?". https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
6. OpenAI, Prompt caching guide. https://developers.openai.com/docs/guides/prompt-caching
7. Anthropic, Prompt caching documentation. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
8. Google, Gemini API optimization and inference (Batch, Flex, Priority, caching). https://ai.google.dev/gemini-api/docs/optimization
9. Anthropic, Extended thinking documentation. https://platform.claude.com/docs/build-with-claude/extended-thinking
10. Anthropic, Steering thinking and cost documentation. https://platform.claude.com/docs/en/build-with-claude/thinking-steering-and-cost
11. OpenAI, Reasoning models guide. https://developers.openai.com/api/docs/guides/reasoning
12. Microsoft Learn, Azure OpenAI deployment types. https://learn.microsoft.com/en-us/Azure/ai-foundry/openai/how-to/deployment-types
13. Anthropic Engineering, "How we built our multi-agent research system" (2025). https://www.anthropic.com/engineering/multi-agent-research-system
14. Chen et al., "Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs", arXiv:2412.21187. https://arxiv.org/html/2412.21187v2
15. OpenAI, "Introducing Structured Outputs in the API". https://openai.com/blog/introducing-structured-outputs-in-the-api
16. Kapoor et al., "AI Agents That Matter", arXiv:2407.01502. https://arxiv.org/html/2407.01502v1
17. NVIDIA Technical Blog, "LLM Inference Benchmarking: Fundamental Concepts". Source：https://dev.to/hussnainoffical/llm-inference-cost-in-2026-why-cost-per-finished-task-beats-cost-per-token-22k2
    My favorites