When teams evaluate how a large language model (LLM) will perform in production, two families of metrics come up repeatedly. One family describes latency: how quickly an individual request is served. The other describes throughput: how much total work the system can accomplish in a given period. Throughput is what determines whether a deployment can serve ten users or ten thousand, and it is the metric most directly tied to infrastructure cost per request. Yet it is frequently misunderstood, partly because the headline figure, "tokens per second," can mean several different things depending on whether you are counting one stream or the whole system.
This article explains what LLM throughput actually measures, why it depends so heavily on concurrency and batching, and how it trades off against latency. The goal is a clear and accurate mental model, grounded in how inference hardware behaves.
Defining throughput
Throughput is the rate at which a system completes work. For LLM inference, "work" is almost always measured in tokens, the sub-word units the model reads and generates. The most common throughput metric is tokens per second, but it is critical to distinguish two senses of it:
- Per-stream throughput: the rate at which a single request's output is generated. If one user's response arrives at 40 tokens per second, that is their per-stream rate. This is essentially the inverse of per-token latency and reflects how fast one answer streams.
- System (aggregate) throughput: the total tokens per second the entire server produces across all concurrent requests combined. A server might deliver 40 tokens per second to each of 50 users simultaneously, for an aggregate of 2,000 tokens per second.
These two numbers can move in opposite directions, which is the single most important and counterintuitive fact about LLM serving. Pushing more requests through the system at once usually raises aggregate throughput while lowering each individual stream's rate. Confusing the two leads to bad capacity decisions, so it is always worth asking which one a given "tokens per second" figure refers to.
Throughput is also sometimes expressed in requests per second, which is useful when requests have similar shapes, but tokens per second is more robust because LLM requests vary enormously in input and output length. A request that generates 10 tokens and one that generates 1,000 tokens are very different amounts of work even though each counts as a single request.
Why batching is the engine of throughput
To understand why concurrency raises aggregate throughput, you have to look at what the hardware is doing during generation. Modern accelerators are massively parallel, with enormous arithmetic capacity. During the token-by-token decoding phase of generation, however, the dominant cost is not arithmetic. It is reading the model's weights from memory. For every single token a model generates, it must stream its full set of parameters (or the relevant portions) out of memory and through the compute units.
If the server generates a token for just one request, it reads all those weights to produce a single token. The expensive memory read is "amortized" over one unit of useful work, and most of the chip's arithmetic capacity sits idle. This is deeply inefficient.
Batching fixes this. By grouping many requests together and generating their next tokens in one pass, the server reads the model weights once and uses them to produce tokens for all requests in the batch. The costly memory read is shared across many units of work. As a result, aggregate throughput climbs almost linearly with batch size, up to the point where the hardware's arithmetic capacity or memory for the KV cache becomes the bottleneck.
Concurrency and its limits
Concurrency is the number of requests a system is actively processing at the same time. Higher concurrency enables larger batches, which drives aggregate throughput up. But concurrency is not free, and it cannot grow without bound. Two resources constrain it.
The first is memory for the KV cache. Each in-flight request maintains a cache of key and value tensors representing the context it has processed so far. This cache grows with the sequence length and consumes accelerator memory alongside the model weights. The more concurrent requests, and the longer their contexts, the more memory the KV cache demands. When memory runs out, the system can no longer admit new requests, must queue them, or in some designs must evict and recompute context, all of which hurt performance.
The second is compute saturation. As batch size grows, the decoding step eventually stops being purely memory-bound and starts being limited by arithmetic throughput. Beyond that point, adding more concurrent requests no longer increases aggregate throughput; it only increases per-request latency, because each step now takes longer. The system has reached its efficient operating ceiling.
The art of serving LLMs efficiently is largely about operating near the "knee" of this curve: enough concurrency to keep the hardware well-utilized and aggregate throughput high, but not so much that latency becomes unacceptable or memory overflows.
Continuous batching
Early batching approaches were "static": the server collected a batch, ran it to completion, and only then started the next batch. This is wasteful because requests in a batch finish at different times. A request generating a short answer finishes early but its slot stays occupied, idle, until the longest request in the batch completes.
Continuous batching (also called in-flight or dynamic batching) solves this. Instead of waiting for the whole batch to finish, the scheduler works at the granularity of individual decode steps. As soon as any request completes and frees its slot, a waiting request is admitted into the batch immediately. The batch is continuously topped up, keeping utilization high and reducing the time new requests spend queuing. Continuous batching is one of the most significant throughput improvements in modern inference servers, often increasing effective throughput several-fold over static batching for mixed workloads.
Estimating throughput and concurrency
A simple back-of-the-envelope relationship connects these quantities. If you know your target per-stream rate and your aggregate capacity, you can estimate how many users you can serve concurrently:
Throughput and concurrency relationships
aggregate_tps = 2000 # system tokens/sec at the chosen batch size
per_stream_tps = 40 # tokens/sec each active request should receive
How many streams can run at the target per-stream rate?
max_concurrency = aggregate_tps / per_stream_tps
= 2000 / 40
= 50 concurrent streams
Throughput for a workload, given average output length and request rate
avg_output_tokens = 300
requests_per_sec = aggregate_tps / avg_output_tokens
= 2000 / 300
~= 6.7 requests/sec sustained
These are approximations, not guarantees. Real aggregate throughput depends on prompt lengths (long prompts spend more time in the compute-bound prefill phase), output lengths, how uniform the requests are, and memory headroom. But the structure of the calculation is sound and useful for capacity planning: aggregate throughput divided by the per-stream rate you are willing to deliver gives you a rough concurrency ceiling.
The throughput-latency trade-off
The central tension in LLM serving is that throughput and latency pull against each other. Driving the system toward maximum aggregate throughput means running large batches at high concurrency, which lengthens each individual request's time per token and can increase queuing delay before a request even starts. Conversely, optimizing for the lowest possible latency for each user means running small batches, which leaves hardware underutilized and raises the cost per token.
There is no universally correct setting. A batch inference job that processes a million documents overnight should be tuned for maximum throughput, since no human is waiting on any individual result. An interactive assistant should cap batch sizes and concurrency to protect per-user responsiveness, accepting a higher cost per token. Most production systems expose configuration, such as maximum batch size, maximum concurrency, and queue limits, precisely so operators can position the system along this trade-off curve for their use case.
Where throughput is managed
In a deployment serving multiple applications or models, throughput and concurrency are managed at the layer where traffic is proxied, at a gateway that sits in front of the model servers. Concentrating traffic management there allows concurrency limits, queuing, and load balancing across model replicas to be enforced consistently, so that one bursty client cannot starve others and aggregate capacity is used efficiently. This separation of concerns, with the model server focused on efficient batched generation and the routing layer focused on distributing and limiting traffic, is a common and effective pattern.
Key takeaways
- Throughput is the rate of work, usually in tokens per second. Always distinguish per-stream throughput (one request's rate) from aggregate system throughput (all requests combined); they often move in opposite directions.
- Batching is the engine of throughput. Because decoding is memory-bandwidth-bound, generating tokens for many requests in one pass amortizes the cost of reading model weights and dramatically raises aggregate throughput.
- Concurrency is bounded by KV cache memory and by compute saturation. Efficient serving means operating near the knee of the curve, high utilization without unacceptable latency.
- Continuous batching admits new requests into a running batch as slots free up, keeping utilization high and cutting queuing delay.
- Throughput and latency trade off. Tune toward throughput for batch jobs and toward low latency for interactive use; there is no single correct operating point.