The dashboard said 44% draft acceptance. My first instinct was to rip speculative decoding out of the serving config. Less than half the draft tokens survived, so the draft model was clearly wasting GPU time, right?
Wrong. A 44% speculative decoding acceptance rate on that dashboard meant the draft model agreed with the big model about 70% of the time per token. That number was fine. The thing actually limiting my speedup was something the dashboard didn't show at all: how many requests were sharing the GPU.
This post is the math I wish I'd done before touching the config. It's short, it fits in a Python function, and it explains almost every confusing speculative decoding result you'll see in production.
TL;DR
- Speculative decoding has a small draft model guess
ktokens, then the big target model checks allkin one forward pass. Output quality is unchanged; only speed changes. - With per-token acceptance
α, the expected tokens per target pass is(1 - α^(k+1)) / (1 - α). Atα = 0.7, k = 4that's 2.77 tokens, and after paying for the draft model (10% of target cost per step) the ceiling is about 1.98x. - "Acceptance rate" on dashboards is often accepted / drafted tokens, which is much lower than
α. 44% accepted/drafted atk = 4corresponds toα ≈ 0.7. - More draft tokens is not better. At
α = 0.7,k = 4peaks at 1.98x;k = 8drops to 1.78x. - The speedup comes from idle compute in memory-bound decoding. Large batches use up that idle compute, and speculative decoding can become a net loss.
How does speculative decoding actually work?
Speculative decoding works because a big model verifying 5 tokens costs about the same as generating 1. At small batch sizes, decoding is bound by memory bandwidth. The GPU spends its time streaming weights from HBM, and the math units mostly sit around waiting.
So you let a cheap model run ahead:
- The draft model generates
ktokens autoregressively (cheap, because it's small). - The target model runs one forward pass over all
kpositions plus one extra. - Walk left to right. Accept draft token
xwith probabilitymin(1, p(x) / q(x)), wherepis the target's distribution andqis the draft's. - At the first rejection, resample that position from the leftover distribution
max(0, p - q)(normalized), and throw away everything after it. - If all
kpass, you also get the target's next token for free.
That rejection-sampling step is the clever part. It guarantees the output follows exactly the target model's distribution. Speculative decoding is lossless. If your outputs change in quality after enabling it, you have a bug somewhere else, not a tradeoff.
It also guarantees every target pass produces at least one token. The worst case is roughly normal decoding plus the wasted draft work.
How many tokens does one verification pass produce?
If each draft token is accepted independently with probability α, the expected number of tokens per target forward pass is:
E[tokens] = (1 - α^(k+1)) / (1 - α) This comes from the original speculative decoding analysis (Leviathan et al., 2023). The independence assumption isn't perfectly true, but it's close enough to reason with.
Here's what it gives at k = 4:
| per-token α | tokens per target pass |
|---|---|
| 0.5 | 1.94 |
| 0.6 | 2.31 |
| 0.7 | 2.77 |
| 0.8 | 3.36 |
| 0.9 | 4.10 |
Notice how brutal the curve is. Acceptance is a chain: token 3 only counts if tokens 1 and 2 also survived. At α = 0.7, the chance all four survive is 0.7^4 = 0.24. Most of the time you're throwing away the tail of the draft.
Why does 70% acceptance cap you near 2x?
Because tokens per pass isn't speedup. You also pay for the draft model. If one draft step costs c relative to one target step, the rough speedup is:
speedup ≈ E[tokens] / (k * c + 1) With α = 0.7, k = 4, c = 0.1:
2.77 / (4 * 0.1 + 1) = 2.77 / 1.4 = 1.98x That's the ceiling under ideal memory-bound conditions. Real kernels add overhead for the extra scheduling and the verification bookkeeping, so expect to land below it.
Why c = 0.1 when a 1B draft against a 70B target is a parameter ratio closer to 0.014? Because a tiny model doesn't run 70x faster. Kernel launches, attention over the KV cache, and sampling all have fixed costs that don't shrink with parameter count. Measure c on your own hardware: time one draft decode step and one target decode step at the same batch size.
Why is my speculative decoding acceptance rate so low on the dashboard?
Because the dashboard is probably reporting accepted draft tokens divided by drafted tokens, and that number is mechanically lower than per-token α.
Expected accepted draft tokens per step is α + α² + ... + α^k. At α = 0.7, k = 4:
0.7 + 0.49 + 0.343 + 0.2401 = 1.77 accepted out of 4 drafted = 44% So "44% acceptance" and "70% acceptance" can describe the exact same system. Going the other way, if your dashboard shows 70% accepted/drafted at k = 4, the per-token α is about 0.86, which is excellent and puts you around 3.6 tokens per pass.
This is the metric artifact that nearly made me turn the feature off. Before you compare numbers across blog posts, papers, or your own experiments, check which definition you're looking at. Mean acceptance length (tokens emitted per target pass) is the least ambiguous one. Recent vLLM versions also log per-position acceptance rates, which show the chain decay directly: position 1 high, position 4 much lower.
Here's the converter I keep around:
def tokens_per_pass(alpha: float, k: int) -> float: return (1 - alpha ** (k + 1)) / (1 - alpha) def speedup(alpha: float, k: int, c: float) -> float: return tokens_per_pass(alpha, k) / (k * c + 1) def alpha_from_dashboard(accepted_over_drafted: float, k: int) -> float: """Invert accepted/drafted back to per-token alpha via bisection.""" lo, hi = 0.0, 0.9999 for _ in range(60): mid = (lo + hi) / 2 accepted = mid * (1 - mid ** k) / (1 - mid) if accepted / k < accepted_over_drafted: lo = mid else: hi = mid return lo print(round(alpha_from_dashboard(0.44, 4), 3)) # ~0.70 print(round(speedup(0.70, 4, 0.10), 2)) # 1.98 How many draft tokens should I use?
Fewer than you think. Each extra draft token costs a full draft step, but only pays off if every token before it was accepted. At α = 0.7, c = 0.1:
| k | tokens per pass | speedup ceiling |
|---|---|---|
| 1 | 1.70 | 1.55x |
| 2 | 2.19 | 1.83x |
| 3 | 2.53 | 1.95x |
| 4 | 2.77 | 1.98x |
| 5 | 2.94 | 1.96x |
| 6 | 3.06 | 1.91x |
| 8 | 3.20 | 1.78x |
The peak is flat and early. Going from k = 4 to k = 8 doubles your draft work for a worse result. If your α is lower, say 0.5, the optimum moves down to 2 or 3. Only push k up when α is genuinely high, like code completion or edits that copy large chunks of the prompt.
Why does speculative decoding get slower under heavy load?
Because the speedup is paid for with spare compute, and batching spends that same spare compute.
Rough numbers for an H100 SXM: about 990 TFLOPS of dense BF16 compute and about 3.35 TB/s of HBM bandwidth. That's roughly 295 FLOPs per byte before the GPU becomes compute-bound. A BF16 weight is 2 bytes and costs about 2 FLOPs per token that passes through it, so each token in the batch adds about 1 FLOP per byte of weight traffic. Ignoring attention and KV cache traffic, you need on the order of 300 tokens in flight per forward pass before the math units are actually busy.
At batch size 1, a 70B model reads 140 GB of weights to do 140 GFLOPs of work. The compute units are idle more than 99% of the time. Verifying 5 tokens instead of 1 is basically free.
Now run 60 concurrent sequences with k = 4. Every verification pass pushes 60 × 5 = 300 tokens through the target model. You're at the ridge point. Every rejected draft token is now real compute you paid for and threw away, and the draft model is competing for the same GPU. Past that point, plain continuous batching without speculation can beat you on throughput, and your tail latency gets worse too.
This is why speculative decoding shines for single-user latency (chat, local inference, coding assistants) and gets disappointing on a saturated throughput server.
What should I actually do?
A checklist, in the order I'd do it:
- Find out which acceptance metric you have. Convert accepted/drafted to per-token
α, or use mean acceptance length directly. - Measure
c. Time a draft step versus a target step on your GPU at your typical batch size. Plug both intospeedup(). - Pick
kfrom the table, not from vibes. Forαin the 0.6 to 0.75 range, start at 3 or 4. - Gate speculation on load. vLLM has exposed a disable-by-batch-size option in its speculative config; check the name in your version. Turn speculation off when concurrency crosses the point where your measured speedup falls below 1.
- Match the drafter to the workload. Acceptance drops at higher temperature and on open-ended creative text. For RAG answers and code edits that copy from the prompt, n-gram or prompt-lookup drafting has a draft cost near zero, which changes the math in your favor.
- Don't evaluate quality. It's lossless. Evaluate latency at p50 and p99 under realistic concurrency.
So why does 70% acceptance only get you about 2x?
Because a 70% per-token speculative decoding acceptance rate decays along the draft chain: with 4 draft tokens, you get about 2.77 tokens per target pass, and once you pay roughly 10% of a target step for each draft step, the ceiling is about 1.98x. If your dashboard shows 70% accepted/drafted, your real per-token rate is closer to 0.86 and you have more headroom. Either way, the speedup only exists while decoding is memory-bound, so at high batch sizes the spare compute disappears and speculative decoding can make your server slower. Measure the right acceptance metric, keep k small, and turn speculation off under load.
Written by the developer behind Preterview, an interview prep platform.