357 seconds.
That's how long the median request waited for its first token when we ran GLM-5.3 on vLLM under sustained load. Eight B200s. 32 concurrent workflows.
Our engine, same model, same box: 3.24 seconds.
I didn't build this. Rob and the team did. They've been cooking on it for a long time, mostly quietly. This week we started telling people. Four reports on the blog, three models, three kinds of GPU.
Quick context. This is not a vLLM fork. It's not a quantized model. We wrote our own inference engine. It runs the full model and gives you the full answer.
The scorecard
| Model | Hardware | Up against | Result |
|---|---|---|---|
| GLM-5.3 (BF16) | 8× B200 | vLLM 0.32 | 4.94× sustained output throughput |
| Qwen3.6-35B-A3B (BF16) | 1× RTX PRO 6000 Blackwell | vLLM 0.33 | Up to 1.95× throughput, up to 5.75× faster median TTFT |
| Qwen3.8-27B (BF16) | 1× RTX PRO 6000 Blackwell | vLLM 0.33 and SGLang | 104/104 metrics vs vLLM, 102/104 vs SGLang |
| Qwen3.8-Flash-Next (BF16) | 8× H100 | SGLang (FP8) | Up to 12.8× p90 interactivity on AgentX |
GLM-5.3 on 8× B200
30-minute window. Target of 32 workflows in flight. Both engines got the same first 116 workflows.
- Output throughput: 370.0 vs 74.9 tok/s (4.94×)
- Workflows admitted: 462 vs 116
- Median TTFT: 3.24 s vs 357.5 s
- Per-request generation: 15.8 vs 7.7 tok/s (2.05×) We cleared work faster, so we kept taking more. vLLM never got past the first batch.
Burst test, same 96 workflows dropped at once: 2.61× throughput, 10× faster median TTFT. We finished 95 of 96. vLLM finished 82.
Qwen3.6-35B on one GPU
One RTX PRO 6000 Blackwell on Google Cloud. Both engines at BF16. Both with 3 MTP draft tokens. Same 32K context, greedy decoding.
We tried a bunch of vLLM settings and landed on 95% GPU memory utilization. If you're going to beat a baseline, beat a tuned one.
| Concurrency | 8 | 16 | 32 | 64 | 128 | 256 |
|---|---|---|---|---|---|---|
| Throughput vs vLLM | 1.63× | 1.57× | 1.62× | 1.65× | 1.87× | 1.95× |
Over 1.5× at every level. From 8 concurrent requests to 256.
Median TTFT hit 5.75× faster under steady traffic and 8.56× under burst. Prompts were a mix of chat, long-doc QA, math, code, and tool use (Dolly, LongBench, MATH, HumanEval, Glaive and a few others).
Qwen3.8-27B vs vLLM and SGLang
Smaller model, less headroom. This is where engines start to look alike. So we brought SGLang in too.
8 workload points × 13 metrics = 104 comparisons per engine.
- vs vLLM 0.33: 104 of 104
- vs SGLang: 102 of 104 Throughput was up to 44% over vLLM and up to 26% over SGLang. Every mean, p50, p90 and p99 end-to-end latency went our way against both.
At 1K input and 64 concurrent requests, median TTFT was 299 ms for us. vLLM took 13.3 seconds.
Somebody will ask, so: we use a proprietary 4-bit draft head here. The BF16 model checks every token it drafts. You get the full model's output.
The agent test
This is the one I'm most interested in.
SemiAnalysis AgentX simulates coding agents. Long multi-turn sessions, shared context, subagents, pauses between turns. Average input in our run was 87K to 106K tokens per request. That's what our customers' traffic looks like. Chat benchmarks don't come close.
Qwen3.8-Flash-Next on 8× H100:
- p90 interactivity, 1 session: 209.0 vs 16.3 tok/s per user (12.8×)
- 12 sessions: 7.0×. 16 sessions: 5.0×
- p90 TTFT, 1 session: 0.78 s vs 10.6 s And we did it at full precision. We ran BF16. The SGLang reference ran FP8. They took the shortcut and we still won.
The SGLang bug
Side story, because it's too good to leave out.
One of our engineers spent a long stretch trying to figure out why SGLang's decode speed beat ours. He tore our engine apart looking for it.
It wasn't us. SGLang was applying simulated acceptance in speculative decoding the wrong way. In his testing that pushed their tokens per second up by around 8%. Their outputs were fine. Their speed numbers weren't.
He was mad enough to start drafting a post about it. Then he found the issue already had a merged fix ([PR link]). And yesterday AgentX republished its SGLang numbers. They came down about 3%.
Our AgentX comparison above used the SGLang reference from before that update. The correction moves their numbers down. Not ours.
SGLang is shipping hard right now, including on DeepSeek V4.1 on B200. Good. We want the competition sharp.
Why we built for throughput
We sell into people who own GPUs. They can't get more of them. What they can get is more work out of the ones they have.
So throughput came first. The latency gains showed up along with it.
And no shortcuts on the model. Full weights. No quantized serving model. No relaxed token acceptance.
Read the full reports
Methodology, configs and every number are in the posts.
- GLM-5.3: 4.94× sustained throughput vs vLLM
- Qwen3.6: up to 95% higher throughput than vLLM
- Qwen3.8-27B: outperforming SGLang and vLLM
- Qwen3.8-Flash-Next: up to 12.8× agent interactivity vs SGLang More models and more hardware are coming.
Think something's off in the methodology? The comments are open. That's what they're for.
