Why your local LLM tok/s numbers don't match mine

A few weeks ago I needed to pick a local inference server for a project. I'm on a Mac, so the candidates were the usual ones: Ollama, LM Studio, a couple of MLX servers. I did what everyone does and went looking for benchmarks.

I found plenty. I just couldn't reproduce any of them.

One post said server A was twice as fast as server B. On my machine they were almost the same. Another one gave a tok/s number for a model I had downloaded, and I was nowhere near it, in either direction. After a while I stopped reading and started measuring, and that turned out to be a lot more interesting than I expected.

It's not that the benchmarks are wrong

Most of those numbers were probably correct on the machine where they were taken. The problem is that a tok/s figure depends on so many things that it doesn't really travel.

The hardware is the obvious one. Memory bandwidth decides most of the speed, and "an Apple Silicon Mac" covers machines that are two times apart. Then there's the model file: "LFM3 1.7B" in MLX 8-bit, in a GGUF Q4_K_M and in Ollama's mxfp8 are three different things that happen to share a name.

The one that surprised me most was who holds the stopwatch. A lot of benchmarks use the numbers the server reports about itself, and every server counts in its own way. Some leave out the first token, some leave out prompt processing, some only tell you the generation speed. Put two of those next to each other and you're comparing different quantities.

Then reasoning models came along and made it worse. The same prompt can give you 30 tokens or 500 depending on whether the model thinks first, and servers don't agree on the default. I found out the hard way that Rapid-MLX turns thinking off unless you explicitly ask for it, while others leave it on. For a while I was comparing a model that reasoned with one that didn't, and wondering why the numbers looked strange. The Rapid-MLX maintainer explained it to me in an issue, which was very kind of him.

And finally, noise. On my laptop the exact same test moves 10 to 20% from one minute to the next. Background apps, temperature, whatever. A 5% difference from a single run means nothing.

What I ended up doing

The fix is boring: hold the stopwatch yourself, and use the same one for everything.

I send the same streaming request to each server and time it on the client with time.perf_counter(), ignoring whatever the server says about itself. What I care about is what my app will actually feel. I look at time to first token, generation speed, the overall effective speed (output tokens divided by total time, prompt processing included) and how much RAM the system is using.

The first run loads the model and warms everything up, so I throw it away. Then I take the median of at least three runs and keep the min and max around, because the spread tells you how much to trust the result. Reasoning gets set explicitly on every request, on or off, never left to the default. And each prompt starts with a unique ID so prompt caching doesn't quietly make the second run look amazing.

What it looked like on my Mac

Here's one example, just to show the kind of thing you get. M5 Pro with 24 GB, LFM3 1.7B at 8-bit, short prompt, 512 output tokens, reasoning off, median of 3 warm runs:

I'd read this as "they're all close", not as a ranking. The first two are a tie given the noise, and the order even changes with the prompt language, which I didn't expect. With a different model, or on your machine, the picture may look completely different. That's kind of the point.

If you want to try it

I turned all of this into a small open-source app called TokenRace. If you have uv installed, this is all you need:

uvx tokenrace 

You pick a server and a model, start the test and watch the numbers come in live. At the end there's a summary, a chart and a CSV you can keep. It all runs locally, nothing gets sent anywhere.

A single session from the app; numbers vary between runs, which is part of the point.

It already knows about 15 servers: Ollama, LM Studio, llama.cpp, vLLM, SGLang, Rapid-MLX, oMLX, mlx_lm.server, LocalAI, Jan, KoboldCpp, text-generation-webui, Lemonade, Docker Model Runner and Unsloth Studio. If yours isn't there but speaks the OpenAI API, you can just change the URL, or add it with a few lines in config/servers.toml.

I've only been able to test a few of these myself, on one Mac. So if you run it on an NVIDIA or AMD box, or on Linux, I'd really like to see your numbers. There's a Share your results section for that. Corrections on ports or reasoning parameters, and PRs for servers I've missed, are just as welcome.