NUMALLAMA is a small C inference engine (one source file, no dependencies) that runs modern LLMs on CPUs: no GPU, no CUDA. It is built for big multi-socket NUMA servers, where most engines leave half the machine idle.
GitHub: https://github.com/miloserdev/numallama
The problem
A dual-socket server has hundreds of cores and a lot of memory bandwidth, but each socket has its own memory. Treat the machine as one uniform CPU and every token crosses the slow link between sockets. On our test box, llama.cpp is fastest on one socket and gets slower when it uses both.
NUMALLAMA splits every weight matrix and the KV cache between the sockets. Each socket's cores read only local memory and synchronize mostly within their own socket. As a result, the second socket adds +50–85% decode speed.
Results
Hardware: 2 × Ampere Altra Max M128-30 (256 Neoverse N1 cores, 3.0 GHz), 1 TB RAM, 2 NUMA nodes, ~180 GB/s memory bandwidth per socket.
Decode, tokens/s:
| Model | NUMALLAMA, 2 sockets | llama.cpp, best (1 socket) | llama.cpp, 2 sockets |
|---|---|---|---|
| Llama 3.2 3B Q8 | 76 | 36 | 4.6 |
| Llama 3.2 3B Q4 | 99 | 64 (Q4_0) | 21 |
| Llama 3.1 8B Q8 | 39 | 19 | 3.6 |
| Llama 3.1 8B Q4 | 59 | 29 (Q4_K_M) | 0.8 |
That is 1.5–2× llama.cpp's best result on this machine. Prompt processing is faster too, e.g. 8B Q8: 680 vs 274 tok/s.
Larger models:
| Model | Size | Decode | Prefill |
|---|---|---|---|
| Qwen3-30B-A3B (MoE), Q4 | 19 GB | 91 tok/s | ~500 tok/s |
| Qwen3.8-27B (hybrid linear attention), Q4 | 18 GB | 17 tok/s | 172 tok/s |
Q4 quality: KL divergence 0.013 to full precision on Llama 3.2 3B, against 0.015 for llama.cpp Q4_K_M.
These are measurements on this one machine, not universal claims. All methods and numbers are in BENCHMARKS.md.
What's inside
- NUMA-aware placement of weights and the KV cache; per-socket work queues and buffers
- Persistent thread pool with tree barriers (~1 µs within a socket, ~2 µs across)
- NEON dot-product kernels on repacked weights, batched prefill, multi-token decode step
- Q8 and asymmetric Q4 with an importance matrix collected by the engine itself
- Exact speculative decoding: prompt lookup or a small draft model
- Mixture of experts (Qwen3-MoE) and hybrid Gated DeltaNet models (Qwen3.5 / 3.8)
- OpenAI-compatible server: streaming, tool calls, prompt-prefix cache
- CLI chat with sandboxed tool calling
Supported models: Llama 3.x, Qwen2.5, Qwen3 / Qwen3-MoE, Qwen3.5 / Qwen3.8.
Quick start
make python3 scripts/export_hf_q8.py hf/Qwen3-30B-A3B-Instruct-2507 qwen3_30b_q80.bin python3 scripts/export_tokenizer.py hf/Qwen3-30B-A3B-Instruct-2507 qwen3_tokenizer.bin ./runq qwen3_30b_q80.bin -z qwen3_tokenizer.bin -m server -P 8080 Any OpenAI client then works with base_url="http://127.0.0.1:8080/v1".
Limitations
- Platform: the optimized kernels target ARM64 with NEON + dot product (Ampere Altra, AWS Graviton, other Neoverse cores). x86-64 builds run on portable C fallbacks and are not optimized yet.
- Linux only, no GPU offload.
- The server handles one request at a time; other requests wait in a queue. There is no multi-user batching yet.
- Text only, and a short list of model families.
Why CPU?
GPUs win on speed per dollar when the model fits in their memory. CPU servers win on capacity: a terabyte of RAM holds models that need several GPUs, and many data centers already have idle ARM servers. NUMALLAMA is an attempt to see how far careful engineering around memory locality can push them.

