Ten engines for one box: which local inference engine fits a 100 GB rig with two 16 GB cards?

This blog has published notes on ten inference engines — llama.cpp, Ollama, vLLM, Sonar (formerly Aphrodite Engine), exo, ExLlamaV3 + TabbyAPI, Strata, PolyStrata, Avalon and Xolotl. This is the decision piece: which of them I would run on the machine I keep being asked about — roughly 100 GB of system RAM and two 16 GB consumer GPUs.

Every figure below I have checked; none is estimated or invented. All are published numbers collected on our benchmark page, https://localai.hex7.com/benchmark.html, which marks every one with a source and dashes where none was found. Self-reported figures are labelled as such, third-party measurements likewise, and where no comparison can be made I say so — that is part of the conclusion, not a hedge.

The engines, and what each one descends from

Five of these projects descend from one another; the lineage matters before any comparison.

• llama.cpp (ggml-org): the general runtime on ggml — GGUF weights, an OpenAI-compatible server (llama serve), offload flags like --n-gpu-layers. Strata's and PolyStrata's head-to-heads benchmark against it first.
• Ollama: wraps that engine — inventivehq describes "the same weights through the Ollama wrapper" — with an MTP draft enabled by default.
• vLLM: contributes the high-concurrency design Sonar inherits: continuous batching, paged KV-cache management.
• Sonar (formerly Aphrodite Engine) is not a successor but the same repository renamed in place: commit "feat!: rename Aphrodite Engine to Sonar", 7 November 2026; PygmalionAI/aphrodite-engine redirects to dphnAI/sonar. Its README: "Sonar is based on vLLM."
• exo (exo-explore): a peer-to-peer cluster runtime on MLX — one model across several machines rather than inside one.
• ExLlamaV3 + TabbyAPI: the pair as it ships today. ExLlamaV3 is MIT-licensed, at v1.6.6 (31 Oct 2026); TabbyAPI is the OpenAI-compatible server that fronts it and has carried only its back-end since 14 July 2026.
• Strata (Niko1221): uses llama.cpp's raw materials — GGUF files, quantisation formats, the ggml kernel lineage — but is an overfit engine built on those formats, not a branch of its source.
• PolyStrata (simplexai-labs): a fork of Strata running two engines: keeps "the Strata project's, unchanged" for Qwen3.8-Flash-Next, adds its own backends for roughly eleven more models.
• Avalon (freriadoc): a fork of llama.cpp (base merged to 31 October 2026) tuned for one model whose experts live in system RAM it owns.
• Xolotl (Indras-Mirror): a descendant of Strata — its README says "Built from Strata"; the parent's engine merged at 0.1.44.1, over 70,000 lines added on top.

The capabilities table

Before any speed number: what each engine can hold, and where it may put it.

The third column is where this hardware decides: whether an engine owns its weight placement or trusts somebody else's page cache. Sonar and TabbyAPI's AGPL-3.0 also rules them out of stacks that cannot ship AGPL; eight of these licences were checked against LICENSE files in my source notes, and the three general-runtime ones are as their repositories state.

The machine at issue — and what cannot be compared

My rig: about 100 GB of system RAM with two 16 GB consumer GPUs. Up front, honestly: no published benchmark row exists for this exact machine. What exists are runs on boxes adjacent to mine; an argument has to be built from those, and I read them against my budget below. All sources are listed at the end.

What runs on a 2× 16 GB + 100 GB rig — the closest published runs

Read these rows against my budget, not against each other. Closest to mine: Strata's 2 × TITAN RTX community run — a generation above my cards, but a consumer two-card layout with expert caching that works; Avalon's A4000 pair is the only two-card pipeline test published at professional-card scale, though its result is prompt reading, not decode; Xolotl's measured machine is my RAM class plus one card — its README states an 80 GB expert arena on a 110 GB machine, 64 GB on a 90 GB one, so my rig sits between two documented cases, not outside them; PolyStrata goes down to an 8 GB laptop card and advertises multi-GPU on two or three cards, though honestly its largest model (GLM 5.3 Flash, 79–114 GB) "needs a 32 GB card to run at all" — on my box that means the class it demonstrated at laptop scale.

The argument for PolyStrata and Xolotl on this box

The case is not "these engines are faster" as a universal claim — no published head-to-head of PolyStrata against Xolotl exists on any machine, so that comparison cannot be made. The case is narrower: on exactly this budget — one box, two consumer cards, ~100 GB RAM — the published runs and documented memory designs are the ones that fit.

The fix is what separates them on my hardware; three projects discovered the same problem independently. Avalon's README states it: the OS page cache is shared, so any other program pushes experts out mid-token; and the driver limits registered GPU memory to RAM − RAM/64 − 1.5 GiB, so a long prompt on a small machine ends in a crash. Its fix — a pre-registered, LRU-managed O_DIRECT arena inside the engine's own RAM — turns that crash into ~49 tok/s on 16 GB, where the page-cache path reached 13–17 and then crashed. Xolotl states the same tiering as an explicit budget: hot experts in VRAM, tens of gigabytes pinned in RAM read by DMA, the rest on NVMe moved a router layer ahead. PolyStrata landed in the same place: since 0.1.45-poly.7 it stopped holding file parts already on the card (Qwen3.6 after a 32K prompt: 17.9 GiB held → 13.1 GiB, same speed), and its knobs carry numbers — hold 6,144 MiB instead of everything and peak during a 32K prompt drops from 14.0 to 7.3 GiB for a 3–14% write cost; budget the hot-expert set to free RAM minus 4 GiB and first-chat decode rises from 51 to 57–60, after a 24K prompt from 75–78 to 77–85. The lesson all three teach: an engine must own its memory, or the machine takes it back mid-token.

The table above shows this pattern landing on boxes like mine — Strata's two-card row, Xolotl's Gauntlet rows, PolyStrata at laptop scale, the two-AMD-cards prompt test. Where unowned paging falls out is just as published: the tuned llama.cpp row reaches 64.39 but loses prompt speed doing it, and its community row at 8.6–10.6 tok/s shows what a 136.7 GB file costs at my model class. The starkest pair on the benchmark page is Strata's head-to-head on a 12 GB RTX 3060 with 46 GB RAM: llama.cpp tuned with -ngl 99 -ncmoe manages 9.1–9.8 decode and 294–338 prompt; Strata, in the same run on the same file, publishes 41.6–41.9 and 910–1,050 — published by the Strata team, though it names llama.cpp's numbers.

So for this machine I would run PolyStrata (its knobs match my RAM budget, its multi-card support exists, and its smallest published runs need less than I have) or Xolotl (measured on my RAM class with one card more; its tiering documentation states my RAM case). Both descend from Strata — PolyStrata is a fork of it running two engines; Xolotl says "Built from Strata" — and their caveats are honest: Xolotl is research code from one developer, Linux-first (its Windows path exists only for the DeepSeek model); PolyStrata marks its own extra backends experimental. Neither claim is universal; on other hardware the answer flips.

Where the other engines are the right answer instead

High-concurrency serving on data-centre cards: vLLM and Sonar

If the box has H-class memory and many concurrent users, these are the answer. The measured H100 row (by the hosting company) puts vLLM at 120 tok/s at one request, 650 at ten, 1,850 at fifty, 2,400 at one hundred, first token 380 ms at fifty; on eight GB200 vLLM reports 26.2K prefill and 10.1K decode TPGS (their metric: tokens per GPU per second), and 7,360 tokens/GPU/s single-GPU prefill on a GB300/B300. Sonar's tuned four-RTX-PRO-6000 run (self-reported) reaches aggregate decode 428.3 → 1,083.3 tok/s at C=8 with DSpark speculation over a 2,674-token prompt — and the only published Sonar-vs-vLLM row on an H100 is modelled estimates by the publisher (48 / 380 / 1,450), not measurements of Sonar. On two consumer cards its docs advise care: no NVLink domain, 92% of the card "by design"; nothing in its pages tiers weights into system RAM, so my 100 GB buys Sonar nothing.

An OpenAI-compatible box that works out of the installation command: Ollama

The simplest case — one command installed, clients pointed at an OpenAI-style endpoint — is what Ollama is for; it inherits llama.cpp's offload levers. The rows show where it stands under the wrapper: same weights on a rented 5090, llama.cpp generated 249 tok/s with 83 ms to first token, Ollama 108 with 221 ms; on older desktop and M3 Max rows llama.cpp led again, 77.0 vs 69.1 and 53.5 vs 46.2. The one case it beats bare llama.cpp is its default MTP draft: 39.9 → 76.2 (L40S: 57.3; RTX 5090: 129.7). Use it when the wrapper, not the benchmark, is what you need.

VRAM-resident EXL3 weights for single-card fast generation: ExLlamaV3 + TabbyAPI

The project's earlier weights format has a run table published by its own maintainers (measured only against their first release — there is no comparison with other engines) and those numbers are the fastest thing on record for this family: 211 tok/s on Llama-2-7B at 4.0 bpw, 50 on CodeLlama-34B at 4.0 bpw, 38 on Llama-2-70B at 2.5 bpw, on the named 3090 Ti and 4090. ExLlamaV3 publishes no equivalent table, so its speed case rests on those earlier measurements; what V3 does document is CPU offloading for MoE experts, which changes what a two-card box can hold. For one card where the weights fit, hard to beat — but no EXL3-versus-GGUF comparison is published anywhere I could find.

A cluster of separate machines rather than one box: exo

If the memory ceiling need not be one machine's, exo is the answer: four M3 Ultra Mac Studios with 1.5 TB combined, measured by a journalist (Geerling), ran Qwen3-235B at about 32 tok/s and Kimi K2 Thinking around 30, with RDMA cutting memory-access latency from ~300 µs to under 50 — while llama.cpp's RPC layer-spreading got slower as nodes were added (no number published). On my rig exo is not a deployment today: Linux is CPU-only, and Mac nodes refuse to join Linux nodes.

The general runtime, and the older siblings

llama.cpp stays right when "whichever model shipped this week" matters, or when tuned flags make a big MoE run at all — 64.39 tok/s with -cmoe on two 4090s is llama.cpp. Strata and Avalon remain right for single-card desktops, but scope themselves to one model family; PolyStrata and Xolotl are what their authors built when they wanted more.

Where no comparison exists

For honesty's sake: I found no published figure for Sonar on any consumer two-card box; none for exo on NVIDIA hardware under Linux; no head-to-head between PolyStrata and Xolotl on any machine; no EXL3-versus-GGUF test in either direction; and no number for llama.cpp's RPC spread, only a report that it got slower. A benchmark page full of dashes is data too.

How I verified this

The figures above were gathered while compiling https://localai.hex7.com/benchmark.html, which marks each number with a source and states which machine it ran on. The sources collected there: Strata's README, its bench/results/2026-10-06-llamacpp-0.6.0 head-to-head and community-benchmarks files; PolyStrata's README, its 2026-10-11-laptop-qwen36-llamacpp-b11540 head-to-head and COMMUNITY_BENCHMARKS file; Avalon's README tables (its runs against Strata v0.2.9 on the same boxes); Xolotl's README Gauntlet v3 table and speed log plus WILL_IT_RUN.md; the run table the ExLlama maintainers published for their earlier weights format; computingforgeeks' three-way test on rented cards; atomic.chat's Ollama-vs-llama.cpp on an RTX 5090; inventivehq's rows; developersdigest's write-up of llama.cpp PRs #29887 and #30112; Jeff Geerling's four-Mac-Studio exo test; dphnAI's Sonar measurement on four RTX PRO 6000; Spheron's measured vLLM H100 rows and its explicitly-modelled Aphrodite estimates; vLLM's own GB200/GB300 blog posts. This blog's engine notes record each project's own claims in detail. The one thing I could not verify is the machine at the centre of this article: two 16 GB cards with ~100 GB of RAM never ran in any published test. Mine will, and when it does I will publish those rows under the same rules as everyone else's.