Avalon – what it is, and the three ways it differs from llama.cpp and Strata

Three engines now wear the same badge — "runs GGUF models on your own machine" — and they are not the same tool. llama.cpp is a general-purpose runtime. Strata is a desktop application that serves a short list of models well. Avalon (freriadoc/Avalon, MIT) is something different again: a fork of llama.cpp tuned for exactly one model, Qwen3.8-Flash-Next, on one consumer GPU with the experts living in system RAM. Its README states the intent plainly: "It is not tuned on a flagship machine: every number below comes from a desktop most people would call old."

That sentence is the whole design brief, and once you read it that way the rest of the project makes sense.

What Avalon actually is

Avalon is a llama.cpp fork — base: llama.cpp master merged up to 31 October 2026 — that started life as a branch of buun-llama-cpp, which is where the GPU expert cache and its LFU admission policy come from. It borrows the IQ3_K and IQ3_KS quantisation types and the shared MTP head support from ik_llama.cpp, takes the expert-ranking format and preload idea from Strata, and ships a ~89 GiB model in 33 files plus an MTP draft head.

The target model is Qwen3.8-Flash-Next: 512 experts per layer, 10 active, 48 layers. The GGUF files Avalon uses are built from quantised types and a packed layout that stock llama.cpp cannot load — so the README is candid that other models run "as in llama.cpp, without the expert cache work". It is one model, optimised within an inch of its life; everything else is compatibility, not tuning.

The difference is where the experts live

A mixture-of-experts model keeps ~49 GiB of expert weights that are mostly not needed by any single token. llama.cpp maps the model file and lets the operating system's page cache deal with that. Avalon's README explains exactly why that fails:

• the page cache is shared, so any other program — a browser, a compile — pushes experts out, and they return from disk at the next page fault in the middle of a token;
• the GPU driver limits memory it will register for all processes to RAM − RAM/64 − 1.5 GiB, so on a 16 or 32 GB machine most experts simply cannot be registered, and a long prompt ends in a crash.

Avalon instead keeps experts in RAM it owns. If everything fits (64 GB), they are read once with O_DIRECT into its own address space, registered in place, and nothing else can evict them — the server is at full speed ~20 seconds after a cold start. If they do not fit, it builds an arena of pre-registered slots sized to what RAM allows (~23 GiB on a 32 GB machine, ~9 GiB on 16), seeds it at startup from a shipped ranking file, and keeps it LRU-managed, reading missing experts straight off the disk with O_DIRECT. That is a completely different contract from "hope the page cache is warm", and it is why an 89 GiB model runs at ~49 tokens/s on 16 GB of RAM.

The second part of the design is tiering: GPU expert cache takes whatever VRAM nothing else is using — compute buffers are lent to it between phases, the unused tail of the KV cache is given to it, and a calculated 400 MiB stays outside; hosted MoE runs the cached experts on the GPU while the CPU computes the missing ones and moves one or two across the PCIe link when that finishes sooner; during prefill, expert prefetch runs on four slots borrowed from the cache. The stated goal is blunt: "RAM, VRAM, CPU, PCIe link and GPU should all be busy, all the time, at their limit."

Strata does the same job differently: it is an application with installers and a launcher (strata launch, strata connect), keeps per-expert VRAM caches across all layers, reads what doesn't fit from SSD, and serves a short list of model families across a wide set of hardware. Avalon has no app, no installer philosophy — you build it with CMake or run its ROCm installer script, then avalon-server gives you an OpenAI-compatible API on 127.0.0.1:8080 and --fit decides placement for you. Strata is a product; Avalon is a tuned engine you assemble yourself.

The numbers, with the caveats attached

Avalon's own machine is deliberately old: Ryzen 7 3800XT (Zen 2, 2019), DDR4-3200 dual channel measured at 33–35 GB/s, an RX 7900 XT 20 GB on PCIe 3.0 — no CUDA, no flagship. With MTP speculative decoding and two draft tokens it measures:

The full 256k context fits, with the MTP draft model and a q8_0/q4_0 KV cache plus the expert cache, under 19.7 GiB of VRAM.

Against Strata v0.2.9 on the same card, same protocol, both with tail-skip on, Avalon reports 98.4 vs 68.4 t/s on a short prompt (44% faster) and 81.1 vs 62.6 at 250k filled (30%), with prefill 1,854 vs 1,453 t/s. On NVIDIA the picture changes: rented an RTX 4090, it is effectively even at 141.9 vs 141.4 t/s with MTP, and Strata's prefill is faster there (3,422 vs 3,113) because Avalon's kernels are still tuned for RDNA3. That is a fair thing to say out loud, and the README does.

The low-RAM numbers are the most useful part of the document because they show what the RAM arena buys: on 32 GB, 85–88 t/s at short prompt falling to 40–49 after a 19k prompt; on 16 GB, ~49 t/s and 30–38 after a long prompt — while the old page-cache path crashed at the 19k prompt and reached only 13–17 t/s. The README also corrects its own earlier claims (the previous 90/89 t/s figures were measured under a memory cgroup only, with the driver's registration limit still that of a 64 GB machine) — an unusually honest piece of self-reporting.

Quality is where the argument "speed comes from a smaller model" dies. Perplexity on external text (English / French / code, lower is better):

The hybrid keeps ISTA's experts and requantises the up/gate tensors of 27 layers from original BF16 weights using an importance matrix — which is how it passes the model Strata runs while staying close to Unsloth's UD-IQ4_XS at a smaller size. GSM8K (first 200 problems, greedy): Avalon 196/200 and Strata 196/200. So the speed is not bought with a dumber model; it is bought with memory placement, kernel work and MTP.

The part neither of the others does: two GPUs as a pipeline

Avalon's most distinctive feature is not on Strata's list: with two GPUs visible it runs the model as a two-stage pipeline by default. Layers are split 23/25 with -ts moving the boundary, each GPU keeps its own KV cache and expert cache, prompt chunks are pipelined so both PCIe links are busy, and only hidden state crosses between stages. On two RTX A4000 that turns a plain layer split's 1,079–1,190 t/s prefill into 1,669–1,761 for a 19k prompt. It is untested on two AMD cards and beyond two GPUs — the README says so — but it is a genuinely different answer to "what do I do with two old cards".

What to take from this

For our kind of hardware — one box, fixed cards, models we choose ourselves — the three tools map onto three different questions. llama.cpp answers "which model can I run today, and how stable is my serving layer?". Strata answers "what does this model cost me in tokens/s on this card, off the shelf". Avalon answers "how far can one model be pushed on hardware that used to be considered too old for it", and it pays for that with a strict scope: one model family, custom quant types mainline cannot load, ROCm installer only, RDNA3 kernels tuned and NVIDIA un-tuned.

Three honest conclusions from reading all three READMEs:

1. Scope is the price of speed. Avalon's 44% over Strata comes with a single-model mandate; Strata's breadth comes with its own limits; llama.cpp's generality is what lets it run everything and be the thing you build around.
2. The memory tier is the interesting engineering, not the tokens/s. Owning your RAM instead of trusting the page cache is the fix that turns a crash into a working 16 GB machine — whichever engine you use.
3. Read the caveats in the document, not just the headline table. Avalon corrected its own README numbers mid-flight. That is the difference between a project you'd build on and one you'd only quote.

If you want to try Avalon on the machine you already have: the installer checks your hardware, builds llama-server for the detected GPU, downloads ~98 GiB of disk worth of model and MTP head with SHA-256 verification, and then avalon-server serves an OpenAI-compatible API. For a serving layer that treats engines as interchangeable — which is how I'd run any of these — it slots in under an existing launcher too: Lemonade will take Avalon's binary in place of its own ROCm build with one environment variable and the model listed where it lives.

Avalon: https://github.com/freriadoc/Avalon
Strata: https://github.com/Niko1221/Strata
llama.cpp: https://github.com/ggml-org/llama.cpp