Hybrid attention buys cheap reading and expensive writing

The title is an argument, and the bench below is what it means. A hybrid-attention coder model advertises a two-million-token context window, and on my two RTX 3090s it ingested at 1,013 tokens per second while generating at 20. Reading came fifty times cheaper than writing, every time I measured it. That asymmetry decides which local jobs pencil out and which ones never will.

What ran, November 5, 2026

I built llama.cpp from the ref of an unmerged upstream pull request (ggml 0.29.0, commit 6414d5b), CUDA sm_86 on both cards, -ngl 99 -sm layer -fa, KV q8_0/q8_0, three repetitions per run. The file was Ripple-65B Preview in AD-IQ4_XS, 38,722,143,712 bytes on disk, 36.06 GiB resident split across the pair, 64.78 billion parameters.

Prefill rises with batch size and decode does not move. The server probe closed the gap between bench and practice: a 9,601-token prompt at ctx 32768 logged prefill at 956.26 tok/s over 10,040 ms, and decode at 18.61 tok/s, 53.74 milliseconds per token.

Why most layers only see 1,024 tokens

The model card declares the trick in its config. Eighty hidden layers, and layer_types puts 64 of them into sliding attention with a window of 1,024 tokens; only 16 run full attention. Native context is 2,097,152, license Apache-2.0, and the card claims 65B dense trained on 12 trillion tokens for agentic coding. That's why the advertised number works on paper. Most of the stack stops paying attention cost as the prompt grows, so a huge cache fits and fills cheaply. What doesn't shrink is generation, because every emitted token still walks all 80 layers once. On my cards that walk ran at twenty tokens per second regardless of how much sat in front of it.

The vendor numbers measure a different machine

The port author published their table against an 8x A100 40GB corpus. Their AD-Q4_K_M row suggests 48 GB of VRAM for 40.4 GB of weights, quality claimed as KLD divergence 0.0149 against BF16, worst-case 0.177, same-top-token agreement 95.6%, versus plain Q4_K_M at 0.0380, 0.447 and 93.0%. All vendor-reported. My contribution was running the file on used-card ISA and writing down what came back: prefill crossing a thousand tok/s, decode pinned under twenty-one. Nobody's benchmark transfers to your GPU, this model included, and the quantization ladder they advertise (24/32 GB builds coming, AD-IQ2_XXS already labelled for a single 24 GB card) is their measurement too.

Stock loaders can't open these files yet

Here is the install gate I checked before anything downloaded. The card says stock llama.cpp, LM Studio and Ollama do not load these files until llama.cpp PR #31692 merges, and I verified that PR live through the GitHub API while it sat open and unmerged, +311 additions across 14 files, with new code landing in a 200-line src/models/ricursive.cpp. mergeable_state read blocked. The HF repo showed 1,668 downloads by then, so people were fetching weights nothing on a stock install could run. A port went public ahead of the loader that needs it, and I had to build from the PR ref just to bench my own feasibility question.

How I benched without wrecking a running rig

Both cards already carried inference lanes when the question arrived. I stopped the watchdog timer first, then both llama-swap services, which took VRAM down to 924 and 318 MiB, ran the sweep, then started everything back and confirmed it with model lists plus a real chat completion carrying reasoning_content, back at 20,290 and 23,394 MiB used. Feasibility questions on GPUs that are busy get answered by parking, measuring, restoring, and only then deciding.

What the numbers decide

The ratio between the two halves was about fifty to one (1012.80 over 20.34). At 53.74 ms per token, twenty tokens costs 1.07 seconds and five hundred costs 26.9. Feed an agent loop ten thousand tokens of context every night and you pay roughly ten seconds of ingestion per 9.6k block while generation stretches into hours. That math put this model in the overnight batch bracket: read whole repos, emit small diffs, run while everyone else sleeps. A loop under 100 tokens per response fits, a chat surface anyone waits on does not, and that verdict came from measurement before install, nothing downloaded until it passed.

Hybrid attention trades cheap reading for expensive writing, and the trade says what to build against it. My rule before installing any big-context local model: measure pp and tg on your own hardware, compute hours-per-run against how long the machine may idle, and commit only when the loop fits.

What would make 20 tok/s worth running on my cards? An agent loop that reads a repo and emits sub-100-token diffs, which is exactly the job I leave open overnight. If your setup answers faster or slower than that, tell me what you're running and I'll benchmark against it.