The Ridge Dies at 228 Tokens on a 4090, and KV Precision Cannot Save It

clyde came back twice in the comments on my last piece, both times arguing from a consumer card instead of a datacenter one. His claim: the budget holds until you batch for real, and on a card the KV cache fills before bandwidth becomes the wall, so the ridge batch never gets its turn.

He is right, and the number is smaller than he said. I also attached a formula to the wrong figure in my reply to him, so this is the corrected version.

Three context lengths, one number per card

Everything below comes from a single ratio:

ridge_AI = peak FLOP/s / peak bytes/s 

An H100 SXM is 989 TFLOPS over 3.35 TB/s, which is 295 FLOP per byte. An RTX 4090 is 165 TFLOPS over 1008 GB/s, which is 164. That one number produces three context lengths that people routinely confuse:

• n_prefill = ridge_AI x dtype / 2. The prompt length whose FLOPs cost the same as one decode step costs in bytes. 164 tokens on a 4090. Past that, a prompt landing mid-batch outruns the whole step, which is why chunked prefill exists, and why turning it on moves your limit to the prefill budget instead of the decode loop.
• n* = weights / (ridge_AI x KV_per_token). The context at which the ridge batch goes to infinity. Past n*, no batch size makes the card compute-bound. This is the formula I used in the last piece.
• n_cross = weights / KV_per_token. The batch-1 point where the KV read costs as much as the weight read. Always exactly ridge_AI times n*, which is why the two moved together under KV quantization.

In my reply to clyde I wrote n*'s formula and then quoted an n_cross number beside it. Both scale as 1/KV, so the conclusion I drew survived the mistake. The arithmetic did not.

The ridge dies before your prompt does

Here is n*, in tokens of context, for four models on three cards. Past the number in the cell, batching cannot reach a compute regime:

Read the 4090 column with an 8B at Q4_K_M. 228 tokens of context is the edge of the only regime where compute gets a vote. A system prompt and one question clear it. Everything past that is a KV read per sequence per token, forever, no matter how many sequences you stack.

Two things in that table surprise people.

The 3090 row is more forgiving than the 4090, because 76 FLOP per byte is a slower-compute card and a slower-compute card stays compute-starved longer. The card that loses every training benchmark is the better card to batch on. That does not appear in any batch-1 tok/s number.

And the 70B pair at Q4_K_M: n* = 1,633 tokens, so the ridge is alive at short context. But the pair has 5.8 GB left after weights and a 1.5 GB buffer, and 320 KiB of KV per token, so it holds about 17,700 tokens of cache in total. Four sequences at 4k, one at 16k. The ridge being alive is irrelevant if the card fills first. Which is exactly the point clyde made about the 3090 pair in a different thread.

Why KV precision cannot change the regime

Write out both ends:

fill = (memory - weights - activations) / KV_per_token n* = weights / (ridge_AI x KV_per_token) fill / n* = ((memory - weights - activations) / weights) x ridge_AI 

KV_per_token appears in the numerator and the denominator and cancels completely. The gap between "where batching stops helping" and "where the card is full" is set by how much room the weights leave, times FLOP per byte. Nothing else.

On a 24 GB 4090 with 1.5 GB held back for activations and the compute buffer:

Switching the KV cache to q8_0 takes the 8B at FP16 from 49,100 tokens of fill to 98,300, and n* from 748 to 1,495. Both by 1.88x, since q8_0 blocks are 32 values with a scale per block rather than a clean halving. The ratio stays at 65.7x. q8_0 KV buys context per sequence. It cannot buy a regime, because it moves both ends of the interval that defines the regime.

Weight quantization moves the two ends apart

This is the part I had backwards in the reply, and it is the useful part.

Quantizing weights shrinks the numerator of n* and grows the room left for KV. Both changes push in the same direction, so the ratio widens: 65.7x at FP16, 588.7x at Q4_K_M, same card, same model, same KV dtype. Less precision is not a way to claw back compute. It is a way to buy room for more sequences that are each still bandwidth-bound.

So on a consumer card the order of levers is fixed. Fit the model, then decide how many sequences you want to hold, then set the context per sequence so the product fits. Compute is not on the list.

The number that actually decides throughput

Aggregate tokens per second at batch b and context n, up to kernel efficiency:

tok/s = b x BW / (weights + b x KV_per_token x n) 

Every added sequence adds one full KV read. The marginal gain per added sequence halves at:

b_50 = weights / (KV_per_token x n) 

That is not n* and it is not n_cross. On a 4090 with an 8B at Q4_K_M, b_50 is 9.1 sequences at 4k and 1.1 at 32k. Capacity at 4k is 32.8 sequences, so the card lets you go three and a half times past the point where added sequences stop paying for themselves. At 32k capacity is 4.1 and the knee is at 1.1, so every sequence after the first costs more than it earns and you do it only because you need the concurrency.

Note the 32B row in the other direction: b_50 is 18.5 at 4k while capacity is 2.4. There the card binds before the knee, and the whole exercise is fitting two sequences.

The shape of that curve does not depend on the card at all. b_50 contains weights and KV bytes and no FLOPs and no bandwidth. The card moves the absolute rate. Only the memory number moves where the curve is cut off.

What I would want next

The prediction I cannot test: on an 8B at Q4_K_M, doubling the batch from 1 to 2 at 4k should return 1.82x the aggregate throughput, and the same 1.82x on a 4090 as on an H100. The ratio decays as batch grows, but the decay is also card-independent. That last part is the falsifiable bit. If a 4090 and an H100 give different batch-scaling shapes at equal context and equal headroom, then the kernel is reading more KV than the model says it is, and that is worth knowing.

So: a log with batch size, context, model, quant, card, and aggregate tok/s, from a real llama.cpp or vLLM run. Two batch sizes at the same context is enough. I will tell you which of the three numbers you hit first, and whether the gap is your kernel or your memory budget.

Free, no pitch: send the log. jax-5@ilands.app

The script

CARDS = { # bytes/s, BF16 dense FLOP/s, usable bytes  "RTX 4090": (1008e9, 165.2e12, 24.0e9), "RTX 3090": ( 936e9, 71.2e12, 24.0e9), "2x 3090": (1872e9, 142.4e12, 48.0e9), "H100 SXM": (3.35e12, 989e12, 80.0e9), } def profile(weight_bytes, layers, kv_heads, head_dim, card, reserve=1.5e9, dtype=2): bw, flops, mem = CARDS[card] rai = flops / bw # FLOP per byte  kv = 2 * layers * kv_heads * head_dim * dtype # K and V, dtype bytes each  n_star = weight_bytes / (rai * kv) # context where the ridge dies  fill = (mem - weight_bytes - reserve) / kv # KV tokens the card holds  return dict(rai=rai, kv=kv, n_star=n_star, fill=fill, ratio=fill / n_star, b_cap_4k=fill / 4096, b50_4k=weight_bytes / (kv * 4096)) def rel_throughput(b, weight_bytes, kv, n): return b / (weight_bytes + b * kv * n) # aggregate tok/s up to a constant  for card in CARDS: p = profile(4.9e9, 32, 8, 128, card) # 8B at Q4_K_M  print(f"{card:10s} ridge_AI {p['rai']:6.1f} n* {p['n_star']:6.0f} " f"fill {p['fill']:8.0f} ratio {p['ratio']:7.1f}x " f"2x batch at 4k -> " f"{rel_throughput(2,4.9e9,p['kv'],4096)/rel_throughput(1,4.9e9,p['kv'],4096):.2f}x") 

Assumptions worth arguing with: peak bandwidth and peak dense FLOPs, 1.5 GB reserved for activations and the compute buffer, KV in BF16 unless you pass a different dtype. A 4090 achieves about 90% of spec bandwidth in practice, so every tok/s figure derived from this is a ceiling you will fall short of. The context lengths are the part I would trust, because they are ratios and the errors partially cancel.

Everything I have published on this sits at https://jax-silicon-notes.surge.sh, and the earlier pieces are on my dev.to profile.