Your prefill/decode split has a network bill nobody puts in the diagram

A colleague's post this week made the case for splitting prefill and decode onto separate GPU pools: an 8,000-token prompt costs about 320 ms of pure compute-bound work, and during those 320 ms every other stream on that GPU freezes. Running prefill elsewhere fixes the tail. All of that is right.

The part missing from most diagrams of the split is the arrow between the two boxes, and that arrow has a size.

The KV cache is bigger than you think per token

The KV cache for one token is:

bytes_per_token = layers x kv_heads x head_dim x 2 (fp16) 

For a 32-layer model with 8 KV heads at 128 dimensions:

32 x 8 x 128 x 2 = 65,536 bytes = 64 KB per token 

That 64 KB is per token, per sequence. An 8,000-token prompt therefore produces a 512 MB cache that has to exist on the decode pool before a single output token can be generated. If prefill and decode are on different machines, those 512 MB move across the network, and it happens on the request's critical path.

What that costs in milliseconds

At 25 Gb/s the transfer eats about half of the win. At 10 Gb/s the split is a net loss for long prompts, which is exactly the case it exists to fix. The saving is real; it just lives on a different cable than the one the blog post is about.

So the split is a network decision, not only a serving decision

Three practical consequences:

1. Measure cache size before you measure throughput. It is four numbers from the model config, and it tells you whether your interconnect can carry the workload at all. A model with fewer KV heads (grouped-query attention) is dramatically cheaper to disaggregate than one without: 8 KV heads instead of 32 is a 4x smaller transfer, which can flip the decision.

2. Long context is where the split helps most and costs most. The 320 ms prefill grows linearly with prompt length, and so does the transfer. They track each other, so the ratio barely improves - you are trading one linear cost for another, plus a constant.

3. Quantized or compressed KV caches are a serving feature and a networking feature at the same time. Halving the bytes per token halves the transfer, which is often worth more than the memory it saves.

A number to put on the diagram

If I were annotating one of those architecture diagrams, the label on the arrow would not be "KV cache". It would be the transfer time in milliseconds at the link speed you actually have:

transfer_ms = bytes_per_token x context_tokens x 8 / link_bits_per_second x 1000 

If that number is a meaningful fraction of the prefill you removed, the split has not moved the work off the critical path - it has moved it onto a wire, where it is harder to see in a flame graph and much harder to shrink.

None of this argues against disaggregation. It argues for doing the arithmetic first, because the answer depends on an interconnect number that never appears in the model comparison table.