Kavish buys hardware for a hospital group, where patient questions cannot leave the building. His first question: how much GPU memory to buy. The surprise: a model whose files are 16 GB needs about 23 GB to run.

The file size is not the memory needed: the file is only the weights. Two more things fill the memory, and the second one is the one that surprises people.

The suitcase: three things fill GPU memory

GPU memory, called VRAM, is a suitcase that does not stretch. Three things go inside.

Heavy clothes are the weights: all the learned numbers, the model itself, packed once and staying in place.

Souvenirs are the KV cache: the model's saved notes about the conversation so far, which save it from re-reading everything for each new word. They pile up: longer chats and more people mean more souvenirs.

Lost space is overhead: odd gaps and buffers, planned with a percentage on top, usually 15%.

Add the three parts: if the total is bigger than the suitcase, the model will not run well, because fit is a yes or no. Unlike a real suitcase, souvenirs are freed when a chat ends, so pack for the busiest moment, not the average one.

The sums, done by hand

Take an 8-billion-parameter model with typical numbers: two bytes per number, chats of 4,096 tokens each, 8 chats open at once, 32 layers, 8 KV heads, head size 128, and a 24 GB card to compare against.

  1. Weights: 8 billion x 2 bytes = 16 GB.
  2. Notes for one token: 2 x 32 layers x 8 heads x 128 size x 2 bytes = 131,072 bytes, where the first 2 is one key plus one value.
  3. Notes for all chats: 131,072 x 4,096 tokens x 8 chats = about 4.29 GB.
  4. Overhead: 15% x (16 + 4.29) = about 3.04 GB.
  5. Add up: 16 + 4.29 + 3.04 = about 23.3 GB.

So the answer is about 23.3 GB: a 24 GB GPU fits it, with about 0.7 GB spare. The card that fits costs real money: a 24 GB RTX 4090 is about $1,600, as of November 2026. A data-centre H100 rents at about $2.50 an hour on major clouds. "Fits" is a budget decision, not just a sum.

Meta's real Llama 3.1 8B has exactly these numbers: 32 layers, 8 KV heads, head size 128, so 128 KiB of notes per token is a real figure for a real model you can download today. The 70B version has 80 layers and the same head size, with weights of about 140 GB in FP16, or 70 GB in FP8.

Change one number and watch it break

The sums are simple, which is what makes them useful: change one number at a time and watch the total move.

Double the batch from 8 chats to 16 and the cache doubles, the total passes 24 GB, and the card flips to "does not fit". The weights never changed. The cache grew with every extra person.

In a "large model, many users" setup the cache can be bigger than the weights, about 86 GB of notes against 70 GB of weights, which is the part people forget to budget for.

On a home PC the answer is friendlier. An RTX 4090 has 24 GB, and the real Llama 3.1 8B's weights take about 16 GB of it, leaving roughly 8 GB. One long chat of 8,000 tokens needs about 1 GB of notes, so that spare room is about 8 long chats at once, before overhead trims it. For one person experimenting, plenty; for a whole team chatting at once, not enough, and now you can show why with sums.

Make the weights smaller with fewer bits

The weights are usually the biggest block, and they can be shrunk by storing each number in fewer bits, called quantization. Think of rounding prices to the nearest dollar: the list gets shorter to write down.

One 8-billion-parameter model stored four ways: 32 GB at 4 bytes per number, 16 GB at 2 bytes, 8 GB at 1 byte, and about 4 GB at 4 bits, half a byte. With 8-bit weights the 16 GB model needs 8 GB, leaving about 16 GB on a 24 GB card for cache and overhead.

The catch is the same as rounding prices: rounding can nudge answers off a little. Prices rounded to the dollar still add up, but a model's answers may not, so test answer quality after you shrink by running the same sample questions before and after.

Two customer cases

A hospital group asks for a big model on one card. A 70-billion-parameter model at 2 bytes per number is 140 GB of weights alone, about 35 GB at 4 bits. Add cache and overhead, and cards come in sizes such as 24, 48 and 80 GB. The sum tells you the answer before you buy anything.

Then the worked-example model goes live, staff like it, and more people join. The weights stay 16 GB, the cache doubles to about 8.59 GB, overhead is about 3.69 GB, and the total is about 28.3 GB, over 24 GB. The fix: a smaller batch, shorter chats, or 8-bit weights.

Three mix-ups to avoid: the file size is not the memory needed; more users do not only mean more speed needed, since each user also adds notes, so memory grows too; and if it fits, it will not necessarily be fast, because fit is yes or no, while speed depends on other things, such as how fast data moves between memory and compute.

The full interactive chapter, with the VRAM calculator and the hand sums worked through, is free at https://deployedlab.dev/lessons/37-gpus-memory-sizing. Eleven chapters are free with no signup, and the full course is $5/month or $39/year at https://deployedlab.dev.