People ask me "will this model run on my box?" almost every week. Most of the time you don't need a benchmark to answer that — you need two numbers: how many parameters the model has, and how many bytes each parameter occupies at the precision you picked.

Here's the back-of-the-envelope math I use, a reference table to check against, and the reason I now treat quantization as the first decision rather than the last.

Two inputs, one estimate

Everything below is arithmetic, not measurement. Take the parameter count in billions, multiply by the number of bytes a single parameter takes at the chosen precision, then pad the result a little.

The rough rules of thumb for 2026:

  • FP16 — about 2 bytes per parameter
  • INT8 — about 1 byte per parameter
  • INT4 — about 0.5 bytes per parameter

On top of that, add roughly 20% for the KV cache, the runtime itself, and the temporary buffers that exist while the weights are being loaded. That buffer grows with context length, so treat the final figure as a floor, not a guarantee.

Putting it in a few lines of code

bytes_per_param = {"fp16": 2, "int8": 1, "int4": 0.5} def estimate_gb(params_b, precision, context_buffer_gb=0.0): weights_gb = params_b * bytes_per_param[precision] return weights_gb * 1.2 + context_buffer_gb estimate_gb(7, "int4") # ~4.2 GB estimate_gb(7, "fp16") # ~16.8 GB estimate_gb(70, "int4") # ~42 GB 

Notice that nothing here cares about the brand of your GPU. The same formula works for any checkpoint you can look up a parameter count for, which is why I keep it around instead of memorising hardware requirements.

Reference table by size and precision

These are theoretical weight sizes only — arithmetic you can repeat yourself, not benchmark results.

Parameters FP16 (~2 B/param) INT8 (~1 B/param) INT4 (~0.5 B/param)
3B ~6 GB ~3 GB ~2 GB
7B ~14 GB ~7 GB ~4 GB
13B ~26 GB ~13 GB ~7 GB
32B ~64 GB ~32 GB ~16 GB
70B ~140 GB ~70 GB ~35 GB

Add the ~20% buffer and your context allowance to any cell in that table and you get the number to compare against your hardware. That is exactly why a 7B model at INT4 feels comfortable on an 8 GB card, while the very same model at FP16 wants 16 GB or more. A 70B at INT4 lands around 35 GB, so it simply will not sit on a single 24 GB GPU.

Quantization is the biggest lever you have

Quantization means storing the model's weights at reduced precision: 16 bits per weight for FP16, 8 bits for INT8, and around 4 bits for INT4.

Each step down roughly halves the footprint. FP16 to INT4 shrinks the weights by about four times, which is the difference between a model that needs a workstation and one that runs on a consumer GPU. Of everything you can change, this moves the hardware target the most.

The trade-off is quality, and it grows with the compression ratio — the harder you squeeze, the more you should expect to give up.

VRAM and RAM do two different jobs

It took me a while to internalise this split: VRAM decides how fast the thing runs, while system RAM decides whether the model loads at all. You generally want both to be comfortable rather than one to be heroic — a machine that barely squeezes the weights into memory is not a machine you'll enjoy using.

A quick checklist before you download

  1. Look up the parameter count.
  2. Pick a precision and run the formula above.
  3. Add ~20% plus your context allowance.
  4. Compare the total against VRAM first, then RAM.
  5. If it doesn't fit, drop one quantization step before you start shopping for a GPU.

When you're still deciding what to actually run, it helps to pick the model first and size the hardware second — the estimate is only as useful as the thing you're estimating for.

This article was originally published on AI Release — https://ai-release.org — where I also post shorter notes in the channel at https://t.me/ai_release1.

Takeaway

You don't need a benchmark to predict whether a local model fits — you need the parameter count, the bytes per parameter at your chosen precision, and a ~20% buffer for context and runtime. Quantization, not raw model size, is usually what decides whether a model runs on your desk or in a data centre.