Most "can my rig run this model?" threads guess at VRAM numbers and ignore the two killers that actually cause CUDA/Metal out-of-memory panics or 3-token/sec crawl speeds: KV-cache growth at 8K–32K context and physical memory bus bandwidth.
Here is the exact memory footprint math and measured token generation ceiling across 24GB GDDR6X CUDA cards vs. Unified Memory nodes so you can size local inference hardware without guessing.
1. The Real Local LLM Memory Formula
Static GGUF file size is never your actual runtime footprint. Active memory usage equals:
Total VRAM = Model Weights + KV Cache (Context Length) + Compute Buffer (~0.8 GB)
And raw generation speed (tokens/sec at batch size 1) is strictly memory-bandwidth bound:
Max Theoretical tok/s ≈ Memory Bandwidth (GB/s) ÷ Active Weights (GB)
(Real-world llama.cpp / Ollama / vLLM execution typically achieves **65%–78%* of theoretical peak bandwidth).*
2. Measured VRAM Matrix (Weights vs. 8K & 32K Context)
| Model | Quant | Static Weights | 8K Context Total | 32K Context Total | Fits Single 24GB GPU? |
|---|---|---|---|---|---|
| Phi-4 14B | Q6_K | 12.1 GB | 13.6 GB | 15.3 GB | Yes (~54 tok/s) |
| Mistral Small 24B | Q4_K_M | 14.6 GB | 16.5 GB | 19.2 GB | Yes (~44 tok/s) |
| Qwen 2.5 Coder 32B | Q4_K_M | 19.7 GB | 22.0 GB | 25.7 GB | 8K Yes / 32K Needs -ctk q8_0 |
| DeepSeek R1 Distill 32B | Q4_K_M | 19.7 GB | 22.0 GB | 25.7 GB | 8K Yes (~33 tok/s) |
| Llama 3.3 70B | Q4_K_M | 42.8 GB | 46.8 GB | 54.9 GB | No (Dual 24GB GPU or 64GB+ Unified) |
| Qwen 2.5 72B | Q4_K_M | 44.1 GB | 48.2 GB | 56.5 GB | No (64GB+ Unified or 2x24GB w/ IQ4_XS) |
Cutting 32B KV-Cache Bloat Inside 24GB VRAM
If Qwen-2.5-Coder-32B or DeepSeek-R1-32B OOMs when your IDE agent pushes past 16K context on a 24GB card, enable Flash Attention and quantize the KV cache to q8_0:
llama-server -m qwen2.5-coder-32b-instruct-q4_k_m.gguf -c 32768 -ngl 99 --flash-attn -ctk q8_0 -ctv q8_0 That drops the 32K context overhead by ~45% with virtually zero perplexity loss on code generation.
3. Why PCIe Spillover Destroys Token Speed (-78% Penalty)
The moment a model exceeds 24.0 GB even by 1.5 GB without KV quantization, layers spill across PCIe 4.0 (~25 GB/s practical bidirectional throughput) into DDR4/DDR5 system RAM (~50–75 GB/s), tanking a 936 GB/s GPU down to ~7–9 tok/s.
- Best Fast 32B Coding & Flux/RVC Workstation: Single 24GB GDDR6X GPU (
936–1,008 GB/s) + DRAM-buffered Gen4 NVMe (7,000+ MB/sso 20GB GGUF cold-loads take under 3 seconds instead of stalling onmmap). - Best Silent 24/7 70B+ Home RAG Node: 48GB–64GB+ Unified Memory (
400+ GB/sbus, ~12W idle wall draw vs. 90W+ multi-GPU idle).
4. Test Your Own Hardware in 2 Seconds (Free Open Tools)
I built a set of zero-signup, client-side calculators and hardware matrices to check bottlenecks before buying parts:
- Interactive LLM Token/s & Bandwidth Simulator — Test exact quantizations against your GPU or Unified Memory bus.
- Local AI Rig Part-Picker & VRAM Builder — Check PSU transient sizing, PCIe lanes, NVMe load speeds, and price-per-GB specs.
- CUDA / Metal Stack-Trace OOM Analyzer — Paste
RuntimeError: CUDA out of memoryorggml_metallogs for instant flag fixes.
Or run a quick read-only POSIX check in macOS/Linux terminal (feel free to curl -sL https://sheerai.net/bench.sh first to inspect the shell script):
curl -sL [https://sheerai.net/bench.sh](https://sheerai.net/bench.sh) | sh What quantization + context window sweet spot are you running right now for local coding agents?