Why Your 19.5GB LLM Crawls on 24GB VRAM (The KV Cache Fix)

You find a promising coding model on Hugging Face. Its GGUF file is approximately 19.5 GB. Your GPU has 24 GB of VRAM.

The math looks simple:

19.5 GB of model weights < 24 GB of VRAM.

It should fit, right?

You download it, load it through Ollama, connect it to Cursor or Cline, and start working on a real codebase.

Then something unexpected happens.

Generation crawls. Your GPU memory is almost full. Some model layers are running on the CPU, and your coding assistant takes forever to finish a response.

You haven't necessarily chosen the wrong model or GPU. You may have overlooked one of the biggest memory consumers in local LLM inference: the KV cache.

Let's understand why this happens, how to diagnose it, and how to reclaim GPU memory without immediately sacrificing model quality.

1. First, check where your model is running

Before changing configurations, determine whether Ollama is offloading model layers to system memory.

While your model is loaded, open another terminal:

ollama ps 

Look at the PROCESSOR column. You might see something like:

NAME SIZE PROCESSOR qwen2.5-coder:32b 22 GB 100% GPU 

Or:

NAME SIZE PROCESSOR qwen2.5-coder:32b 22 GB 82%/18% CPU/GPU 

Here is how to interpret the result:

• 100% GPU: The model is running entirely on the GPU.
• 100% CPU: The model is running entirely on the CPU.
• 82%/18% CPU/GPU: Some model layers are assigned to CPU execution.

Important: ollama ps reports model layer placement, not a direct measurement of KV cache allocation or every possible memory bottleneck.

For a more complete diagnosis, also monitor GPU memory and utilization:

nvidia-smi -l 1 

Pay attention to:

• GPU memory consumption
• GPU utilization
• Power consumption
• Other processes using VRAM (browsers, Electron apps, IDE helpers, displays)

CPU offloading can significantly reduce generation performance, but it isn't the only explanation for slow inference. Prompt processing, memory bandwidth, model architecture, quantization, and runtime configuration also matter.

2. The hidden VRAM consumer: KV cache

When an LLM generates text, it repeatedly processes tokens. To avoid recalculating attention information for every previous token, the model stores intermediate key and value tensors. This is called the Key-Value (KV) cache.

The longer your conversation or context window becomes, the more memory the KV cache consumes.

Your actual VRAM requirement looks closer to:

Total VRAM = Model Weights + KV Cache + Compute Buffers + Runtime Overhead 

This is why a model that appears to fit based on its GGUF file size can still exceed your available VRAM.

Calculate KV cache requirements

For a conventional transformer using grouped-query attention (GQA), an approximate KV cache calculation is:

Where:

Let's use Qwen2.5-Coder-32B as an example. Its relevant architecture parameters are:

• 64 transformer layers
• 8 KV heads
• 128 dimensions per attention head

We can calculate the approximate KV cache memory for different precisions at 32K context:

def kv_cache_gib( layers: int, kv_heads: int, head_dim: int, context: int, bytes_per_element: float, ) -> float: kv_bytes = 2 * layers * kv_heads * head_dim * context * bytes_per_element return round(kv_bytes / (1024 ** 3), 2) layers = 64 kv_heads = 8 head_dim = 128 context = 32768 for name, precision in [ ("FP16", 2.0), ("Q8_0 (approx.)", 1.0), ("Q4_0 (approx.)", 0.5), ]: memory = kv_cache_gib(layers, kv_heads, head_dim, context, precision) print(f"{name}: {memory} GiB") 

Approximate output:

FP16: 8.00 GiB Q8_0 (approx.): 4.00 GiB Q4_0 (approx.): 2.00 GiB 

(Note on units: 8.00 GiB is binary gigabytes, or ~8.59 GB decimal; similarly, a 19.5 GB GGUF is ~18.16 GiB. Actual quantized cache allocations can also be slightly higher due to block metadata, alignment, and backend implementation details.)

Now consider the implications.

A 32B Q4_K_M model with roughly 19.5 GB of weights already consumes most of a 24 GB GPU's capacity. At 32K context, an FP16 KV cache asks for another ~8.00 GiB (~8.59 GB), plus memory for CUDA/compute buffers and the inference runtime.

The model fits. The workload doesn't. That's the KV cache cliff.

Why coding agents make this worse

When you use an IDE assistant such as Cursor or Cline, the context isn't necessarily limited to your latest question. Depending on the application and configuration, a request can include:

• Previously exchanged messages
• Open source files and relevant code snippets
• Project structure
• Tool execution results and terminal output
• Repository instructions

A coding assistant working on a real repository can therefore consume considerably more context than a simple chat conversation.

Also, keep in mind that Ollama's default context length is generally 4,096 tokens unless configured otherwise—some apps and API clients explicitly request larger contexts (num_ctx). Don't assume every coding agent automatically uses 32K; check the actual context configuration and request payload.

3. Reclaim VRAM by quantizing the KV cache

Before downgrading your model weights from Q4_K_M to a smaller (and potentially less capable) quantization, consider quantizing the KV cache first.

Ollama supports different KV cache precisions:

These are approximate comparisons, not exact runtime allocation guarantees. For many local inference workloads, q8_0 is a practical starting point because it cuts cache memory roughly in half while preserving quality far better than q4_0.

Option A: Optimize Ollama

Configure these server environment variables before starting the Ollama server:

export OLLAMA_FLASH_ATTENTION=1 export OLLAMA_KV_CACHE_TYPE=q8_0 export OLLAMA_NUM_PARALLEL=1 ollama serve 

Tip: If Ollama is already running as a background service (systemd on Linux or the macOS/Windows app), restart that service with these environment variables set. Exporting them in an unrelated terminal will not change the running daemon.

Here is why each setting matters:

• OLLAMA_FLASH_ATTENTION=1: Ensures Flash Attention is enabled (Ollama auto-enables it on supported backends, and it is required for KV cache quantization).
• OLLAMA_KV_CACHE_TYPE=q8_0: Uses an 8-bit quantized KV cache instead of FP16, roughly halving KV cache VRAM usage.
• OLLAMA_NUM_PARALLEL=1: Prevents Ollama from multiplying the KV cache allocation across multiple concurrent sequence slots on a single-user machine.

Set an explicit context length

Avoid allocating a larger context window than your hardware can keep in VRAM. Create a Modelfile:

FROM qwen2.5-coder:32b PARAMETER num_ctx 16384 

Build and run your customized variant:

ollama create qwen32b-16k -f Modelfile ollama run qwen32b-16k 

Remember that the client application making API requests can still override num_ctx in its request options (and OLLAMA_CONTEXT_LENGTH controls the server default). Verify actual runtime behavior with ollama ps and nvidia-smi rather than assuming the Modelfile always wins.

Your goal is to keep model execution 100% GPU while leaving enough VRAM headroom for your desktop and coding context.

4. Prefer llama.cpp when you need direct control

If you want explicit CLI control over KV quantization, offloading, and server flags, llama.cpp gives you fine-grained options:

llama-server \ -m ./models/qwen2.5-coder-32b-q4_k_m.gguf \ -c 16384 \ -ctk q8_0 \ -ctv q8_0 \ --flash-attn on \ -ngl 99 

What each flag does:

Exact support depends on your llama.cpp build, GPU backend, and hardware. If it still exceeds VRAM, step down the context size or test q4_0 KV cache before abandoning your weight quantization.

5. Calculate VRAM requirements before downloading a model

Here's a mistake I see frequently in local LLM experimentation: checking a GGUF's file size, comparing it against GPU VRAM, and assuming it will run smoothly.

Before downloading a 20–40 GB model, account for:

• Model weight memory
• KV cache precision (f16 vs q8_0 vs q4_0)
• Target context length
• Runtime overhead and compute buffers
• VRAM already used by your display/IDE/other apps
• Number of concurrent sequences
• Attention architecture (e.g., GQA vs MLA)

For example, with Qwen2.5-Coder-32B (Q4_K_M, 19.5 GB weights) using a q8_0 KV cache on a 24 GB RTX 4090, dropping from 32K context (~4.29 GB KV, ~25.22 GB total — spilling to system RAM) to 16K context (~2.15 GB KV, ~23.08 GB total estimate) can be the difference between CPU offload and fitting inside 24 GB.

This is why I built the VRAMStack LLM VRAM Calculator to model weight quantization, KV cache precision, context windows (4K–128K), and multi-GPU splits before downloading large weights.

Related breakdowns and benchmarks:

• Can I Run Qwen2.5-Coder-32B on an RTX 4090?
• Can I Run DeepSeek-R1-Distill-14B on an RTX 3060?
• Best LLMs for 24GB VRAM
• KV Cache VRAM Formula & Sizing Guide

(Treat calculator outputs as pre-download planning estimates, not runtime guarantees—real memory usage varies across engines and backends.)

6. Practical troubleshooting checklist for slow local LLMs

Before buying another GPU or dropping to a much smaller model, run through these checks:

• [ ] Run ollama ps to verify 100% GPU vs CPU/GPU split
• [ ] Run nvidia-smi to check actual VRAM used and background GPU apps
• [ ] Verify the effective context length requested by your IDE/client
• [ ] Confirm Flash Attention is enabled
• [ ] Switch KV cache precision from f16 to q8_0
• [ ] Lower context length if your task doesn't need 32K+
• [ ] Set OLLAMA_NUM_PARALLEL=1 while testing single-user speed
• [ ] Compare tokens/sec before and after each change
• [ ] Measure prompt processing (prefill) separately from token generation

If the model is 100% GPU and still feels slow, check GPU memory bandwidth, compute utilization, prompt length, batching, and backend settings—not every slow local model is a KV cache issue.

Final thoughts

Running a local coding LLM isn't just about whether the weights fit in VRAM. It's about fitting the entire runtime workload—weights, KV cache, buffers, and IDE context—into GPU memory together.

Before downgrading your model quality, test context length and q8_0 KV cache quantization, measure ollama ps and nvidia-smi, and dial in the sweet spot for your card.

Happy local LLM experimenting!

Useful references

• Ollama FAQ and configuration
• Ollama Modelfile documentation
• llama.cpp on GitHub
• VRAMStack Calculator