You pull a 14B or 32B model in Ollama. On the first prompt, response generation is blazing fast—delivering a smooth 35 to 45 tokens per second. But on your third turn, after feeding in a 3,000-token function definition or log trace, generation suddenly grinds to a halt. The model stutters, your CPU fans rev to 100%, and token generation plummets to a painful 1.2 tokens per second.
Your GPU has not crashed, and Ollama hasn't thrown an error. What you are witnessing is silent CPU fallback—one of the most frustrating performance traps in local AI engineering.
In this technical guide from The Zero-Dollar Dev, we break down why Ollama silently offloads transformer layers from VRAM to system RAM, how to verify it instantly with CLI diagnostics, and the exact configuration flags to lock your models into GPU memory permanently.
1. The Physics of Silent CPU Fallback
When Ollama initializes an LLM, its runner (built on llama.cpp) inspects your system's available CUDA or Metal memory. If the entire model cannot fit into contiguous GPU VRAM, the runtime does not throw an out-of-memory (OOM) error by default. Instead, it partitions the model's neural network layers between two distinct memory spaces:
| Memory Domain | Hardware Bus | Peak Theoretical Bandwidth | Latency Profile |
|---|---|---|---|
| GPU VRAM (GDDR6X) | Direct On-Die Memory Bus (e.g. RTX 3080/4080) | 760 – 1,008 GB/s | Sub-microsecond tensor transfers |
| System RAM (DDR4 / DDR5) | PCIe Gen4 / Gen5 Bus (x16 slot) | 38 – 64 GB/s (PCIe bottleneck: ~31.5 GB/s) | Severe bus transit latency |
Notice the massive discrepancy: GPU VRAM is 15x to 25x faster than system DDR5 RAM, and the PCIe interconnect creates a severe data transfer bottleneck. During inference, every single autoregressive token requires forward passes through all layers sequentially. If even 2 out of 48 layers are offloaded to system RAM, the entire inference pipeline stalls at every layer transition while weights and activation tensors are copied back and forth across the PCIe bus.
2. How to Detect Silent CPU Fallback in Real Time
Because Ollama does not print warning messages to standard output when offloading layers, you must inspect the runtime engine directly.
Method A: Inspect Ollama Active Process State
While your model is running a query, open a second terminal and execute:
ollama ps Examine the PROCESSOR column in the output:
NAME ID SIZE PROCESSOR UNTIL qwen2.5-coder:14b b38006e8e6e5 9.0 GB 68%/32% CPU/GPU 4m52s If you see anything other than 100% GPU, silent CPU fallback is actively degrading your throughput.
Method B: Inspect Ollama Server Daemon Logs
Ollama logs the exact layer allocation breakdown during model initialization. To view this:
# On Linux (systemd) journalctl -u ollama --no-pager -n 50 | grep -E "load_tensors|offloaded" # On macOS cat ~/.ollama/logs/server.log | grep -E "load_tensors|offloaded" # On Windows (PowerShell) Get-Content "$env:LOCALAPPDATA\Ollama\server.log" -Tail 100 | Select-String -Pattern "offload" You will see log entries detailing layer assignments:
llm_load_tensors: offloaded 34/48 layers to GPU llm_load_tensors: 14 layers remaining on CPU llm_load_tensors: VRAM used: 7842 MB, CPU RAM used: 3120 MB 3. The Three Hidden Triggers of Silent Fallback
Trigger 1: Dynamic KV-Cache Growth Exceeding Headroom
When you start Ollama, it might fit 40 layers into an 8GB GPU, leaving only 300MB of free VRAM. As your conversation lengthens to 8,000 tokens, the Key-Value (KV) cache dynamically expands (requiring up to 2.5GB of additional memory). The CUDA memory allocator fails, forcing the runtime to evict layer tensors or spill dynamic activations to host system memory. (See our deep dive: Llama 3.3 70B VRAM Calculation Math).
Trigger 2: High Default Context Window Allocation
In modern GGUF models, the default context window (num_ctx) can be 32,768 or even 128,000 tokens. Ollama attempts to pre-allocate memory structures for this full context window, consuming multiple gigabytes of VRAM before the first prompt token is even processed.
Trigger 3: Competing CUDA Context Overhead
Operating systems, display servers (Xorg/Wayland/Windows DWM), and open browser tabs hardware-accelerated by Chromium consume between 600MB and 1.5GB of GPU VRAM. If Ollama calculates available VRAM without accounting for display buffer spikes, it over-allocates and trips fallback.
4. The Fix: Permanent Configuration to Prevent CPU Fallback
Step 1: Quantize the KV-Cache to 8-Bit (Cuts Memory by 50%)
By default, Ollama stores KV-cache tensors in 16-bit floating point precision (FP16). By switching to 8-bit quantized KV-cache (Q8_0), you instantly cut KV-cache VRAM consumption in half with zero loss in coding precision:
# Linux / macOS export OLLAMA_FLASH_ATTENTION=1 export OLLAMA_KV_CACHE_TYPE=q8_0 # Windows (Persistent System Environment Variable) [System.Environment]::SetEnvironmentVariable('OLLAMA_FLASH_ATTENTION', '1', 'Machine') [System.Environment]::SetEnvironmentVariable('OLLAMA_KV_CACHE_TYPE', 'q8_0', 'Machine') Step 2: Lock the Context Window in Your Modelfile
Never leave num_ctx unbound when operating near your GPU's VRAM ceiling. Create a custom Modelfile to set a deterministic context limit:
FROM qwen2.5-coder:14b-instruct-q4_K_M # Pin context window to 8,192 tokens to guarantee deterministic VRAM headroom PARAMETER num_ctx 8192 # Pin GPU layer offload count (e.g. 48 for full offload) PARAMETER num_gpu 48 # Low temperature for precise code generation PARAMETER temperature 0.2 PARAMETER top_p 0.95 Build and run the locked model:
ollama create qwen14b-locked -f ./Modelfile ollama run qwen14b-locked 5. Automated Watchdog Script (Auto-Kill on CPU Spillover)
To automatically detect when an inference query spills over to CPU and restart Ollama before your machine locks up, save this lightweight Bash watcher:
#!/usr/bin/env bash # ollama-watchdog.sh - Detect and alert on CPU fallback while true; do PROCESSOR=$(ollama ps | awk 'NR>1 {print $4}') if [[ "$PROCESSOR" =~ "CPU" ]]; then echo "[ALERT] Silent CPU Fallback Detected: $PROCESSOR" notify-send "Ollama Fallback" "Model running on CPU: $PROCESSOR" fi sleep 5 done 6. Related Technical Benchmarks & Guides
Explore our complete library of local AI hardware benchmarks:
