Which Ollama model fits a CPU-only server? A RAM sizing cheat sheet

No GPU? Ollama still works. It loads the model into RAM and runs it on the CPU. The question is which model fits the memory you have, with room left for the context and the OS.

Here's how we size it, using the download sizes from Ollama's library and one check on the server.

1. Read the size on the library page

Every tag on ollama.com lists its download size. That file is roughly what lands in RAM before context.

2. Check the quantization in the tag

Many default tags are 4-bit, but not all of them:

• llama3.2:3b is the same file as llama3.2:3b-instruct-q4_K_M (2.0 GB). The 8-bit q8_0 is 3.4 GB.
• llama3.1:8b: 4.9 GB at 4-bit, 8.5 GB at 8-bit.
• llama3.2:1b defaults to 8-bit (1.3 GB). The 4-bit version is 808 MB.

Pull the full tag when you care:

ollama pull llama3.2:1b-instruct-q4_K_M 

3. Rule of thumb: model file ≤ half the RAM

This is our rule, not Ollama's: keep the model file at or under about half the server's RAM at the default context, then verify.

A 27B model needs more than 16 GB. On CPU, bigger also means slower, so measure before you commit.

4. The settings that add memory on top of the model

Four server settings decide how much RAM sits on top of the model file:

• OLLAMA_CONTEXT_LENGTH: default context. Ollama picks 4096 tokens without a large GPU. Longer context costs more memory.
• OLLAMA_NUM_PARALLEL: requests served at once per model (default 1). Ollama's FAQ says RAM scales with NUM_PARALLEL × CONTEXT_LENGTH.
• OLLAMA_MAX_LOADED_MODELS: models kept loaded at once (3 for CPU). On a small server, set it to 1.
• OLLAMA_KV_CACHE_TYPE=q8_0: about half the cache memory of the default f16, with a very small precision loss. It applies when Flash Attention is on (OLLAMA_FLASH_ATTENTION=1 forces it).

Set them with sudo systemctl edit ollama.service:

[Service] Environment="OLLAMA_CONTEXT_LENGTH=8192" Environment="OLLAMA_NUM_PARALLEL=1" Environment="OLLAMA_MAX_LOADED_MODELS=1" Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_KV_CACHE_TYPE=q8_0" 

sudo systemctl restart ollama 

5. Verify on the server

Load the model without a prompt, then look at what it takes:

ollama run llama3.2:3b "" ollama ps free -h 

In ollama ps, SIZE is the memory the loaded model uses, PROCESSOR should say 100% CPU, and CONTEXT is the context it loaded with. In free -h, the available column should still have a margin.

Then measure speed. The API returns eval_count (tokens) and eval_duration (nanoseconds):

curl -s http://localhost:11434/api/generate \ -d '{"model": "llama3.2:3b", "prompt": "Explain DNS in two sentences.", "stream": false}' \ | python3 -c 'import json,sys; r=json.load(sys.stdin); print(round(r["eval_count"]/r["eval_duration"]*1e9,1), "tokens/s")' 

Run it a few times. If it's too slow for your use, go smaller or add vCPUs.

Troubleshooting

• Model won't load or the service restarts under load: out of memory. Use a smaller tag or quantization, lower CONTEXT_LENGTH/NUM_PARALLEL, or add RAM. Check journalctl -e -u ollama.
• First answer after a pause is slow: the model unloaded (default keep-alive is 5 minutes). Set OLLAMA_KEEP_ALIVE=30m.
• Long chats forget the start: the prompt is longer than the context window. Raise OLLAMA_CONTEXT_LENGTH if RAM allows.
• Disk fills up: every pulled tag stays on disk. ollama ls, then ollama rm the ones you don't use.

Is CPU worth it?

For small models, batch jobs and private internal tools, yes. For high-volume chat, often not. In our self-hosting vs SaaS cost study, a small hosted API model was cheaper below roughly 150M input + 30M output tokens a month.

The full guide, with server sizes and hourly prices: Choose an Ollama model size for a CPU server.

Disclosure: we run VeloxaHost, a CPU server host. Model sizes are from Ollama's library at the time of writing; check the tag page before you pull.