Disclosure: this post contains affiliate links; I may earn a commission at no cost to you.
TL;DR
Before you rent a GPU for an open-weight model, do the VRAM math. Weights take roughly parameters x bytes per weight, then add 15 to 25 percent for KV cache and runtime. A 70B model at 4-bit fits one 80 GB card. A 8B model fits a cheap 24 GB card. Rent by the second, stop the pod when idle, and compare against API pricing before committing.
The VRAM calculator (copy-paste)
def vram_gb(params_b, bits=4, overhead=0.2): weights = params_b * bits / 8 # GB for weights return round(weights * (1 + overhead), 1) for name, p in [("8B", 8), ("32B", 32), ("70B", 70)]: print(name, {b: vram_gb(p, b) for b in (16, 8, 4)}) | Model size | BF16 (16-bit) | 8-bit | 4-bit | Fits on |
|---|---|---|---|---|
| 8B | 19.2 GB | 9.6 GB | 4.8 GB | one 24 GB card |
| 32B | 76.8 GB | 38.4 GB | 19.2 GB | 4-bit on 24 GB, 8-bit on 48 GB |
| 70B | 168 GB | 84 GB | 42 GB | 4-bit on one 80 GB card |
Long contexts and many concurrent users grow the KV cache, so treat the table as a floor, not a promise.
Launch an OpenAI-compatible endpoint with vLLM
pip install vllm vllm serve <your-huggingface-model-id> \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --api-key $MY_KEY Then point any OpenAI SDK at http://<pod-host>:8000/v1. For quantized weights add --quantization awq.
What it costs (assumptions stated)
RunPod Secure Cloud on-demand rates seen on runpod.io/pricing in late October 2026, billed per second (they move often): A100 80GB about $1.79/hr, H100 PCIe about $2.89/hr.
Worked example: test a 4-bit 70B model for 4 hours on an A100 80GB, then stop the pod: about $7. Left running all month at the same rate: about $1,290 (1.79 x 720 h). The money-saving habit is simply stopping the pod. Serverless endpoints bill only while workers run, but their hourly rate is higher, so check utilization first.
| Situation | Better choice |
|---|---|
| Spiky, a few hours a day | API, or serverless GPU |
| Steady high volume, one model | Dedicated pod |
| Data must stay in your control | Rented GPU |
| Fine-tuned or niche model | Rented GPU |
Try it
To test this without buying hardware, spin up a pod on RunPod here and be serving a model in about ten minutes. Start with a 24 GB card for 8B models and step up only when the math says you must.
Full disclosure: TokenLean participates in affiliate programs, including RunPod's. If you sign up or buy through links on this page, I may earn a commission at no extra cost to you. Prices are from public pages at the time of writing and may change.
