Introduction

Earlier in this series I set up the basic platform: vLLM serving the model, LiteLLM as the gateway and an ingress in front. I kept the vLLM flags quite conservative there on purpose. In this part I want to go through what I actually changed afterwards, and why.

My setup: the GPUs are not passed through to the Kubernetes nodes, they are NVIDIA vGPUs on vSphere. Each node VM gets a GRID A100D-40C profile, so half of an A100 with 40 GB of memory. The model used is Qwen3.8-27B in NVFP4 and I want to achieve the best performance for the following requirements:

  • a context window of 250k tokens
  • serving many requests in parallel
  • one 40 GB vGPU

A lot of what follows applies to any GPU, but a few things are really vGPU specific, and those are the ones that cost me the most time.

Info: All the numbers in this post were measured on my hardware with vLLM v0.28.0. Your numbers might differ, but the reasoning behind them should still hold.

vGPU memory isn't what nvidia-smi tells you

The first thing I ran into was vLLM refusing to start at all, even though the model is only around 24 GB and the GPU has 40 GB.

The reason is that the vGPU reserves memory for itself. You can see it like this:

nvidia-smi --query-gpu=memory.total,memory.reserved,memory.free --format=csv 

On my 40C profile, memory.total says 40960 MiB, but around 3093 MiB are reserved, so only ~37867 MiB are actually free when the GPU is idle.

Why does this matter? vLLM has a startup check, that compares the free memory against --gpu-memory-utilization times the total memory. So on a 40C vGPU:

--gpu-memory-utilization Requested Result
0.95 38912 MiB can never start, more than is free
0.91 36864 MiB starts, but leaves almost no headroom
0.85 34816 MiB starts safely

I started with 0.85, and later went up to 0.91 to get more space for the KV cache. That works, but it's right at the limit. If a pod ever fails to come up after a driver or image update, rolling back to 0.85 is the first thing I'd do.

Also don't get confused by nvidia-smi showing "No running processes found" while vLLM is clearly using the GPU. For containerized workloads on vGPU that is normal. Look at the memory usage instead.

Never set expandable_segments on vGPU

When you hit an out-of-memory error, PyTorch helpfully suggests setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. On vGPU, don't. It uses the CUDA virtual memory management APIs (cuMemCreate, cuMemMap, ...), which vGPU simply doesn't implement. vLLM then dies at the very first tensor allocation with:

RuntimeError: CUDA driver error: operation not supported 

This one took me a while, because the error message has nothing to do with memory anymore.

Don't use --enforce-eager

When vLLM has trouble starting, a lot of people on GitHub issues suggest --enforce-eager. It disables CUDA graphs, which saves some memory and startup time. I tried it once, and measured this:

Mode Single stream tok/s
--enforce-eager 10.9
CUDA graphs (default) 52.2

That's 4.8x slower. The Qwen3.8 model is a hybrid architecture (only every fourth layer is full attention, the rest are linear/Mamba-style layers), so it runs a lot of small GPU kernels. Without CUDA graphs, every single kernel launch has overhead, and on vGPU that overhead seems to be even bigger. CUDA graphs are by far the biggest performance lever in my setup, so --enforce-eager is something I'd only use for debugging.

Fitting 250k tokens of context

The context length is limited by the KV cache, which needs memory for every token of every running request. After the model weights and the CUDA graphs, there's only a couple of GB left for it on a 40 GB GPU.

The trick here is --kv-cache-dtype fp8. It halves the memory per token compared to bf16:

KV cache dtype Tokens that fit in the KV cache
bf16 ~120,000
fp8 ~286,000

With bf16, 250k context isn't possible at all. With fp8 it fits with some headroom. vLLM logs the actual size at startup, so that's the line I always check after changing anything:

kubectl -n vllm logs deploy/vllm-qwen38 | grep -E "GPU KV cache size|Maximum concurrency" 

Info: The fp8 KV cache is lossy. The checkpoint doesn't ship KV scales, and vLLM logs a warning about accuracy. For my use cases (coding assistant, chat, document Q&A) I haven't noticed a difference, but if output quality matters more to you than context length, keep that in mind.

max-num-batched-tokens: bigger isn't better

--max-num-batched-tokens sets how many tokens vLLM processes in one engine step during prefill (reading the prompt). My first thought was: bigger chunks, faster prefill. So I tested it:

Value Result
4096 280,120 tokens of KV cache
8192 263,554 tokens of KV cache, prefill speed within noise (-1.9% to +1.4%)
16384 and up vLLM doesn't start anymore, the KV cache gets too small for 250k context

The bigger buffer for each step takes memory away from the KV cache, but doesn't make prefill any faster. To understand why, I sampled nvidia-smi every 100 ms during a prefill benchmark: the GPU was already at 94-99% utilization. Prefill on this model is compute-bound, so there is simply no idle capacity left that a bigger batch could fill. I went back to 4096.

Vision: first off, then on again

The Qwen3.8 checkpoint is actually a multimodal model. By default, vLLM reserves memory for the vision encoder and runs a warmup with dummy images. At 0.85 memory utilization this pushed the process to 36.6 GiB, and it crashed during startup with torch.OutOfMemoryError.

My first fix was to just run it text-only:

--language-model-only --skip-mm-profiling 

Later, after raising --gpu-memory-utilization to 0.91, I removed both flags again and vision fit. The KV cache went down from ~349k to ~275k tokens, which is still above 250k, and image requests worked end to end. The margin is thin though. If it ever stops fitting, I'll put those two flags back in, since the text-only setup was running stable before.

Prefix caching and chunked prefill

These two are on by default in recent vLLM versions, but I set them explicitly anyway, so the intent is documented and a future change of the defaults doesn't surprise me:

--enable-prefix-caching --enable-chunked-prefill 

Prefix caching reuses the KV cache of prompts that start the same way. The effect is huge:

Request Time to first token
128k token prompt, cold 73.6 s
same prompt, cached 1.65 s
same 128k prefix + 507 new tokens 2.11 s

That's 44x faster. In production I see a hit rate of around 66%, mostly from coding assistants sending the same big context again and again. The lesson for anyone using the platform: put the stable part of your prompt first (system prompt, documents, examples) and only change the end.

Chunked prefill splits a large prompt into chunks of --max-num-batched-tokens and interleaves them with the token generation of other requests. Without it, one user sending a 200k token prompt would freeze the generation for everybody else for a minute.

Info: To make prefix caching work across multiple vLLM instances, LiteLLM needs to send all requests of one conversation to the same instance. I'm doing that with LiteLLM's session_affinity pre-call check in the router_settings.

Cold starts and the compile cache

On the first start, vLLM loads the model (~60 s) and then runs torch.compile and captures the CUDA graphs (~200 s). That's more than 4 minutes before the pod is ready.

I put the compile cache onto the same PVC as the model weights:

env: - name: VLLM_CACHE_ROOT value: /models/hf/vllm-cache 

With a warm cache, the next restart is way faster. Only the first rollout after an image update pays the full price again. Still, I give the startup probe up to 20 minutes and the Deployment a progressDeadlineSeconds of 1800, just to be safe.

If vLLM behaves weirdly at startup right after a version upgrade, delete the old compile cache from the PVC. Stale cache files from a previous version can cause strange errors.

Multi-GPU on vGPU: NCCL

At some point I wanted to try tensor parallelism, so splitting one model across two vGPUs with --tensor-parallel-size 2. On vGPU this just crashed during startup in NCCL:

Cuda failure 'operation not supported' 

After quite some digging, the root cause turned out to be this: the vGPU reports CU_DEVICE_ATTRIBUTE_MEMORY_POOLS_SUPPORTED = 0, but NCCL up to 2.31 calls cudaMemPoolCreate anyway. NCCL 2.32.3 finally checks for that support first and falls back to synchronous allocations. vLLM v0.28.0 ships NCCL 2.30.7, so I built an image that only swaps the NCCL library:

FROM vllm/vllm-openai:v0.28.0 RUN pip install --no-cache-dir --no-deps --force-reinstall nvidia-nccl-cu13==2.32.3 \  && python3 -c "import ctypes; c = ctypes.c_int(); ctypes.CDLL('libnccl.so.2').ncclGetVersion(ctypes.byref(c)); assert c.value >= 23203, c.value" USER 2000 

Info: Don't trust torch.cuda.nccl.version() for this. It reports the version PyTorch was compiled against, not the library that is actually loaded at runtime. That's why I check libnccl.so.2 directly in the build.

With that image, tensor parallelism worked. I had to lower --gpu-memory-utilization to 0.88 though, since the NCCL communicators take ~0.4 GiB per GPU before vLLM does its memory check.

But was it worth it? I compared TP=2 on two vGPUs against two independent vLLM instances on the same two vGPUs:

Scenario TP=2 2 independent instances
1 request, 1k context (tok/s) 64.9 48.4
16 requests, 1k context (total tok/s) 403 487
64 requests, 1k context (total tok/s) 603 860
8 requests, 100k context (TTFT p95) 278 s 216 s

TP=2 is ~1.4x faster for a single request and has double the KV cache. But as soon as multiple people use it, two independent instances win by 20-30%. The reason is that vGPUs have no peer-to-peer connection, so every layer has to sync over host memory. Two independent instances don't need to talk to each other at all.

So for a shared platform, I stayed with one instance per vGPU and let LiteLLM balance across them. TP only makes sense for models that don't fit on one GPU at all.

My final flags

Putting it all together, this is what my vLLM container runs with today:

args: - "--model" - "Inferact/Qwen3.8-27B-NVFP4" - "--served-model-name" - "qwen3.8-27b-vllm" - "--gpu-memory-utilization" - "0.91" - "--max-model-len" - "250000" - "--max-num-seqs" - "32" - "--kv-cache-dtype" - "fp8" - "--max-num-batched-tokens" - "4096" - "--enable-prefix-caching" - "--enable-chunked-prefill" - "--enable-auto-tool-choice" - "--tool-call-parser" - "qwen3_coder" - "--reasoning-parser" - "qwen3" 

The result: 52 tok/s for a single request, and over 600 tok/s total with 15 parallel requests, with a 250k token context window, on half of an A100.

Conclusion

vLLM on vGPU works really well, but the defaults and the common advice from the internet don't always fit. Check the real free memory, keep CUDA graphs on, use the fp8 KV cache if you need long context, and measure before you raise any batch sizes. And if you need multiple GPUs for one model, make sure your NCCL version is at least 2.32.3.

In the next part of this series I'll show how I monitor all of this, so I actually see GPU utilization, time to first token and model downtimes, instead of finding out from my users.