A point release halved your KV cache memory. You did not choose that trade.

Two vLLM point releases, three weeks apart, changed the memory-versus-latency trade under your product. Neither required you to opt in.

A point release halved your KV cache memory. You did not choose that trade.

What actually changed

v0.32.0, 18 October 2026. GLM-5.x models began defaulting to an FP8 KV cache on SM100 hardware — described in the release notes as "halving KV memory per rank", at a 7–11% TTFT cost in the project's own PR benchmark.

v0.33.0, 2 November 2026. GLM-5.3-Flash got the same default, reported there as 9.8% higher end-to-end throughput.

Read that as an engineer rather than as a changelog reader. Somebody decided, on your behalf, that half your KV memory was worth roughly a tenth of your time-to-first-token.

That may well be the right call. For most serving setups, halving KV memory is the single most valuable thing you can do — it buys concurrency, which buys throughput, which buys unit economics. I would probably choose it too.

But I would want to have chosen it.

The rest of the sharp edges

The same two releases also shipped:

Look at the first one for a moment, because it is the most interesting and the least discussed.

max_tokens no longer being reserved from the context window means admission control that used to be pessimistic is now optimistic. Previously a request was only admitted if its input plus its maximum possible output would fit. Now it can be admitted on input alone.

In the common case this is strictly better: you stop refusing requests that would never have used their full budget, and your utilisation goes up. In the tail case the shape of your failures changes — the pressure moves from admission time to generation time. If you had capacity planning built on the old behaviour, it is now modelling something that no longer happens.

That is not a bug. It is a change in where your system is allowed to be wrong, and it arrived in a minor version.

Why this bites teams rather than individuals

Nothing above is a mistake by the vLLM project. Fast-moving inference projects should move fast, and they documented every one of these.

The failure is downstream. It looks like this:

1. A dependency bot opens a bump. Or someone runs pip install -U on a rebuild.
2. CI passes. Of course it does — correctness tests do not measure time-to-first-token, and nothing threw.
3. The deploy goes out.
4. Six weeks later someone asks why p99 latency drifted, or why a capacity model stopped predicting anything useful.

By then the change is buried under thirty other commits and nobody connects it to a line in a release note from October.

The structure that survives it

Pin exact versions in production. A version range is a promise you did not read. >=0.32 means "whatever ships next, we accept."

Read the breaking-changes section of every minor release, not only majors. In this ecosystem the minors carry the sharp edges. Semantic versioning is a convention, and inference projects are moving faster than the convention assumes.

Re-benchmark on your own traffic after any bump. A vendor benchmark is their workload, their hardware, their batch shape. It is useful evidence that the direction is right. It is not your result. Your prompts are not their prompts.

Write the default into your own config explicitly. This is the cheapest habit here and the most underrated. A value you typed into a config file cannot silently change. A default can, and will, and you will not get an email about it.

# not this - whatever the library decides this week kv_cache_dtype: null # this - a decision with your name on it kv_cache_dtype: fp8 # chosen 11/11/2026, halves KV memory,  # measured +9% TTFT on our traffic 

That comment is the point. It records that you decided, when, and what you measured — so the next person does not have to guess whether it was deliberate.

Two hard rules

A default you did not choose is still a decision you own. Your users experience the behaviour, not the provenance of the behaviour.

If a version bump can change your output, your version number is part of your output. Treat it like one: pin it, log it, and include it when you record what a system did.

Honest limitation

I am reading release notes and reasoning from them. Every figure above is the project's own, from its PR benchmarks on its hardware, and I have reproduced none of it. I am also not claiming the FP8 KV default degrades answer quality — quantising a KV cache is a precision trade and whether it shows up in your outputs is something you would have to measure on your own evaluation set. I have not.

Checkable references

• vLLM v0.32.0 release notes (18 October 2026)
• vLLM v0.33.0 release notes (2 November 2026)

One open question

Which inference defaults changed under your product in the last month — and would you know if they had?