llm-compressor for vLLM: The FP8/AWQ Deployment Tradeoff

Why We Evaluated llm-compressor

We brought llm-compressor into our infrastructure review because adding another GPU is often the most expensive way to solve a model-weight problem. Before changing the serving topology, we want to know whether a smaller numerical representation can preserve the behavior our application actually needs.

Quantization needs workload validation

FP8_DYNAMIC requires no calibration data, but without measured GPU results, deployment depends on validating accuracy, kernel compatibility, and serving cost.

Our interest is specifically post-training quantization, not fine-tuning. We take an existing checkpoint, change how selected weights and possibly activations are represented, save a deployment artifact, and hand that artifact to vLLM.

We need to establish the evidence boundary immediately: we did not execute GPU quantization or serving benchmarks for this review. Our available environment could not accommodate the PyTorch/model stack or provide GPU measurements. We reviewed the primary implementation examples and built the worked example below; its output is explicitly illustrative. We have no measured VRAM, accuracy, or throughput result to present as an Effloow benchmark.

That boundary matters more than a convincing-looking tokens-per-second table.

We see llm-compressor's main architectural advantage in its handoff: it saves models in compressed-tensors format for consumption by vLLM. The compression job and serving process can live in separate environments, with the saved checkpoint as the interface between them. We prefer that boundary to installing every optimization dependency inside a production server image.

We also separate three concepts that are frequently collapsed into “quantization”:

• FP8 is a numerical format. A particular scheme determines weight scaling and activation handling.
• AWQ is an activation-aware quantization algorithm, commonly used for low-bit weight-only deployment.
• GPTQ is another quantization algorithm, not another name for AWQ.

Our starting candidate is FP8_DYNAMIC: static per-channel FP8 weights with dynamic per-token activation quantization. We can follow that path without calibration data. For a calibrated low-bit comparison, we can use GPTQ; we do not label that artifact AWQ.

Our reference points are the project repository, the compression guide, and the vLLM FP8 integration guide. We use those as implementation references, not substitutes for workload measurements.

Hands-On Walkthrough: Setup, Execution & Output

We would start with a small, ungated text model and one GPU, rather than introduce MoE routing, multimodal processors, or distributed serving into the first experiment. The worked example uses Qwen/Qwen2.5-0.5B-Instruct as a practical candidate, not a model/hardware combination we have validated here.

We isolate compression from serving

For this workflow, we use pip install llmcompressor: the project name is llm-compressor, but the installation package name is llmcompressor. We would not treat those names as interchangeable without checking the package registry.

python3 -m venv .venv-compress source .venv-compress/bin/activate python -m pip install --upgrade pip python -m pip install llmcompressor datasets accelerate python -m pip check python -m pip freeze > requirements-compress.lock nvidia-smi > gpu-environment.txt deactivate python3 -m venv .venv-serve source .venv-serve/bin/activate python -m pip install --upgrade pip python -m pip install vllm python -m pip check python -m pip freeze > requirements-serve.lock deactivate 

We deliberately do not invent a tested version pair. These commands resolve packages at execution time; the lock files record the resolved versions but do not establish that they work together. Before promoting the workflow, we would rebuild from exact pins and retain the Python, CUDA, driver, GPU, model revision, and dataset revision alongside them.

We keep compression and serving in separate environments to avoid potential dependency incompatibilities between the two stacks; we have not tested a compatible version pair here.

We make the compression choice explicit

Save this as compress.py. Our default branch follows the documented QuantizationModifier FP8 flow. The optional GPTQ branch shows explicit calibration controls; it is an adaptation to validate against the installed release, not an executed compatibility claim.

import argparse import json import os from pathlib import Path from transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier parser = argparse.ArgumentParser() parser.add_argument("--mode", choices=["fp8", "gptq"], default="fp8") args = parser.parse_args() model_id = "Qwen/Qwen2.5-0.5B-Instruct" # Set an immutable Hub commit for a reproducible run. revision = os.environ.get("MODEL_REVISION", "main") output = Path("artifacts") / f"qwen2.5-0.5b-{args.mode}" tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision) model = AutoModelForCausalLM.from_pretrained( model_id, revision=revision, torch_dtype="auto", device_map="auto", ) model.eval() if args.mode == "fp8": recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"], ) oneshot(model=model, recipe=recipe) else: from datasets import load_dataset from llmcompressor.modifiers.quantization import GPTQModifier dataset = load_dataset( "json", data_files="calibration.jsonl", split="train" ) if len(dataset) < 128: raise ValueError("Provide at least 128 calibration records.") dataset = dataset.shuffle(seed=42).select(range(128)) dataset = dataset.map( lambda row: tokenizer( row["text"], truncation=True, max_length=1024, padding=False, add_special_tokens=False, ), remove_columns=dataset.column_names, ) recipe = GPTQModifier( targets="Linear", scheme="W4A16", ignore=["lm_head"], ) oneshot( model=model, recipe=recipe, dataset=dataset, num_calibration_samples=128, max_seq_length=1024, ) output.mkdir(parents=True, exist_ok=True) model.save_pretrained(str(output), save_compressed=True) tokenizer.save_pretrained(str(output)) print(json.dumps({"artifact": str(output), "mode": args.mode})) 

Our calibration file has one {"text": "..."} record per line. We require those strings to contain the actual application prompt rendering, including the chat template where applicable. That explains add_special_tokens=False: we do not want the tokenizer adding a second set of special tokens to already-rendered inputs.

We chose 128 records, a 1,024-token cap, and seed 42 as experiment controls, not established optimal settings. We would include representative retrieval passages, code, structured-output requests, and long instructions in proportion to the intended workload. Our evaluation set stays separate.

source .venv-compress/bin/activate python compress.py --mode fp8 # Optional calibrated comparison: # python compress.py --mode gptq sha256sum artifacts/qwen2.5-0.5b-fp8/config.json deactivate 

Illustrative application stdout, not captured execution output:

{"artifact": "artifacts/qwen2.5-0.5b-fp8", "mode": "fp8"} 

We would also inspect the saved configuration for quantization metadata and retain the complete compression log. A directory containing safetensors files is not sufficient evidence of a correct compressed export.

We hand the artifact to vLLM

source .venv-serve/bin/activate vllm serve "$PWD/artifacts/qwen2.5-0.5b-fp8" \ --served-model-name quant-review \ --max-model-len 2048 \ --gpu-memory-utilization 0.80 \ 2>&1 | tee serve-fp8.log 

In another terminal:

curl --fail http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "quant-review", "messages": [{"role": "user", "content": "Reply with exactly: ready"}], "temperature": 0, "max_tokens": 8 }' 

We leave quantization detection to the checkpoint metadata rather than overriding it with an unrelated serving flag. Our first acceptance gate is a clean load, an identified supported kernel path, and a valid response—not merely an HTTP listener.

What We Could Not Verify and What We Would Test

Our immediate roadblock was verification capacity. We could not run the model stack in the available environment. We therefore cannot honestly describe an observed kernel crash, memory spike, or accuracy regression. The following are the concrete deployment gates our review identified, with the workarounds we would apply.

We cannot infer kernel performance from “FP8 supported”

For NVIDIA, we use compute capability 8.9 or newer as our planning threshold for native FP8 computation, and distinguish it from weight-only W8A16 execution through FP8 Marlin on compute capability 7.5 or newer. We have not verified the exact 8.9 boundary in execution. Those are different execution paths, not equivalent performance promises.

We would capture the selected kernels at startup. For a no-error inference hang, our troubleshooting sequence would include VLLM_USE_DEEP_GEMM=0 or a supported --linear-backend cutlass setting. We would first check the installed release's help: explicitly selecting an unsupported backend can raise an error instead of falling back. MoE expert backends have separate controls.

We cannot assume every architecture follows the dense-text example

Our example targets Linear modules and excludes lm_head. That is a starting recipe for a dense text decoder, not a blanket policy for routers, vision towers, audio encoders, or hybrid attention.

For the Qwen3-30B-A3B FP8_BLOCK recipe, we also exclude gate modules matching re:.*mlp.gate$. We would inspect module names and quantization coverage for every architecture before applying that recipe. Loading through Transformers does not establish that compression mappings and vLLM kernels support the resulting artifact.

We cannot equate compressed size with compression-time memory

GPTQ needs auxiliary Hessian-related storage. The amount depends on layer dimensions and the processing strategy; we would not estimate peak memory from the final four-bit checkpoint.

For an out-of-memory compression job, we would first check the installed release's sequential loading/offloading support, the calibration length, and the host-memory requirements. Adding tensor parallelism to the serving command does not automatically shard the compression job.

We cannot substitute GPTQ evidence for AWQ evidence

We include AWQ among the supported algorithms, but have not validated an AWQ-specific runnable recipe, architecture mapping, or serving result here. We therefore keep AWQ as a separate candidate requiring its own versioned example and benchmark.

This is a meaningful limitation of the review, not a reason to rename the GPTQ branch “AWQ.”

We cannot judge accuracy from fluent output

We would score task correctness, schema validity, tool arguments, refusal behavior, and long-context retrieval separately. A model can remain fluent while changing the one identifier that matters.

We would also match tokenizer settings, chat templates, and BOS behavior across runs. For the FP8 lm_eval evaluation path, we would set add_bos_token=True because quantized models can be sensitive to BOS tokens and lm_eval does not add one by default. We would also avoid duplicate BOS tokens in application-rendered prompts.

Scale, Latency & Cost vs. Alternatives

We use 2× model-memory reduction and up to 1.6× throughput improvement only as FP8 planning estimates, not as results we measured. We would validate both on the target configuration rather than apply either multiplier to total process memory, every GPU, or our workload.

At the raw representation level, changing eligible weights from 16 bits to 8 bits halves their payload. Four-bit weights reduce that payload further. Scales, metadata, unquantized modules, temporary buffers, and KV cache prevent those ratios from describing total serving memory.

We would run two memory experiments. First, hold context limits, concurrency, and KV-cache allocation comparable to isolate weight-related savings. Second, let each configuration use the same overall GPU budget to test useful serving capacity. With a fixed gpu_memory_utilization, vLLM may use released space for more KV cache; nvidia-smi alone can therefore hide the benefit.

Our throughput sweep would include concurrency 1, 4, 16, and 32 with identical prompt and output-length distributions. We would retain output tokens/second, p50/p95 time to first token, inter-token latency, request failures, and actual generated lengths. Shorter answers must not masquerade as faster service.

For cost, we use:

Net benefit = avoided GPU-hours × GPU price per hour − compression cost − validation cost − additional operating cost 

Our hypothetical break-even case is deliberately simple: if migration costs $600 and removes one $1.20/hour GPU for 12 billable hours per day, daily savings are $14.40 and payback is approximately 42 days. These are assumptions, not measured savings.

If quantization improves a benchmark but neither reduces billable capacity nor increases useful throughput within the latency SLO, we count no realized infrastructure saving. We also include revalidation after model updates: a successful quantization job is not a permanent quality certificate.

Our Final Verdict: When to Deploy, When to Skip

Our verdict is proceed to a bounded proof of concept, not directly to production. We like the separation between offline compression and vLLM serving, and FP8_DYNAMIC is a clear first experiment because it removes calibration-data selection from that initial path.

We would deploy this when:

• We have a supported architecture and verified GPU/kernel combination.
• We can reproduce both environments and identify the exact source checkpoint.
• We have a disjoint, application-specific accuracy suite.
• We have measured a capacity or cost improvement under our latency SLO.
• We can retain the original checkpoint and roll back cleanly.

We would hold off when:

• We need an AWQ production recommendation based on this review alone.
• We cannot inspect kernel selection or reproduce the environment.
• Our bottleneck is predominantly KV cache, retrieval, or network latency.
• Small behavioral regressions carry unacceptable business risk.
• Our cost model depends on assuming headline throughput gains.

We would begin with FP8, measure it against the original checkpoint, and add calibrated low-bit candidates only if the remaining memory pressure justifies the extra validation surface. Our decision unit is not “bits per weight.” It is cost per successful request at an acceptable latency.

For adjacent infrastructure choices, explore our tools collection. For a workload-specific quantization and serving evaluation, talk with us about the deployment constraints.