vLLM is a fast inference and serving library for Large Language Models (LLMs). It offers integration with popular Hugging Face models, optimized CUDA kernels for NVIDIA GPUs, tensor parallelism support, and fast model execution. This guide walks through cloning the vLLM project repository and building container images for both CPU-only and GPU-powered systems, then pushing the result to a container registry. By the end, you'll have a working vLLM container image tagged and pushed to a registry, ready to deploy.

1. Prerequisites

Before you begin:

  • Deploy a GPU-enabled server with NVIDIA GPU drivers, CUDA, and Docker preinstalled.
  • Access the server using SSH.
  • Start the Docker service.

1. Start the Docker service:

$ sudo systemctl start docker 

2. Set Up the Server

1. Create a new directory to store your vLLM project files:

$ mkdir vllm-project 

2. Switch to the directory:

$ cd vllm-project 

3. Clone the vLLM project repository using Git:

$ git clone https://github.com/vllm-project/vllm/ 

4. List files and verify that a new vllm directory is available:

$ ls 

5. Switch to the vllm project directory:

$ cd vllm 

6. List the directory files and verify that the necessary Dockerfile resources are available:

$ ls 

Output:

benchmarks collect_env.py Dockerfile docs LICENSE pyproject.toml requirements-common.txt requirements-dev.txt rocm_patch vllm cmake CONTRIBUTING.md Dockerfile.cpu examples MANIFEST.in README.md requirements-cpu.txt requirements-neuron.txt setup.py CMakeLists.txt csrc Dockerfile.rocm 

The vLLM project directory includes the following Dockerfile resources:

  • Dockerfile: Contains the main vLLM library build context with support for NVIDIA GPU systems.
  • Dockerfile.cpu: Contains the vLLM build context for CPU systems.
  • Dockerfile.rocm: Contains the build context for AMD GPU systems.

Use these resources in the next sections to build a CPU or GPU system container image.

3. Build a vLLM Container Image for CPU Systems

Use Dockerfile.cpu, which contains the build context with all necessary packages and dependencies for CPU-based systems.

1. Build a new container image using Dockerfile.cpu with all files in the project working directory. Replace vllm-image with your desired image name:

$ docker build -f Dockerfile.cpu -t vllm-image . 

2. View all Docker images on the server and verify that your new vLLM image is available:

$ docker images 

Output:

REPOSITORY TAG IMAGE ID CREATED SIZE vllm-image latest 70f07e7c923f About a minute ago 3.22GB 

4. Build a vLLM Container Image for GPU Systems

The vLLM project directory contains two Dockerfile resources for building container images for GPU-powered systems. Use the main Dockerfile resource to build a new container image for GPU systems.

1. Build a new container image vllm-gpu-image using Dockerfile with all files in the project directory:

$ docker build -f Dockerfile -t vllm-gpu-image . 

2. View all Docker images on the server and verify that the new vllm-gpu-image is available:

$ docker images 

Output:

REPOSITORY TAG IMAGE ID CREATED SIZE vllm-gpu-image latest bf92416d18b4 8 hours ago 8.88GB 

To run the vLLM GPU container image, verify that your target host runs the minimum or higher CUDA version referenced in the Dockerfile, and use the --gpus all option when starting the container.

3. Verify the minimum CUDA version:

$ cat Dockerfile | grep CUDA_VERSION= 

Output:

ARG CUDA_VERSION=12.3.1 

5. Upload the vLLM Container Image to a Container Registry

1. Open your cloud provider's console and navigate to your container registry to view the registry access credentials. Copy the Registry URL, Username, and API Key.

2. Switch to your server terminal session and log in to your container registry. Replace your-registry.example.com, exampleregistry, exampleuser, registry-password with your actual registry details:

$ docker login https://your-registry.example.com/exampleregistry -u exampleuser -p registry-password 

3. Tag the vLLM container image with your desired registry tag. For example, your-registry.example.com/exampleregistry/vllm-gpu-image:

$ docker tag vllm-gpu-image your-registry.example.com/exampleregistry/vllm-gpu-image 

4. View all Docker images on the server and verify that the new tagged image is available:

$ docker images 

Output:

REPOSITORY TAG IMAGE ID CREATED SIZE vllm--gpu-image latest bf92416d18b4 8 hours ago 8.88GB your-registry.example.com/exampleregistry/vllm-gpu-image latest bf92416d18b4 8 hours ago 8.88GB 

5. Push the tagged image to your container registry:

$ docker push your-registry.example.com/exampleregistry/vllm-gpu-image 

6. Open your container registry's management panel and verify that the new repository is available.

Next Steps

  • Reference the pushed image from your orchestrator (Docker Compose, Kubernetes, etc.) to deploy vLLM at scale.
  • Benchmark inference throughput with vLLM's built-in benchmarking scripts before moving to production.
  • Automate the build with a CI pipeline so new vLLM releases are rebuilt and republished automatically.
  • Explore vLLM's tensor parallelism options for multi-GPU serving.

For the full guide with additional tips, visit the original article on Vultr Docs.