A basic Deployment suits a stateless web server. Model serving and vector stores add failure domains that web apps rarely have: memory-exhaustion crashes, cold-boot timeouts and broken load-balancing rings. This lesson covers the Kubernetes building blocks that handle them.
Deployments and Services for inference
An inference pod ingests context, generates tokens and discards its KV-cache. It is stateless, so a Deployment gives you horizontal scaling, rolling updates and automatic restarts. A ClusterIP Service puts one stable address in front of the pods and sends traffic only to those that pass their readiness probe, so a saturated pod can step out of rotation without being killed.
Diagram: FIG 1 · The Service sends requests only to pods whose readiness probe passes. A saturated pod is skipped until it recovers. Tune the probe so it does not remove every pod at once. See the animated version.
Headless Services for peers
A normal Service gives one virtual IP. A headless Service (clusterIP: None) makes CoreDNS return one A record per pod, such as pod-0.db-svc and pod-1.db-svc. Distributed databases such as Qdrant and Milvus use that to find their peers.
Diagram: FIG 2 · A normal Service gives one virtual IP and load-balances. A headless Service (clusterIP None) makes DNS return each pod's own address, which database peers use to discover each other. See the animated version.
A model-serving manifest
apiVersion: apps/v1 kind: Deployment metadata: name: inference-engine-gemma namespace: ai-platform spec: replicas: 2 strategy: type: RollingUpdate rollingUpdate: { maxSurge: 1, maxUnavailable: 0 } selector: matchLabels: { app: inference-engine-gemma } template: metadata: labels: { app: inference-engine-gemma } spec: containers: - name: vllm image: vllm/vllm-openai:v0.5.0 # pin a tested tag, not "latest" args: - --model=/models/gemma-2-2b-it-awq - --quantization=awq - --max-model-len=4096 - --gpu-memory-utilization=0.85 ports: [{ name: http-api, containerPort: 8000 }] resources: requests: { cpu: "2000m", memory: 4Gi } limits: { cpu: "4000m", memory: 8Gi } # add nvidia.com/gpu: 1 on GPU nodes startupProbe: httpGet: { path: /health, port: http-api } initialDelaySeconds: 15 periodSeconds: 10 failureThreshold: 18 # 15 + 18 x 10 = about 3 minutes to load readinessProbe: httpGet: { path: /v1/models, port: http-api } periodSeconds: 5 timeoutSeconds: 2 livenessProbe: httpGet: { path: /health, port: http-api } periodSeconds: 15 failureThreshold: 3 volumeMounts: - { name: model-weights, mountPath: /models, readOnly: true } volumes: - name: model-weights persistentVolumeClaim: { claimName: shared-model-weights-pvc } --- apiVersion: v1 kind: Service metadata: name: inference-engine-service namespace: ai-platform spec: type: ClusterIP selector: { app: inference-engine-gemma } ports: [{ port: 8000, targetPort: http-api }] Treat this as a structure to adapt. vLLM needs a GPU in practice, so on GPU nodes you would also request nvidia.com/gpu: 1 and run the NVIDIA device plugin, and the model files must exist on the volume you mount (Day 10).
Why the startupProbe is critical
Inference runtimes read gigabytes of weights before they open a network socket. If you rely on a livenessProbe alone, Kubernetes sees a silent port, concludes the pod is dead and restarts it, over and over. The startupProbe holds off the other probes until the server first passes, and here it allows about 3 minutes.
Diagram: FIG 3 · Model servers read gigabytes of weights before opening a socket. A startupProbe holds off the liveness check during that window, so Kubernetes does not kill the pod mid-load. See the animated version.
After startup, the readinessProbe decides whether the pod receives traffic, and the livenessProbe catches real hangs.
Rolling updates without a capacity dip
With maxSurge: 1 and maxUnavailable: 0, Kubernetes starts a new pod, waits for it to become ready, and only then retires an old one.
Diagram: FIG 4 · With maxUnavailable set to 0, Kubernetes starts a new pod, waits for its readiness check, then retires an old one, so serving capacity never dips during an update. See the animated version.
Together these settings turn a fragile model process into a service that restarts, scales and updates without dropping requests.
Run it yourself
This is the exact example from the video. Run against a real K3d cluster. A small nginx image stands in for the vLLM image so the pods start in seconds. The video wraps the command in a throwaway namespace and cleans up afterwards.
apiVersion: apps/v1 kind: Deployment metadata: {name: vllm} spec: replicas: 2 selector: {matchLabels: {app: vllm}} strategy: {rollingUpdate: {maxSurge: 1, maxUnavailable: 0}} template: metadata: {labels: {app: vllm}} spec: containers: - name: vllm image: nginx:1.27-alpine startupProbe: httpGet: {path: /, port: 80} periodSeconds: 2 failureThreshold: 30 Run it:
kubectl apply -f deploy.yaml Output I got while recording the video:
vllm ready: 2/2 startup budget: 60 s rolling update: done ready pods: 2 revisions: 2 Coming Up Next
Day 10: Persistent volumes (PV and PVC): managing heavy model weights and vector stores.
Kubernetes #K8sDeployments #MLOps #SystemDesign #vLLM #PlatformEngineering #HighAvailability
Originally published at https://sureshpallapothu.in/blog/ai-int-day-9-kubernetes-core-for-ml, where this post includes animated diagrams.
