This capstone ties Week 2 together. You will run a small but realistic platform on one machine: a multi-node cluster, persistent storage, a clustered vector database, scalable inference and a Helm release, then break it on purpose to see it heal.
The architecture
Diagram: FIG 1 · Traffic enters through the k3d proxy and ingress, reaches a FastAPI orchestrator, which calls Qdrant for embeddings and vLLM for tokens. Each tier has its own storage. See the animated version.
Traffic enters through the k3d proxy on port 80 and reaches an ingress controller. The ingress sends it to the ai-gateway-service, a ClusterIP in front of a FastAPI orchestrator. The orchestrator calls the Qdrant StatefulSet (with its headless service and a dedicated PVC per pod) for embeddings, and the vLLM Deployment (with a ReadOnlyMany shared-weights volume) for tokens.
One choice to reconcile: this diagram uses an ingress controller, so do not disable Traefik for the capstone. Day 8 disabled it to make room for Envoy later, but here we use K3s's built-in Traefik so the stack works end to end.
The steps
Diagram: FIG 2 · Create the cluster, prepare storage, install the Helm chart, then verify the pods. Keep the ingress controller enabled if your architecture depends on it. See the animated version.
# 1. Create the cluster (keep the built-in Traefik this time) mkdir -p ~/ai-storage/qdrant ~/ai-storage/models k3d cluster create ai-capstone --servers 1 --agents 2 \ -p "80:80@loadbalancer" \ --volume ~/ai-storage/models:/opt/ai/models@agent:* # 2. Install the chart into its own namespace helm upgrade --install enterprise-ai ./helm-ai-stack \ --set global.namespace=ai-system --create-namespace \ -f ./helm-ai-stack/values-staging.yaml # 3. Check where everything landed kubectl get nodes -o wide kubectl get pods -n ai-system -o wide kubectl get pvc -n ai-system A note on permissions. You will often see sudo chmod -R 777 in tutorials to get a volume working. That is a lab shortcut that makes a folder writable by everyone, and you should not carry it into production. Prefer creating the folders as your own user, as above, or set the right owner and a tight mode such as 755.
If your machine has no GPU, replace the vLLM image in the chart with an Ollama-based one. The manifests keep the same shape, and you can still exercise every Kubernetes concept in the lesson.
Break it on purpose
Fault injection is how you learn whether the design holds.
kubectl delete pod -n ai-system -l app=local-ai-llm --now kubectl get pods -n ai-system -w Diagram: FIG 3 · Deleting a model pod triggers an immediate replacement. Because the weights live on a shared volume, the new pod only has to load them, so it recovers far faster than it would after a re-download. See the animated version.
The Deployment controller notices the missing pod and creates a replacement at once. Because the weights live on a ReadOnlyMany volume that is already on the nodes, the new pod does not re-download anything. It passes its startup probe as soon as the model has loaded, which is much quicker than a fresh download.
Verify the whole system
Diagram: FIG 4 · Verify nodes, pod placement, volume binding, database clustering, autoscaling and recovery. On a machine without a GPU, run a CPU-friendly engine such as Ollama in place of vLLM. See the animated version.
- Nodes show
Ready. - Pods are spread across the agents, with Qdrant and vLLM on different nodes.
- PVCs show
Bound. - Qdrant peers can see each other (check the cluster endpoint on port 6333).
- The HPA can read its metric.
- Deleting a pod recovers, and you can time how long that takes.
That completes Week 2. You now have a cluster you can create in seconds, storage that outlives pods, a database that keeps its identity, inference that scales on the right signal and a Helm chart that ships it all. Week 3 turns to the network and security layer: eBPF, Cilium and service meshes.
Run it yourself
This is the exact example from the video. Run against a real K3d cluster. A small nginx image stands in for the model server, and the video deletes one of the two pods and counts what comes back.
apiVersion: apps/v1 kind: Deployment metadata: {name: llm} spec: replicas: 2 selector: {matchLabels: {app: llm}} template: metadata: {labels: {app: llm}} spec: containers: - name: llm image: nginx:1.27-alpine readinessProbe: httpGet: {path: /, port: 80} periodSeconds: 1 Run it:
kubectl apply -f deploy.yaml kubectl delete pod <one of the two pods> kubectl get pods Output I got while recording the video:
before: 2/2 ready deleted: 1 pod after: 2/2 ready new pods created: 1 deleted pod gone: yes Coming Up Next
Day 15: AI network bottlenecks: moving from iptables to eBPF.
Kubernetes #MLOps #SystemDesign #CapStone #CloudNative #DevOps #DataScience #AIEngineering
Originally published at https://sureshpallapothu.in/blog/ai-int-day-14-week-2-capstone, where this post includes animated diagrams.
