Self-hosting open-source Large Language Models (LLMs) like Llama 3, DeepSeek, or Mistral on dedicated On-Demand cloud GPU instances gets expensive rapidly. A typical 4x GPU setup easily exceeds 2,000 USD/month on AWS or GCP.

Spot/Preemptible instances offer steep 65% to 75% discounts, but cloud providers can reclaim them with only a 30 to 120-second warning. Without an automated eviction mechanism, node preemptions cause dropped TCP connections, broken streaming completions, and severe HTTP 502/504 spikes.

In this guide, we break down a production-grade Kubernetes architecture that achieves a 0.0% request failure rate during spot node evictions while automatically falling back to On-Demand nodes when spot capacity evaporates.

💰 FinOps & Cost Breakdown

Strategy

Compute Nodes (4x GPUs)

Monthly Cost (est.)

Preemption Failure Rate

Reliability Risk

Standard On-Demand

g4dn.xlarge (AWS) / g2-standard-4 (GCP)

~$2,050 / mo

None (High Cost)

Naive Spot Instances

Spot / Preemptible equivalents

~$615 / mo

12% - 25% dropped reqs

High (HTTP 502/504 spikes)

Spot + Resilient Architecture

Spot with On-Demand Fallback

~$635 / mo

0.0% (Zero dropped reqs)

Zero-Downtime

🏗️ Architecture & Failover Mechanics

 [ Ingress / In-Cluster Service Proxy ] | +-----------------------+-----------------------+ | | v v

The 4 Pillars of Resilient Failover

Autonomous Preemption Handler (DaemonSet):
Runs strictly on nodes labeled node-type: spot. It accesses the host node name via the Kubernetes Downward API (spec.nodeName) and monitors cloud preemption metadata. Upon receiving a warning, it cordons the node and initiates an asynchronous kubectl drain.

Pod Disruption Budget (PDB):
Configured with minAvailable: 1. When the node drain fires, Kubernetes rejects simultaneous evictions. It forces sequential evacuation, keeping active inference pods alive until fallback pods pass readiness probes.

Graceful Traffic Handover (preStop Hook):
Inference pods delay shutdown using a lifecycle.preStop sleep hook (5–15 seconds). This grace period allows kube-proxy and ingress controllers to deregister the pod endpoint before the container process receives SIGTERM.

Soft Node Affinity:
Inference workloads use preferredDuringSchedulingIgnoredDuringExecution to prioritize Spot instances during standard scheduling cycles, but fall back seamlessly to On-Demand nodes when Spot capacity is exhausted.

🧪 Validating the Failover Locally with Kind

You can validate this entire failover flow locally without renting cloud GPU instances by using kind (Kubernetes in Docker).

Step 1: Provision the Multi-Node Cluster

Create a cluster with heterogeneous node labels representing Spot and On-Demand pools:

kind-config.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:

role: control-plane
role: worker labels: node-type: on-demand
role: worker labels: node-type: spot

Apply the configuration:

kind create cluster --name spot-lab --config kind-config.yaml

Step 2: Deploy Workloads & Preemption DaemonSet

Deploy the inference service, PodDisruptionBudget, and the preemption listener:

kubectl apply -f https://raw.githubusercontent.com/riyankhamdani/spot-llm-k8s/main/k8s/vllm-resilient.yaml
kubectl apply -f https://raw.githubusercontent.com/riyankhamdani/spot-llm-k8s/main/k8s/spot-handler.yaml

Step 3: Run the Chaos Test

In your first terminal, simulate continuous live inference requests from an in-cluster client locked to the on-demand worker:

kubectl run traffic-tester --rm -it --image=curlimages/curl --restart=Never \
--overrides='{"spec":{"nodeSelector":{"node-type":"on-demand"}}}' -- sh -c '
SUCCESS=0; FAILED=0;
while true; do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 2 -X POST http://vllm-service:8000/v1/completions -H "Content-Type: application/json" -d "{\"prompt\": \"Test\"}")
[ "$STATUS" -eq 200 ] && SUCCESS=$((SUCCESS+1)) || FAILED=$((FAILED+1))
echo "[$(date +%T)] Status: $STATUS | Success: $SUCCESS | Failed: $FAILED"
sleep 0.3
done'

In a second terminal, trigger a simulated spot termination signal:

POD_NAME=$(kubectl get pods -l app=spot-termination-handler -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -- touch /tmp/preempt_signal

Observation:

The DaemonSet detects the signal, immediately marks the spot worker as SchedulingDisabled, and initiates sequential pod eviction.

The PodDisruptionBudget maintains an active replica while the displaced replica spins up on the fallback on-demand node.

The traffic generator maintains continuous 200 OK responses with a 0.0% failure rate.

☁️ Production Cloud Migration

To run this in live production environments (AWS EKS or GCP GKE):

Metadata Endpoint Polling:
Replace the local file trigger in the DaemonSet with HTTP calls pointing to cloud provider metadata:

AWS IMDSv2:

TOKEN=$(curl -s -S -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/spot/instance-action

GCP Metadata:

curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/preempted

Production Image & GPU Limits:
Switch the mock inference container image to official vllm/vllm-openai:latest and declare explicit GPU limits (nvidia.com/gpu: 1).

📦 Source Code

All Kubernetes manifests, kind configuration files, and chaos testing scripts are available in the GitHub repository:

👉 github.com/riyankhamdani/spot-llm-k8s