🔧 The Problem

Spot instances are 60-90% cheaper than on-demand, and the pitch is always the same: "just make your workload fault-tolerant." Great, thanks. In practice, most teams discover their batch job, training run, or queue worker isn't actually fault-tolerant until the first 3am page from a spot reclamation mid-run.

The naive fix is restarting the job from scratch after interruption. For a 20-minute job, fine. For an 8-hour fine-tuning run or a multi-hour ETL batch, restarting from zero isn't resilience, it's just burning compute budget until you get lucky with instance availability.

This post is about the pattern that actually makes spot interruptions a non-event: a tight interruption handler plus a checkpoint cadence tuned to your workload's actual cost of re-doing work.

⏱️ The 2-Minute Window Isn't a Myth, But It's Not Enough Alone

AWS gives you a real signal: the instance metadata service exposes a termination notice roughly 2 minutes before reclamation, and EventBridge can push the same notice. Two minutes is plenty of time to flush a checkpoint if your checkpoint is already mostly written. It is nowhere near enough time to checkpoint 40GB of in-memory state from scratch.

The mental model that works:

  • Checkpoint cadence handles the common case (do this periodically regardless of interruption).
  • Interruption handler handles the urgent case (flush whatever's pending, save minimal state, exit cleanly).

If you only build the handler and skip cadence, your worst case is still "lose 2 minutes of progress times however long your last checkpoint took to trigger." If you only build cadence and skip the handler, you're gambling that your cadence happens to line up with the interruption.

🧱 The Checkpointing Pattern

Three pieces, independent of language or framework:

  1. A checkpoint writer that serializes progress state (offsets, iteration counts, model weights, partial aggregates) to durable storage (S3, EFS, EBS snapshot) outside the instance's local disk.
  2. A progress ledger — a small, cheap-to-write record of "what's been durably checkpointed" separate from the checkpoint payload itself, so on restart you know exactly where to resume without re-parsing large files.
  3. An interruption watcher running as a background thread or sidecar, polling the metadata endpoint, that triggers an emergency checkpoint + clean shutdown on notice.

Here's the watcher, the part most people skip because it feels like boilerplate until it saves your run:

python
import time
import threading
import requests

IMDS_TOKEN_URL = "http://169.254.169.254/latest/api/token"
IMDS_ACTION_URL = "http://169.254.169.254/latest/meta-data/spot/instance-action"

class SpotInterruptionWatcher:
def init(self, on_interrupt, poll_interval=5):
self.on_interrupt = on_interrupt
self.poll_interval = poll_interval
self._stop = threading.Event()

def _get_token(self): resp = requests.put( IMDS_TOKEN_URL, headers={"X-aws-ec2-metadata-token-ttl-seconds": "21600"}, timeout=2, ) resp.raise_for_status() return resp.text def _poll(self): token = self._get_token() while not self._stop.is_set(): try: resp = requests.get( IMDS_ACTION_URL, headers={"X-aws-ec2-metadata-token": token}, timeout=2, ) if resp.status_code == 200: # payload looks like: {"action": "terminate", "time": "2024-05-01T12:00:00Z"} self.on_interrupt(resp.json()) return except requests.RequestException: pass # 404 when no interruption pending; transient errors are fine to ignore time.sleep(self.poll_interval) def start(self): threading.Thread(target=self._poll, daemon=True).start() def stop(self): self._stop.set()

Wire it into your job with an emergency checkpoint callback:

python
def emergency_checkpoint(notice):
print(f"Spot interruption notice received: {notice}")
checkpointer.save(state, tag="emergency", force=True)
ledger.record(checkpointer.last_path, final=True)
worker.request_shutdown()

watcher = SpotInterruptionWatcher(on_interrupt=emergency_checkpoint)
watcher.start()

Note the force=True — your regular checkpoint logic probably has guards like "only checkpoint every N batches," and the interruption handler needs to bypass that entirely.

📊 Picking a Checkpoint Cadence

This is where most designs go wrong in one of two directions.

Too infrequent: you minimize I/O overhead but maximize replay cost. A training run checkpointing every 2 hours means an average loss of 1 hour of GPU-hours per interruption, which on multi-day runs across a spot fleet adds up to real money and real schedule risk.

Too frequent: writing a 15GB checkpoint every 60 seconds will saturate your network/EBS throughput and can actually slow down the useful work. I've seen training throughput drop 20% because checkpoint writes were competing with gradient all-reduce traffic.

A cadence formula that's held up in practice:

checkpoint_interval ≈ sqrt(2 * checkpoint_write_cost / interruption_rate)

This comes straight from the classic checkpoint-interval optimization problem (same math used for fault-tolerant HPC jobs). If your checkpoint write takes 90 seconds and your spot pool's historical interruption rate is roughly once every 6 hours, you land around an 8-9 minute interval — frequent enough to bound your loss, infrequent enough that checkpoint I/O stays under 20% of wall-clock time.

For queue workers (SQS-backed), the equivalent isn't a timer at all — it's checkpointing per message batch:

python
def process_batch(messages):
for msg in messages:
result = handle(msg)
ledger.commit(msg.receipt_handle, result) # durable, small write
sqs.delete_message_batch(Entries=[
{"Id": m.message_id, "ReceiptHandle": m.receipt_handle} for m in messages
])

Here the "checkpoint" is just not deleting the SQS message until work is durably recorded — interruption just means the message goes back to the queue after its visibility timeout, which should always be shorter than your emergency-save time (set it to 3-4 minutes, not the SQS default of 30).

🔁 Resuming Without Replaying Everything

The ledger matters as much as the checkpoint payload. On restart:

python
def resume():
last = ledger.latest()
if last is None:
return fresh_state()
state = checkpointer.load(last.path)
print(f"Resuming from step {state.step}, checkpoint age: {last.age_seconds()}s")
return state

Keep the ledger in something with read-after-write consistency and no eventual-consistency surprises — a small DynamoDB table or even a single S3 object with conditional writes works fine. Don't infer "latest checkpoint" by listing S3 prefixes and sorting by timestamp; clock skew and multi-writer races make that fragile exactly when you need it to be reliable.

🧪 Testing This Without Waiting for a Real Interruption

You don't need to wait for AWS to reclaim an instance to validate this. Two options:

  • Use aws ec2 rebalance-recommendation style testing isn't self-service, but you can simulate the metadata response locally by running a tiny Flask app on 169.254.169.254-equivalent (or just mock the URL in tests) that returns a 200 with a termination payload after N seconds.
  • AWS Fault Injection Simulator (FIS) has a native aws:ec2:send-spot-instance-interruptions action that triggers a real interruption notice on a target instance without actually waiting for market pricing to force one. This is the one to use before trusting the pattern in production — it exercises the real IMDS path, not a mock.

🙋 One Question for You

If you're running stateful spot workloads today — are you checkpointing on a timer, on a work-unit boundary, or only reactively on the interruption notice? I've found teams converge on hybrid approaches but rarely start there. What's your current cadence, and has an interruption ever actually caught you mid-write?

✅ Wrap-Up

Spot interruption handling isn't exotic — it's the same checkpoint/restart discipline from HPC and distributed training applied to cloud economics. The pattern that holds up: a background watcher polling IMDS for the real interruption signal, a cadence tuned by actual checkpoint cost versus your spot pool's interruption rate, and a ledger that makes "where do I resume" a cheap lookup instead of a guess. Do those three things and the 2-minute warning stops being a crisis and becomes just another event in your job's lifecycle.