The nightly export job took 40 minutes. The team deployed at 02:15. Kubernetes sent SIGTERM, waited the default 30 seconds, then sent SIGKILL. The export stopped halfway, a partial file got uploaded, and the downstream system imported half a day of data as if it were complete.

Nobody had ever thought about what a deploy does to a job that's already running. Most teams haven't.

TL;DR

  • On deploy or scale-down, your process gets SIGTERM, then SIGKILL after a grace period (30 s by default in Kubernetes, 10 s in Docker).
  • If you don't handle SIGTERM, long jobs are cut off at a random point.
  • Handle it by stopping new work, finishing or checkpointing the current item, and exiting cleanly.
  • Make long jobs resumable and outputs atomic, so a kill is a delay rather than corruption.
  • My opinion: "don't deploy while the job runs" is not a strategy. Jobs should survive being interrupted.

The shutdown sequence 

deploy starts -> SIGTERM to your process -> grace period (terminationGracePeriodSeconds / docker stop -t) -> SIGKILL (cannot be caught)

Two common surprises:

  • Your app may not receive SIGTERM at all. If the container runs sh -c "python job.py", the shell is PID 1 and may not forward signals. Use exec form (CMD ["python", "job.py"]) or a small init such as tini.
  • Classic cron jobs on a VM are often unaffected by app deploys, but they get killed by reboots and by deploy scripts that restart everything. Check what your deploy actually does.

Step 1: catch SIGTERM and stop taking new work

Python:

import signal, sys stopping = False def on_term(signum, frame): global stopping stopping = True print("SIGTERM received, finishing current item", flush=True) signal.signal(signal.SIGTERM, on_term) for batch in batches(): if stopping: save_checkpoint(batch.start_id) sys.exit(0) process(batch) save_checkpoint(batch.end_id)

Node:

let stopping = false; process.on('SIGTERM', () => { stopping = true; }); for await (const batch of batches()) { if (stopping) { await saveCheckpoint(batch.startId); process.exit(0); } await processBatch(batch); await saveCheckpoint(batch.endId); }

The loop checks a flag between units of work. It never tries to abort in the middle of one, because that's where partial writes come from.

Step 2: give it enough time

Set the grace period to longer than your largest unit of work, not longer than the whole job:

spec: terminationGracePeriodSeconds: 120

If one batch takes 20 seconds, 120 seconds is plenty. If the job is one giant 40-minute transaction, no grace period is long enough. That's the real bug, so split it into batches.

Step 3: make the job resumable

A checkpoint is just "where did I get to":

CREATE TABLE job_checkpoints ( job_name text PRIMARY KEY, run_key text NOT NULL, -- e.g. the scheduled date last_id bigint NOT NULL, updated_at timestamptz NOT NULL DEFAULT now() );

On start, the job reads the checkpoint for today's run_key and continues from last_id. A killed run becomes "slightly late", not "half done". Processing the same row twice around the checkpoint is still possible, so per-item work needs to be idempotent. The at-least-once post covers how.

Step 4: make outputs atomic

Never write the final file or table in place:

tmp=$(mktemp /data/export.XXXXXX) generate_export > "$tmp" mv "$tmp" /data/export-$(date +%F).csv # rename is atomic on the same filesystem

For object storage, upload under a temporary key and write a small "manifest" or "_SUCCESS" marker last. Consumers only read exports that have the marker. For databases, load into a staging table and swap in one transaction.

Step 5: separate jobs from web deploys

If jobs run inside the same process as your web app, every web deploy interrupts them. Options:

  • run scheduled work in its own deployment or worker pool, deployed less often
  • trigger jobs over HTTP from an outside scheduler, so each run is a short request instead of a long-lived process inside your app (the setTimeout post compares these options)
  • on Kubernetes, use CronJob, where each run is its own pod and app deploys don't touch it

With HTTP-triggered jobs, from cron and curl or a scheduler such as delay.run, the request should hand the work to a durable worker and return quickly. A deploy then only affects the worker, which handles SIGTERM as above.

How to know a run was cut off

The scary part of killed jobs is silence. Two cheap signals:

Test it on purpose: start the job and send kill -TERM <pid> halfway through. Then run it again and check that it resumes and that the output is complete.

My take

Every scheduled job will be killed mid-run eventually: deploys, node drains, OOM kills, spot instance reclaims. The question is only whether that's a non-event or a data incident. Batches, checkpoints and atomic outputs make it a non-event, and they cost less than one incident to build.

Over to you

Do you pause deploys while important jobs run, or make jobs interruption-safe? I'd like to hear how people handle jobs that genuinely can't be split, like one huge third-party export.