Your scheduled job will run twice at the same time one day, and the second copy is the one that corrupts state. So build the lock before you build the model call. What follows is a from-zero tutorial in five stages: each stage ships a command you can run and a check that proves it works before you move on.

Most AI writing this week is about ambition. This one is about a job that fires at 03:00 and must not talk to itself.

Everything below is provider-agnostic. You need one always-on Linux box (a free one is fine) and one HTTP endpoint you can POST to.

Stage 1 — Lay out four directories and one exit-code contract

Your future self debugs this at 3 a.m. from a phone. Give them a filesystem that explains itself.

mkdir -p ~/worker/{bin,var/tmp,var/done,logs} cd ~/worker python3 -m venv .venv && . .venv/bin/activate pip install "requests>=2.31" 
~/worker ├── bin/job.py # the only file that touches the network ├── bin/run.sh # wrapper: env + hard timeout ├── var/tmp/ # partial writes, renamed into place ├── var/done/ # finished results └── logs/job.log 

Now the contract. Cron ignores exit codes, but you won't — and neither will the script you write in three months.

Code Meaning What the next run should do
0 finished nothing
75 (EX_TEMPFAIL) busy, or upstream failed transiently try again on the next tick
78 (EX_CONFIG) your code or config is wrong stop and page a human
124 hard timeout killed it (timeout) investigate, don't just raise the number

Verify: echo $? after a deliberate exit 78 in a one-liner. If you can't reproduce exit codes by hand, you can't read them from logs either.

Stage 2 — Take a lock the kernel releases for you

PID files go stale. kill -9 leaves them behind, and now your job refuses to run forever. flock doesn't have that problem, because the lock lives on an open file descriptor and the kernel drops it the instant the process dies.

# bin/job.py — top of file import fcntl, os, pathlib, sys, time ROOT = pathlib.Path(__file__).resolve().parent.parent LOCK = ROOT / "var" / "lock" def take_lock(): fh = open(LOCK, "w") try: fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB) except BlockingIOError: print("busy: another run holds the lock", file=sys.stderr) sys.exit(75) fh.write(f"{os.getpid()} {int(time.time())}\n") fh.flush() return fh # keep this reference alive for the whole run 

Verify, in two terminals.

# terminal 1 .venv/bin/python -c "import fcntl,time; f=open('var/lock','w'); fcntl.flock(f,fcntl.LOCK_EX); time.sleep(20)" # terminal 2 timeout 5 .venv/bin/python bin/job.py; echo "exit=$?" # expect 75, not 1 

Then the test that matters: kill -9 the first process and re-run. Exit 0. No cleanup step, no stale file to delete. That's the whole argument for flock over a PID file.

One rule: pick one lock mechanism. Don't mix flock(1) from a shell and fcntl.flock in Python on the same file unless you've actually read how they interact on your kernel.

Stage 3 — Put a hard ceiling on every run

A job that can run forever is not a scheduled job. It's a slow-motion outage on a shared machine.

#!/usr/bin/env bash # bin/run.sh set -uo pipefail cd "$(dirname "$0")/.." || exit 78 set -a; . ./.env; set +a # MODEL_ENDPOINT etc., cron has no PATH for you exec timeout -k 5 600 .venv/bin/python bin/job.py >> logs/job.log 2>&1 

timeout -k 5 600 sends TERM at ten minutes and KILL five seconds later, so a wedged requests call can't pin the box. Pair it with a per-request timeout, never a bare requests.post(...):

r = requests.post(url, json=payload, headers=headers, timeout=(5, 60)) # (connect, read) 

Verify: point MODEL_ENDPOINT at a server that accepts the connection and then says nothing. Use nc -l 9999. Your wrapper should return 124 inside ~10 minutes, and the lock should be free immediately after. If it isn't, your lock file is wrong, not your timeout.

Stage 4 — Make the write atomic and the failure boring

Half-written results are worse than missing results, because downstream code treats them as truth.

def write_result(key: str, payload: dict) -> None: tmp = ROOT / "var" / "tmp" / f"{key}.json.part" final = ROOT / "var" / "done" / f"{key}.json" final.parent.mkdir(parents=True, exist_ok=True) tmp.write_text(json.dumps(payload, sort_keys=True)) os.replace(tmp, final) # atomic on the same filesystem 

And a retry budget that distinguishes "upstream hiccup" from "I shipped a bug":

def call_model(prompt: str) -> dict: url, key = os.environ["MODEL_ENDPOINT"], os.environ.get("MODEL_API_KEY", "") for delay in (0, 2, 8): if delay: time.sleep(delay) try: r = requests.post(url, json={"input": prompt}, headers={"Authorization": f"Bearer {key}"}, timeout=(5, 60)) if r.status_code in (429, 502, 503, 504): continue r.raise_for_status() data = r.json() if "output" not in data: # contract check, not a retry  print(f"unexpected shape: {list(data)}", file=sys.stderr) sys.exit(78) return data except requests.RequestException as e: last = e print(f"giving up: {last}", file=sys.stderr) raise SystemExit(75) 

Verify: return {} from your fake server. You want exit 78 and zero retries — because retrying a wrong shape three times just wastes quota you'll want later.

Stage 5 — Schedule it, and know why you picked that scheduler

crontab -e 
*/15 * * * * $HOME/worker/bin/run.sh 
Mechanism Overlap-safe Restart on crash Log story Good fit
cron + in-process lock yes next tick append to file periodic job, box already has cron
systemd timer + oneshot service yes (one unit) Restart=on-failure journald you have systemd and want systemctl status
long-running service loop n/a yes journald job should never "finish"

Verify: run the cron line by hand twice, back to back. Second run exits 75 and the log stays one line. That's your proof the schedule can't overlap, before it ever fires on its own.

Where a free endpoint and a free box actually fit

You'll want to break this harness repeatedly while you build it: wrong shapes, 429s, a lock you forget to release. That's a bad time to be paying per call. This is the part of the tutorial where MonkeyCode is relevant — the project offers free model access to point MODEL_ENDPOINT at, and a free server option to run the timer on. Its materials also describe a token allowance (10M tokens) for the free tier; availability claims like these are exactly the kind of thing that changes, so read the current terms instead of planning around a number in a blog post. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the code above provider-agnostic anyway. Env vars aren't politeness — they're how you test the supervisor once and then swap endpoints without touching logic.

Who should not use this approach

  • You need exactly-once semantics. This gives you at-least-once with atomic outputs. Billing emails, ledger writes, and anything that must happen once need idempotency keys on the downstream service, not a wrapper script.
  • Your job outlives its interval. Raising the timeout is the wrong fix. That workload is a queue with a worker pool, not a cron job.
  • You need sub-minute scheduling. Cron won't do it; reach for a real scheduler.
  • You're about to put an SLA on a free box. Don't. Free tiers are for building and for learning the failure modes — see the note above about changing terms.

The checks, in one place

Check Command Expected
lock contention two concurrent runs 75 from the second
stale lock kill -9 then rerun 0, no cleanup
hard timeout silent fake server 124 within the window
bad shape fake server returns {} 78, no retry
no overlap run cron line twice one log line, second exits 75

Five checks, maybe twenty minutes. What's the alternative — finding out at 03:00 that two copies of your job were writing the same file? Start with the kill -9 test; it's the one that tells you whether the rest of this is real. And if you'd rather spend the afternoon on the supervisor than on provisioning, the free tier is there to be poked at.