Your scheduled job will run twice at the same time one day, and the second copy is the one that corrupts state. So build the lock before you build the model call. What follows is a from-zero tutorial in five stages: each stage ships a command you can run and a check that proves it works before you move on.
Most AI writing this week is about ambition. This one is about a job that fires at 03:00 and must not talk to itself.
Everything below is provider-agnostic. You need one always-on Linux box (a free one is fine) and one HTTP endpoint you can POST to.
Stage 1 — Lay out four directories and one exit-code contract
Your future self debugs this at 3 a.m. from a phone. Give them a filesystem that explains itself.
mkdir -p ~/worker/{bin,var/tmp,var/done,logs} cd ~/worker python3 -m venv .venv && . .venv/bin/activate pip install "requests>=2.31" ~/worker ├── bin/job.py # the only file that touches the network ├── bin/run.sh # wrapper: env + hard timeout ├── var/tmp/ # partial writes, renamed into place ├── var/done/ # finished results └── logs/job.log Now the contract. Cron ignores exit codes, but you won't — and neither will the script you write in three months.
| Code | Meaning | What the next run should do |
|---|---|---|
0 | finished | nothing |
75 (EX_TEMPFAIL) | busy, or upstream failed transiently | try again on the next tick |
78 (EX_CONFIG) | your code or config is wrong | stop and page a human |
124 | hard timeout killed it (timeout) | investigate, don't just raise the number |
Verify: echo $? after a deliberate exit 78 in a one-liner. If you can't reproduce exit codes by hand, you can't read them from logs either.
Stage 2 — Take a lock the kernel releases for you
PID files go stale. kill -9 leaves them behind, and now your job refuses to run forever. flock doesn't have that problem, because the lock lives on an open file descriptor and the kernel drops it the instant the process dies.
# bin/job.py — top of file import fcntl, os, pathlib, sys, time ROOT = pathlib.Path(__file__).resolve().parent.parent LOCK = ROOT / "var" / "lock" def take_lock(): fh = open(LOCK, "w") try: fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB) except BlockingIOError: print("busy: another run holds the lock", file=sys.stderr) sys.exit(75) fh.write(f"{os.getpid()} {int(time.time())}\n") fh.flush() return fh # keep this reference alive for the whole run Verify, in two terminals.
# terminal 1 .venv/bin/python -c "import fcntl,time; f=open('var/lock','w'); fcntl.flock(f,fcntl.LOCK_EX); time.sleep(20)" # terminal 2 timeout 5 .venv/bin/python bin/job.py; echo "exit=$?" # expect 75, not 1 Then the test that matters: kill -9 the first process and re-run. Exit 0. No cleanup step, no stale file to delete. That's the whole argument for flock over a PID file.
One rule: pick one lock mechanism. Don't mix flock(1) from a shell and fcntl.flock in Python on the same file unless you've actually read how they interact on your kernel.
Stage 3 — Put a hard ceiling on every run
A job that can run forever is not a scheduled job. It's a slow-motion outage on a shared machine.
#!/usr/bin/env bash # bin/run.sh set -uo pipefail cd "$(dirname "$0")/.." || exit 78 set -a; . ./.env; set +a # MODEL_ENDPOINT etc., cron has no PATH for you exec timeout -k 5 600 .venv/bin/python bin/job.py >> logs/job.log 2>&1 timeout -k 5 600 sends TERM at ten minutes and KILL five seconds later, so a wedged requests call can't pin the box. Pair it with a per-request timeout, never a bare requests.post(...):
r = requests.post(url, json=payload, headers=headers, timeout=(5, 60)) # (connect, read) Verify: point MODEL_ENDPOINT at a server that accepts the connection and then says nothing. Use nc -l 9999. Your wrapper should return 124 inside ~10 minutes, and the lock should be free immediately after. If it isn't, your lock file is wrong, not your timeout.
Stage 4 — Make the write atomic and the failure boring
Half-written results are worse than missing results, because downstream code treats them as truth.
def write_result(key: str, payload: dict) -> None: tmp = ROOT / "var" / "tmp" / f"{key}.json.part" final = ROOT / "var" / "done" / f"{key}.json" final.parent.mkdir(parents=True, exist_ok=True) tmp.write_text(json.dumps(payload, sort_keys=True)) os.replace(tmp, final) # atomic on the same filesystem And a retry budget that distinguishes "upstream hiccup" from "I shipped a bug":
def call_model(prompt: str) -> dict: url, key = os.environ["MODEL_ENDPOINT"], os.environ.get("MODEL_API_KEY", "") for delay in (0, 2, 8): if delay: time.sleep(delay) try: r = requests.post(url, json={"input": prompt}, headers={"Authorization": f"Bearer {key}"}, timeout=(5, 60)) if r.status_code in (429, 502, 503, 504): continue r.raise_for_status() data = r.json() if "output" not in data: # contract check, not a retry print(f"unexpected shape: {list(data)}", file=sys.stderr) sys.exit(78) return data except requests.RequestException as e: last = e print(f"giving up: {last}", file=sys.stderr) raise SystemExit(75) Verify: return {} from your fake server. You want exit 78 and zero retries — because retrying a wrong shape three times just wastes quota you'll want later.
Stage 5 — Schedule it, and know why you picked that scheduler
crontab -e */15 * * * * $HOME/worker/bin/run.sh | Mechanism | Overlap-safe | Restart on crash | Log story | Good fit |
|---|---|---|---|---|
| cron + in-process lock | yes | next tick | append to file | periodic job, box already has cron |
| systemd timer + oneshot service | yes (one unit) | Restart=on-failure | journald | you have systemd and want systemctl status |
| long-running service loop | n/a | yes | journald | job should never "finish" |
Verify: run the cron line by hand twice, back to back. Second run exits 75 and the log stays one line. That's your proof the schedule can't overlap, before it ever fires on its own.
Where a free endpoint and a free box actually fit
You'll want to break this harness repeatedly while you build it: wrong shapes, 429s, a lock you forget to release. That's a bad time to be paying per call. This is the part of the tutorial where MonkeyCode is relevant — the project offers free model access to point MODEL_ENDPOINT at, and a free server option to run the timer on. Its materials also describe a token allowance (10M tokens) for the free tier; availability claims like these are exactly the kind of thing that changes, so read the current terms instead of planning around a number in a blog post. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Keep the code above provider-agnostic anyway. Env vars aren't politeness — they're how you test the supervisor once and then swap endpoints without touching logic.
Who should not use this approach
- You need exactly-once semantics. This gives you at-least-once with atomic outputs. Billing emails, ledger writes, and anything that must happen once need idempotency keys on the downstream service, not a wrapper script.
- Your job outlives its interval. Raising the timeout is the wrong fix. That workload is a queue with a worker pool, not a cron job.
- You need sub-minute scheduling. Cron won't do it; reach for a real scheduler.
- You're about to put an SLA on a free box. Don't. Free tiers are for building and for learning the failure modes — see the note above about changing terms.
The checks, in one place
| Check | Command | Expected |
|---|---|---|
| lock contention | two concurrent runs | 75 from the second |
| stale lock | kill -9 then rerun | 0, no cleanup |
| hard timeout | silent fake server | 124 within the window |
| bad shape | fake server returns {} | 78, no retry |
| no overlap | run cron line twice | one log line, second exits 75 |
Five checks, maybe twenty minutes. What's the alternative — finding out at 03:00 that two copies of your job were writing the same file? Start with the kill -9 test; it's the one that tells you whether the rest of this is real. And if you'd rather spend the afternoon on the supervisor than on provisioning, the free tier is there to be poked at.

