I ran the same batch command on my laptop on Tuesday and on a remote host on Wednesday, and the two runs disagreed about which records belonged to Monday. Nothing crashed, both processes exited 0, and both wrote a summary file that looked entirely reasonable until I diffed the per-day counts. One day's worth of records had quietly landed on the wrong side of the boundary, which is the kind of bug that never pages anyone but poisons every comparison you make afterwards.

So I spent the next 48 hours turning an ad-hoc script into something that refuses to start when the environment is not what I believe it is. Below is the workflow, the three root causes I could actually reproduce on demand, the code that guards against them, and an honest list of what this approach still does not fix. If your batch job ever produced two different answers from the same input, this should look familiar.

What the job actually does

It is a small labeling pipeline, and the shape matters more than the domain:

  1. Read *.jsonl files from a data directory.
  2. Bucket every record by calendar day, using a timestamp field in the record.
  3. Assign each record to one of N shards so workers can run in parallel.
  4. Ask a model for a one-line label per record, cached by content hash.
  5. Write out/<day>/<shard>.jsonl plus a summary.json with counts.

Step 4 is the only step that costs wall-clock time rather than microseconds, so I ran the worker on MonkeyCode's free server option and used its free model access for the labeling calls, because my laptop sleeps and a container does not. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The interesting failures, though, all happened before any request left the machine.

The symptom: same input, different buckets

My first assumption was the model, because the model is the part I do not control. I re-ran the classification step with a fixed list of records and got identical labels back, which ruled it out quickly. Then I diffed the two summary.json files and noticed the totals were equal while the per-day split was not, which means the input was fine and the bucketing was not.

Why would restarting a job change which calendar day a record belongs to? That question is the whole article, and the answer is that I had written a program whose behavior depended on three ambient settings I never pinned.

Three differences I could reproduce

All three are one-liners you can verify yourself in under a minute.

1. Local time is not a property of your code

time.strftime("%Y-%m-%d") reads the process timezone, which came from the host on my laptop and from the container image on the server.

TZ=America/Los_Angeles python3 -c "import time; print(time.strftime('%Y-%m-%d %H:%M %Z'))" TZ=UTC python3 -c "import time; print(time.strftime('%Y-%m-%d %H:%M %Z'))"

Two different dates, one record, same script. Anything within the offset window flips buckets, and a nightly job that runs near midnight local time flips a whole slice of the dataset.

2. Sorting is locale-dependent

I built the file list with find ... | sort in a shell wrapper, and sort obeys LC_ALL. Under LC_ALL=C you get byte order; under a UTF-8 locale you get collation order, and the two disagree the moment uppercase and lowercase names coexist.

LC_ALL=C printf '%s\n' a A b B | sort LC_ALL=en_US.UTF-8 printf '%s\n' a A b B | sort

Different order means different batch boundaries, different resume points, and a manifest that never matches across machines.

3. hash() is salted per process

The worst one, because it looks so innocent in review. I had written shard = hash(item_id) % 8, which is stable within a process and unstable across processes, since string hashing is randomized unless PYTHONHASHSEED is fixed. Two workers on the same host could disagree about which shard owns a record, and the cache keyed on shard path then missed for no visible reason.

# do not do this shard = hash(item_id) % 8 # do this instead import hashlib def shard_for(day: str, item_id: str, shards: int = 8) -> int: key = f"{day}:{item_id}".encode("utf-8") return int(hashlib.sha256(key).hexdigest()[:8], 16) % shards

The fix: fingerprint, manifest, content-keyed cache

Three artifacts, none of them clever.

An environment fingerprint that the worker prints at startup and refuses to continue past if it differs from an expected value:

import hashlib, json, locale, os, platform, sys, time def fingerprint() -> dict: return { "python": sys.version.split()[0], "platform": platform.system(), "tz_env": os.environ.get("TZ", "<unset>"), "tz_offset_now": time.strftime("%z"), "local_now": time.strftime("%Y-%m-%dT%H:%M:%S"), "lc_all": os.environ.get("LC_ALL", "<unset>"), "lc_collate": locale.setlocale(locale.LC_COLLATE, None), "fs_encoding": sys.getfilesystemencoding(), } def digest(fp: dict) -> str: return hashlib.sha256(json.dumps(fp, sort_keys=True).encode()).hexdigest()[:12]

Run it both places. If the digests differ, stop and diff the fields instead of diffing your results an hour later. Notice that local_now is in there on purpose: a fingerprint that only captures the timezone name hides the offset change.

A file manifest written on the laptop and checked on the server, so a different file set fails loudly rather than producing a slightly different summary:

import hashlib, json, pathlib, sys def build(root: pathlib.Path) -> dict: files = {} for p in sorted(root.rglob("*.jsonl"), key=lambda q: q.as_posix()): files[p.as_posix()] = hashlib.sha256(p.read_bytes()).hexdigest()[:16] return {"file_count": len(files), "files": files} if __name__ == "__main__": got = build(pathlib.Path(sys.argv[1])) expected = json.loads(pathlib.Path(sys.argv[2]).read_text()) if expected != got: print("manifest mismatch", file=sys.stderr) sys.exit(3) print("manifest ok", got["file_count"], "files")

A cache keyed on content, not on location, so re-sharding never re-buys a label:

CREATE TABLE IF NOT EXISTS labels ( item_sha TEXT NOT NULL, model TEXT NOT NULL, label TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT (datetime('now')), PRIMARY KEY (item_sha, model) );

That last table is the piece I would keep even if I threw away everything else, because it makes the expensive step idempotent by construction and lets me re-run cheap steps freely.

The test plan I now run before leaving the laptop

A fingerprint only helps if something exercises it, so I kill the worker on purpose:

for delay in 3 7 13 29; do timeout -s KILL "$delay" python3 worker.py --manifest data.manifest.json || true python3 check_invariants.py out/ data/ || exit 1 done

And check_invariants.py asserts four things:

  • Every input content hash appears exactly once across all output shards.
  • No output row exists without a matching row in labels.
  • Re-computing the shard map twice in separate processes yields the same digest.
  • The number of model calls never exceeds the number of distinct content hashes.

If any of those fail, I have a bug I can describe in one sentence, which is worth more than a clean exit code.

What still breaks, and who should skip this

This workflow pins inputs and identifiers; it does not make model output deterministic, and it will not rescue you from a provider that silently swaps the model behind an alias. If an upstream record has no explicit timezone in its timestamp, pinning TZ=UTC only moves the ambiguity, it does not remove it. Edits that change whitespace will change content hashes and quietly miss the cache.

Do not bother with any of this if your batch is a hundred records you run once, if you already use a workflow engine that assigns deterministic run keys for you, or if you cannot pin the environment at all because you do not control the host. The fingerprint is cheap, but the discipline around it is not free.

What I would repeat

The fingerprint printout at startup, the content-keyed cache, and the deliberate kill -9 loop are the three things I would carry into the next job without hesitation, because each one converts a silent disagreement into a loud failure. The shell wrapper, the ad-hoc find | sort, and the trust in hash() are gone for good. Where do you draw the line between pinning the environment and admitting you cannot reproduce the run at all?

If you want to rehearse the loop without provisioning anything, the free server option on MonkeyCode is where I ran the kill-and-restart pass, and free model access is enough to exercise the cache path end to end.