The Take-Home That Must Run Twice

Here is a scenario I use to explain the problem. It is hypothetical, not a report from a real hiring loop. You send out a take-home: given twenty support tickets, return a triage label and a one-line justification for each, using a model. Six candidates send back repos. You have three reviewers and one Tuesday. Nobody re-runs anything.

Not because they are lazy. Because every re-run costs money, takes a couple of minutes, and produces slightly different text than the last one. So the review collapses into reading code, and reading code is the weakest signal on the table. A candidate can write a beautiful README around a function that would throw on the first real ticket.

The fix is not a stricter rubric. It is a smaller requirement: the artifact must be runnable twice, offline, with identical output.

The prompt

Ask for one thing and be exact about it. The service reads a fixed input file, writes a fixed output file, and supports two modes.

./run.sh replay # no network. Deterministic. Reads recorded model responses. ./run.sh record # network allowed. Calls a live model. Writes recorded responses. 

Say plainly that replay is the mode you will grade, that it must not touch the network, and that two consecutive runs must produce byte-identical out/answers.json. Say that record exists so the candidate can regenerate fixtures, and so a reviewer can spot-check that the fixtures were not hand-written fiction.

Notice what this does to the task. It stops being "build an LLM feature" and becomes "build an LLM feature whose expensive, non-deterministic part sits behind a boundary you can name." That boundary is the actual skill you are testing, and it is the one thing a reviewer can verify in thirty seconds.

Where a free endpoint belongs in this

A two-mode task only works if record is cheap enough to be casual. If recording twenty fixtures requires a paid API key and a billing account, candidates will fake the fixtures and reviewers will skip record entirely.

This is where MonkeyCode is relevant to the workflow. It is an open-source project that offers free model access and a free server option, which means both sides of the loop can run the real path without a purchase order: the candidate to record fixtures, the reviewer to re-record and compare. Treat the availability figures as time-sensitive — the project currently advertises a free token allowance on the order of ten million tokens plus a free server option, and you should confirm the current numbers on the official page before you paste anything into a rubric or a job description.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

An illustrative sample solution

Everything below is illustrative. It was not executed for this article, and it is a sketch of the shape rather than a library. Pretend the file is runner.py.

# runner.py — ILLUSTRATIVE, untested in this article import hashlib, json, os, sys from pathlib import Path CASSETTES = Path("cassettes") MODEL = "<model-id-from-your-provider>" def key_for(prompt: str, params: dict) -> str: blob = json.dumps({"model": MODEL, "prompt": prompt, "params": params}, sort_keys=True) return hashlib.sha256(blob.encode()).hexdigest()[:16] class CassetteMissing(RuntimeError): pass class ReplayClient: """Never opens a socket. Missing cassette is a hard failure.""" def complete(self, prompt: str, params: dict) -> str: path = CASSETTES / f"{key_for(prompt, params)}.json" if not path.exists(): raise CassetteMissing(f"no fixture for {path.name}; run ./run.sh record") return json.loads(path.read_text())["text"] class LiveClient: def __init__(self, endpoint: str, api_key: str, max_calls: int = 40): self.endpoint, self.api_key = endpoint, api_key self.calls, self.max_calls = 0, max_calls def complete(self, prompt: str, params: dict) -> str: self.calls += 1 if self.calls > self.max_calls: raise RuntimeError(f"budget cap hit at {self.max_calls} calls") # provider call omitted; see the project docs for the real client  ... 

And the part candidates usually skip: keeping volatile data out of the graded file.

# main() — ILLUSTRATIVE def run(client, tickets, mode): rows = [] for t in tickets: answer = client.complete(PROMPT.format(ticket=t["body"]), {"temperature": 0}) rows.append({"id": t["id"], "label": parse_label(answer)}) Path("out").mkdir(exist_ok=True) with open("out/answers.json", "w", encoding="utf-8") as f: json.dump(rows, f, sort_keys=True, indent=2, ensure_ascii=False) f.write("\n") # timestamps, latency, token counts, model id: a separate file  with open("out/run.log", "a", encoding="utf-8") as f: f.write(json.dumps({"mode": mode, "calls": getattr(client, "calls", 0)}) + "\n") 

Now the check you actually run:

./run.sh replay && cp out/answers.json /tmp/first.json ./run.sh replay && diff -u /tmp/first.json out/answers.json && echo REPLAY_OK 

Two things to be honest about. temperature: 0 is not a determinism guarantee from any provider, and the sketch does not claim one — determinism comes from replay, not from the model. And record is expected to differ run to run; that is why it is not the graded mode.

The rubric I would use

The weights are a suggestion, not a measurement. What matters is that every line is checkable by a reviewer who did not write the code and does not want to read all of it.

How it fails

A timestamp inside the answer object. datetime.utcnow() in the row, and diff lights up on the second run. Candidates fix it by sorting keys, which fixes nothing.

A random identifier. uuid4() for each ticket works right up until the artifact is compared, then produces a two-line diff that looks like a substantive bug.

A live call hiding in the replay path. Usually a fallback: except CassetteMissing: return live_client.complete(...). It reads as resilience. It destroys the property you are grading, and it is invisible unless you run replay with networking off.

Fixtures that lie. Someone hand-edits a cassette so the happy path looks better. This is why record must be present and why a reviewer should re-record one fixture and compare — a cheap spot check that catches the whole class.

Unordered collections. A set of labels serialized directly, or a dict built from asyncio.gather completion order. Deterministic on one machine, not the next.

Who should not use this

The approach costs the candidate real time — recording fixtures is not free, and it is not the fun part. If your take-home is already at the edge of reasonable, adding a replay harness makes it unreasonable, and you will lose good people to the length rather than the difficulty.

Skip it if you are grading a design document, if you have one reviewer and no re-run budget of any kind, or if the thing you actually want to observe is how a candidate handles live flakiness — there, indeterminism is the subject and you should grade it directly instead of fencing it off.

And a free tier is not a production promise. Do not build a load test, a latency benchmark, or a cohort-wide automated grader on top of a free allowance and assume it holds; check the terms, cap your own usage, and keep the offline path as the thing you depend on.

If you want to try the shape this week, record five fixtures instead of twenty. That is enough to find out whether your rubric survives contact with a second run — which is the only run that tells you anything.