Twenty tasks. That is the smallest suite I would trust to decide whether a team standardizes on a free AI coding stack — and most teams that ask me the question have zero.
Here is a pattern I keep hearing from platform leads (a composite, not one company): a free tier lands, someone runs three prompts against their own repo, two diffs look fine, and a Slack thread turns into a proposal. Then a manager asks the only question that matters — "what breaks if we're wrong?" — and nobody has a number.
So let's get the number first. Not a benchmark score from a vendor blog. A pass rate measured on your repo, with your tests as the grader.
The decision is a substitution decision, not a price decision
You are not buying tokens. You are substituting one endpoint for another in a workflow that already exists. That reframing kills most of the debate.
Ask three questions in this order:
- Can the substitute finish the tasks I actually assign? Measured as a pass rate on your own test suite, not a public leaderboard.
- What does a rejected diff cost me? Rejections are not free; they consume review attention and then get rewritten by hand.
- Can I leave? If replaying your prompt suite on another endpoint takes a week, you have a lock-in problem disguised as a free trial.
Number 1 is the gate. Numbers 2 and 3 decide the shape of the rollout.
Build the swap harness before you argue about vendors
The artifact is boring and that is the point: one script, one task file, one grader you already trust. It should run against any OpenAI-compatible base URL so that switching endpoints is an environment variable, not a rewrite.
Here is the skeleton I hand to teams. I have not run it against your repo — treat it as a starting point you adapt, not a validated tool.
#!/usr/bin/env python3 """swap_harness.py: grade any OpenAI-compatible endpoint on your own tasks. tasks.jsonl, one JSON object per line: {"id":"t01","fixture":"fixtures/t01","prompt":"...","verify":"python -m pytest -q -x tests"} Run: OPENAI_BASE_URL=... OPENAI_API_KEY=... SWAP_MODEL=... \ python swap_harness.py --suite tasks.jsonl --out results_a.json """ import argparse, json, os, re, shutil, subprocess, tempfile, time from pathlib import Path from openai import OpenAI DIFF = re.compile(r"``` (?:diff|patch)\n(.*?) ```", re.S) def ask(client, model, prompt, timeout=120): t0 = time.monotonic() r = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], temperature=0, timeout=timeout, ) return r.choices[0].message.content or "", time.monotonic() - t0 def run(cmd, cwd): p = subprocess.run(cmd, shell=True, cwd=cwd, capture_output=True, text=True) return p.returncode, (p.stdout + p.stderr)[-2000:] def one(client, model, task, workdir): ws = Path(tempfile.mkdtemp(dir=workdir)) / task["id"] shutil.copytree(task["fixture"], ws) text, secs = ask(client, model, task["prompt"]) m = DIFF.search(text) if not m: return {"id": task["id"], "applied": False, "passed": False, "secs": secs} patch = ws / "_swap.patch" patch.write_text(m.group(1)) code, log = run(f"git apply --whitespace=nowarn {patch}", ws) if code != 0: return {"id": task["id"], "applied": False, "passed": False, "secs": secs, "log": log} code, log = run(task["verify"], ws) return {"id": task["id"], "applied": True, "passed": code == 0, "secs": round(secs, 1), "log": log if code else ""} def main(): ap = argparse.ArgumentParser() ap.add_argument("--suite", required=True) ap.add_argument("--out", required=True) ap.add_argument("--runs", type=int, default=1) a = ap.parse_args() client = OpenAI(base_url=os.environ["OPENAI_BASE_URL"], api_key=os.environ["OPENAI_API_KEY"]) model = os.environ["SWAP_MODEL"] tasks = [json.loads(l) for l in open(a.suite) if l.strip()] rows = [] with tempfile.TemporaryDirectory() as tmp: for t in tasks: for _ in range(a.runs): rows.append(one(client, model, t, tmp)) passed = sum(r["passed"] for r in rows) summary = {"model": model, "n": len(rows), "pass_rate": round(passed / len(rows), 3)} json.dump({"summary": summary, "rows": rows}, open(a.out, "w"), indent=2) print(summary) if __name__ == "__main__": main() Three practical notes from people who have run variants of this:
- Use
--runs 3at temperature 0 anyway. Endpoints are nondeterministic under load; a single run measures luck. - Keep fixtures small. A 40-file fixture makes setup failures look like model failures. Two failing tests in one module is enough.
-
git applyneeds a repo. Eithergit initinside each fixture or switch topatch -p1 --dry-runfirst so you can tell "bad diff" from "bad patch application".
The output is a JSON file per endpoint. Diff them, and you have a substitution argument with a receipt.
Do the rework arithmetic out loud
Pass rate alone doesn't decide anything. Convert it into minutes per merged diff:
effective minutes per merged diff = (M + (1 − P) × R) / P
Where M is touch time on a passing diff (prompt, review, small fixes), P is the measured pass rate, and R is the minutes lost on a rejected diff before you abandon it.
Illustrative numbers — not measurements from your team: M = 12 min, R = 25 min.
| Pass rate P | Effective minutes per merged diff |
|---|---|
| 0.60 | 36.7 |
| 0.70 | 27.9 |
| 0.80 | 21.3 |
| 0.90 | 16.1 |
Now add a paid lane with M = 10 and P = 0.88: (10 + 0.12 × 25) / 0.88 = 14.8 minutes. The free lane at P = 0.80 loses by six minutes per merged diff, and no amount of "it's free" fixes that, because your reviewers are the scarce resource, not the tokens.
Run the sensitivity honestly: in that setup, there is no rework value R that makes the free lane win. It is worse on both inputs. If that is your result, say it out loud, because the free lane may still deserve a role — just not the one people were arguing for.
Where a free stack actually fits
Free model access plus a free server option (operator-supplied availability; confirm current quotas, specs, and terms yourself before you depend on them) tends to win in two places the arithmetic above misses:
- Experiment volume. Fifty throwaway probes on an unfamiliar codebase, none of which merge. Zero marginal cost matters when P is irrelevant because nothing gets reviewed.
- Procurement friction. A pilot that needs no purchase order can start Monday. That is a schedule argument, not a quality argument — and it expires the moment you ask the endpoint to produce mergeable diffs.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project offers free model access and a free server option, which makes it a reasonable first target when you want somewhere to point OPENAI_BASE_URL and start collecting your own numbers instead of reading someone else's.
The fit table I would actually sign
| Criterion | How you measure it | Rule |
|---|---|---|
| Substitution pass rate | Harness, 3 runs, your tasks | Must be within your stated delta of the incumbent, or restrict the endpoint to non-merge work |
| Rework minutes per rejection | Timer during review | If R > 30, fix your task prompts before blaming the model |
| Data class | Enumerate every file and prompt you send | No secrets, no customer PII, no production dumps. If you can't enumerate, stop |
| Loop latency | p50/p95 time to a usable diff on a 200-line change | If it breaks your inner loop, batch tasks only |
| Concurrency shape | 10 parallel agents against one endpoint | Know whether you get bursts or a queue before your sprint depends on it |
| Exit cost | Replay the full suite elsewhere | Must take under one hour, or you don't have an option, you have a dependency |
This table is a conversation tool, not objective truth. Two teams with the same numbers can reasonably decide differently, because their reviewers, data classes, and release cadence differ.
Who should not do this
- Teams with under 20 gradable tasks. You cannot measure substitution. Build tests first; adopt later.
- Regulated or customer-data workloads where you cannot enumerate what leaves the network.
- Anyone whose reviews are already the bottleneck. A free lane that raises your diff volume makes the actual constraint worse.
Add an expiry
Write down the decision, the pass rate that justified it, and a date. Re-run the harness at 30 and 90 days. Endpoints drift, your repo drifts faster, and a pass rate from last quarter is folklore.
So: which number would change your mind — a pass rate two points below the incumbent, or a rework cost that turns each rejection into a forty-minute archaeology session? Measure that one first.
