Stop Peeking at Run Five: A Sequential Stopping Rule for Free-Tier Agent Benches

You know the scene. Someone drops a free endpoint URL into the team channel, a forty-task slice gets run by hand, and by Thursday night there are five green checks and a message reading "this one's good, ship it."

The five green checks are not the problem. The stopping is.

Sample five tasks, stop the moment the fourth one passes, and you have an 80% endpoint. Repeat that on ten endpoints and the one you adopt is the one that got lucky early — not the one that patches your code. This post replaces that instinct with a rule you pre-register, run in a loop, and defend in a review. The whole artifact is about forty lines of Python.

What this rule decides, and what it refuses to decide

It answers exactly one question: on this frozen task pool, is this endpoint worth adopting? It returns adopt, drop, or a budget-cap fallback.

It does not estimate a pass rate with a confidence interval. It does not rank two endpoints that differ by three points, and it will not tell you which of two acceptable endpoints is "better." If you need effect sizes, you need a fixed-N design with far more tasks than you probably have.

That refusal is the feature. A rule that returns a decision can be written down before you see any results. A rule that returns a number invites you to keep sampling until the number looks right.

You are not benchmarking models in general here. You are grading fitness-for-purpose against one slice, and the slice is the argument.

Step 1 — Freeze the pool before you touch the endpoint

Write the task records down first. If you can edit the pool after seeing a failure, you do not have a bench, you have a curation exercise.

Build a pool of 60 held-out tasks, not 20. The stopping rule decides how many of the 60 you actually run, which means the pool has to be bigger than any likely verdict point — otherwise your cap is doing all the work and the sequential part is decoration.

Then shuffle once, with a seed you publish, and commit the resulting order. This kills the quiet cheat of skipping the task you suspect will fail.

Step 2 — Pin the runner so the environment is not a variable

Most "the endpoint got worse" stories are toolchain drift with a model-shaped alibi. Pin the image digest, the Python version, the test runner version, the per-task wall-clock limit, and whether the test phase gets network access.

The free server option is relevant for a boring reason: you want the harness on a machine whose image you control end to end, not on whatever a hosted sandbox hands you this week. Record the pin next to every ledger row.

# append the environment pin to the manifest, once per session printf 'image=%s\n' "$(docker inspect -f '{{.Image}}' bench-runner)" >> manifest.txt printf 'python=%s\n' "$(python3 -V 2>&1)" >> manifest.txt printf 'order_seed=%s\n' 20261107 >> manifest.txt printf 'wall_clock_s=%s\n' 300 >> manifest.txt 

If the digest changes mid-plan, stop. The plan is void, not "roughly comparable."

Step 3 — Keep invalid runs out of the numerator and the denominator

Free-tier endpoints get interrupted. Rate limits fire, connections drop, a run returns a patch that never applied cleanly.

You have three options when that happens and only one is honest: mark the attempt invalid, re-run the same task in the pre-registered order, and continue. Silently retrying until you get a completion leaks selection into your sample — you have just conditioned on the endpoint eventually cooperating.

Report invalid rate as its own number, never folded into pass rate. In practice it is often the most decision-relevant metric you produce all week: an endpoint with a strong pass rate and a 30% invalid rate is not usable for the thing you are doing, no matter how good the numerator looks.

Step 4 — Apply the rule

The rule is Wald's sequential probability ratio test on pass/fail, with a hard run cap so an undecided case cannot run forever. Save this as sprt.py.

"""sprt.py — pre-registered sequential stopping rule for a pass/fail bench.""" import math from dataclasses import dataclass @dataclass(frozen=True) class Plan: p_adopt: float = 0.60 # H0: the pass rate you would ship with  p_drop: float = 0.40 # H1: the pass rate you would walk away from  alpha: float = 0.10 # tolerated false-adopt rate  beta: float = 0.10 # tolerated false-drop rate  n_cap: int = 60 # hard ceiling = pool size  def llr(self, passes: int, fails: int) -> float: return ( passes * math.log(self.p_adopt / self.p_drop) + fails * math.log((1 - self.p_adopt) / (1 - self.p_drop)) ) def decide(self, passes: int, fails: int) -> str: n = passes + fails llr = self.llr(passes, fails) upper = math.log((1 - self.beta) / self.alpha) lower = math.log(self.beta / (1 - self.alpha)) if llr >= upper: return "adopt" if llr <= lower: return "drop" if n >= self.n_cap: # out of budget and still undecided: fall back to the point estimate  return "cap-adopt" if passes / n >= self.p_adopt else "cap-drop" return "continue" def run_plan(results, plan=None): """results: iterable of 0/1 in the pre-registered order. Yields ledger rows.""" plan = plan or Plan() passes = fails = 0 for n, ok in enumerate(results, start=1): passes += int(ok) fails += 1 - int(ok) verdict = plan.decide(passes, fails) yield { "n": n, "passes": passes, "fails": fails, "llr": round(plan.llr(passes, fails), 3), "verdict": verdict, } if verdict != "continue": return 

Here is the part that makes review meetings easier. With p_adopt=0.60 and p_drop=0.40 the two log-odds ratios are equal and opposite, so the LLR collapses to 0.4055 * (passes - fails) and both boundaries sit at ±2.197. You can verify every ledger row in a spreadsheet without running any code, and so can the reviewer who does not trust your script.

Step 5 — Calibrate the rule before you point it at anything

A stopping rule you have not checked is just a lookup table with confidence. Simulate it first. Save as calibrate.py.

import random from sprt import Plan def simulate(p_true, plan=None, trials=20_000, seed=7): plan = plan or Plan() rng = random.Random(seed) tally = {"adopt": 0, "drop": 0, "cap-adopt": 0, "cap-drop": 0} runs_used = 0 for _ in range(trials): passes = fails = 0 while True: if rng.random() < p_true: passes += 1 else: fails += 1 verdict = plan.decide(passes, fails) if verdict != "continue": tally[verdict] += 1 runs_used += passes + fails break return tally, runs_used / trials if __name__ == "__main__": for p in (0.30, 0.50, 0.70): tally, avg_runs = simulate(p) print(f"p_true={p:.2f} avg_runs={avg_runs:5.1f} {tally}") 

python calibrate.py 

Check three regimes, and check them in this order.

1. p=0.70 — above the adopt boundary. Most trials should stop with adopt, and reasonably early.
2. p=0.30 — below the drop boundary. Most trials should stop with drop.
3. p=0.50 — sitting between the boundaries, where the random walk has no drift. Most trials should exhaust the cap rather than invent a verdict.

If any of those three fails, your alpha, beta, or n_cap is wrong. Fix the rule before it touches a model, not after it tells you something inconvenient.

And do not treat my prose as evidence about the output. The script is short precisely so the numbers you trust are the ones your machine printed with your plan's settings.

The decision table

Run the pool in order and read the ledger against this.

Note what the table does not contain: a column for "how good does it feel." Every row is arithmetic you can redo by hand, which is the whole difference between a bench and a leaderboard screenshot.

Where free access actually changes the math

Cost pressure biases small benches in a predictable direction. When each run costs money you run fewer, when you run fewer you stop earlier, and stopping earlier is exactly the bias the rule above exists to remove. Free model access removes that pressure, which is the only reason it belongs in a methodology post at all.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option covers the second requirement — a runner whose image and toolchain you pin yourself, so environment drift stops masquerading as model quality. Stated plainly as a limitation: availability windows on any free tier are not something you can schedule around, so the invalid-run path from Step 3 has to be a first-class branch in your harness, not an exception handler you write at 6pm.

Who should not use this

• Fewer than ~40 tasks in the pool. Run ten tasks, call it a smoke run, and never quote a percentage from it. The rule needs a pool bigger than the verdict point to mean anything.
• Anyone distinguishing 71% from 74%. Sequential tests decide; they do not resolve small differences. You need hundreds of tasks and a different instrument.
• Anyone asking "does this endpoint emit patches at all?" One task, two minutes, no ledger. Match the rigor to the question.
• Teams that cannot pin the runner. Without a pinned environment this is a vibe check with extra paperwork, and the paperwork will make it look more trustworthy than it is.

Closing the loop

Put the plan, the order seed, the environment pin, and the ledger path in the pull request that switches the endpoint. Six weeks later, "why did we pick this one" has an answer that is not a chat screenshot.

If you want to run a pre-registered 60-task plan without paying per call, MonkeyCode's free model access and free server option cover the two inputs this workflow needs — the stopping rule, the pinned image, and the ledger are the forty lines above, and they are yours to keep either way.