Six of Your Forty Tasks Should Never Have Been Scored: A Task-Admission Gate for Agent Benchmarks

Here is an illustrative setup. You assemble 40 repository tasks for a coding-agent benchmark, run three agents against them, and get 41%, 44%, and 39%. Two weeks later someone notices that six of those tasks pass on the unmodified repo — the agents were being credited for work the test suite already accepted.

That is not a model-quality finding. It is a dataset defect that survived all the way to a published number. This post is about closing that hole before the model runs, not after.

Why a single green baseline poisons the whole table

A task whose hidden tests pass on an untouched checkout contributes a free "success" to every agent. It also compresses the spread between agents, because the free success is identical for all of them.

Flaky tasks do the opposite: they add variance that looks like model variance. And tasks that quietly reach the network make your runs incomparable across days, because the environment changed and your harness didn't record it.

None of these are exotic failures. All three are cheap to detect — if you check before scoring.

Step 1 — Write the admission contract before touching the model

An admission contract is a short, fixed list of properties a task instance must hold to enter your scored set. Fix it in writing first, so you cannot quietly relax it when a favourite task gets rejected.

The gate below uses five checks. Each one answers a question about the task, not about any model.

1. A1 — Empty-patch must fail. Apply no changes; the task's verification command must return non-zero (or your harness's failure signal).
2. A2 — Known-good must pass. Apply the task's reference patch, if one exists; verification must succeed.
3. A3 — Repeatable. Run A1 k times from a clean checkout; the result must agree every time.
4. A4 — Hermetic. Repeat A1 with outbound network disabled; the result must not change.
5. A5 — Bounded. Verification must finish inside your per-task timeout on the box you will actually use.

A2 is optional in the sense that many task sets have no reference patch. It is not optional in what it buys you: without a known-good patch, you cannot distinguish "unsolvable task" from "hard task", and the two look identical in your results table.

Step 2 — Implement the empty-patch and known-good checks

The script below is illustrative and has not been executed as written. It shows the shape of the check, not a tested library. Adapt it to your harness.

# illustrative only — not executed, not tested import subprocess, time FAIL = object() # sentinel: we care about fail/pass, not the exact code  def verify(cmd: str, cwd: str, timeout: int) -> tuple[bool, float]: """Return (passed, seconds). Any non-zero exit counts as failure.""" t0 = time.perf_counter() try: p = subprocess.run(cmd, cwd=cwd, shell=True, capture_output=True, text=True, timeout=timeout) passed = (p.returncode == 0) except subprocess.TimeoutExpired: passed = False return passed, time.perf_counter() - t0 def admit(task, checkout, apply_patch, timeout=180): reasons = [] # A1: empty patch must fail  empty_passed, secs = verify(task["verify_cmd"], checkout(), timeout) if empty_passed: reasons.append("A1_baseline_green") # A5: bounded runtime  if secs > task.get("budget", timeout * 0.8): reasons.append("A5_over_budget") # A2: known-good must pass (skip if no reference patch)  if task.get("ref_patch"): d = checkout() apply_patch(d, task["ref_patch"]) good_passed, _ = verify(task["verify_cmd"], d, timeout) if not good_passed: reasons.append("A2_known_good_fails") return reasons 

Two details matter more than the code. First, run each check from a fresh checkout, not a reused directory — leftover build artifacts are the most common cause of a false green. Second, record the reason string, not just a boolean. A rejection log is the artifact you will actually reread.

Step 3 — Repeat, and report flake with its denominator

A single baseline run cannot tell you whether a task is stable. Run A1 k times, where k is small but not one — three is a reasonable starting point for a cheap suite, more if your tasks touch anything time- or port-sensitive.

Then report flake as a fraction with an explicit denominator:

flake_rate(task) = disagreeing_runs / total_runs 

A task with 1 disagreement in 3 runs and a task with 30 in 90 are not the same risk, even if both round to "sometimes flaky". Keep the raw counts in your log and set a threshold you decided in advance — for example, reject any task with a single disagreement, or reject above a stated rate. Either is defensible. Choosing the threshold after seeing which tasks it removes is not.

Step 4 — Cut the network, then re-run A1

Disable outbound access for the verification step and repeat the baseline check. If the outcome changes, the task was never hermetic and its results depend on a service you do not control.

This is the check people skip most often, because it is the one that produces inconvenient rejections. A task that clones a dependency at verify time is not a task — it is a task plus a remote endpoint's uptime.

Step 5 — Log rejections as first-class metrics

The gate's output is not one number. It is a table with denominators, and the rejection distribution is often more informative than the admission rate.

Publish the first four. If you cannot publish the fifth because it is low, say so — that limitation is part of the result.

Where a free tier genuinely fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is described by its maintainers as an open source project offering free model access and a free server option; quotas, included models, and hardware are set by the project and change, so confirm the currently published terms rather than trusting any number you read in a blog post — including this one.

Within the admission gate, two roles are honest:

• Spec linting as advice, never as a gate. Free model access is a cheap way to get a second read on ambiguous task wording — "does this prompt admit two reasonable interpretations?" Treat every answer as a suggestion that a human adjudicates. An LLM's opinion must never decide A1–A5, because the checks are mechanical and yours should stay mechanical.
• Baseline replays on the free server. The A1 repeat loop is boring, CPU-bound, and runs many times. A throwaway free box is a fine place to run it instead of burning your own laptop or shared CI minutes, provided the box matches the one you will score on closely enough that A5's timeout is meaningful.

What it does not give you: a substitute for pinned hardware when you measure latency, and a reason to believe a shared free machine is representative of your production runner. If your tasks need GPUs or long runs, check availability before designing around it.

There is also a circularity trap worth naming. If a model helps author or triage the tasks, and that same model is later scored on them, your admission step has leaked into your measurement. Keep authoring assistance and scored agents in separate lanes, and say which lane each model was in.

Controls that keep the gate from becoming decoration

Three cheap controls make the difference between a gate and a ritual.

1. Seed a known-bad task. Add a task you know fails A1. If it gets admitted, your gate is broken, not your dataset.
2. Seed a known-good task. Add a task you know passes everything. If it gets rejected, your checker is over-triggering.
3. Shuffle execution order across runs. Order-dependent failures show up as flake and are frequently misattributed to the agent.

Run these seeds every time you change the harness. A gate that silently stopped working produces exactly the kind of confident, wrong score this article exists to prevent.

What these numbers mean, and what they do not

Admission metrics describe your dataset. They say nothing about model capability. A high admission rate is not evidence that your benchmark is good — it may mean your checks are weak or your tasks are easy.

There is also a bias you should state out loud: rejecting flaky tasks systematically removes some of the hardest instances. That usually flatters every agent's score, and it can flatten the differences between them. Report the pre-gate and post-gate counts so a reader can see how much the surgery moved the table.

Who should skip this

If your suite has fewer than about ten tasks, the overhead of an admission contract probably exceeds its value — one careful manual read is cheaper. If your target is inherently non-deterministic (UI timing, third-party services, model-in-the-loop judging), A3 as written will reject nearly everything, and you need a distributional check instead of a pass/fail gate. And if you have no hidden tests at all, A1 is unimplementable regardless of how you feel about it.

A checklist to run before you publish a score

• Every admitted task failed on an empty patch, from a fresh checkout.
• Every admitted task was run more than once and its flake count is in the log.
• Network-dependent tasks were removed, not "noted".
• Rejection reason counts are published next to the scores.
• Known-bad and known-good seed tasks still behave correctly.

If you try this on a small candidate set, keep the rejection log — in my experience reading other people's benchmarks, it is usually the more interesting artifact than the score. The free tier at MonkeyCode is one place to run the baseline replays if you would rather not spend CI minutes on them; check its current published terms first and decide whether the included hardware fits your timeouts.