Runtime Is a Budget: Ranking Property Checks by Detection per Second

Most property suites fail in a boring way: they grow, nobody prunes them, and the CI job that used to take four minutes quietly becomes the reason people stop running it locally. A property that never fails on a known-bad patch is decoration. A property that does fail, but costs more than the bug it catches, is debt.

So rank properties the way you rank any other expensive resource — by detections per second. This post builds a small calibration bench that does exactly that, using a mutant tree corpus as the oracle, a repeat pass as the stability check, and a freeze ledger for anything that flips.

1. The failure mode this fixes

Property checks are cheap to write and expensive to keep. An agent proposes ten invariants for a patch, all ten pass, and the suite is now ten functions heavier with no evidence that any of them can distinguish a correct patch from a broken one. That is not a testing strategy; it is a runtime tax.

Three questions decide whether a property earns its slot:

1. Does it pass on the clean tree, every time, without retries?
2. Does it fail on at least one tree that we already know is broken?
3. Is the median cost under the per-property budget the PR gate can afford?

Anything that answers "no" to (2) or (3) is a candidate for removal, not for a longer timeout.

2. Build a tiny mutant corpus

You do not need a mutation-testing framework. You need three to five directories, each containing the same small module with one deliberate defect. Keep each mutant pinned to a defect class you actually ship: off-by-one boundaries, missing input validation, wrong default.

# bench/trees/clean/ranges.py def parse_range(spec): lo, hi = spec.split("-") return range(int(lo), int(hi) + 1) 

# bench/trees/mut_exclusive/ranges.py (defect: end is exclusive) - return range(int(lo), int(hi) + 1) + return range(int(lo), int(hi))  # bench/trees/mut_no_validate/ranges.py (defect: no split guard) - lo, hi = spec.split("-") + lo, hi = spec.split("-")[0], spec.split("-")[-1] 

Now the properties. Each one takes a tree path, imports from that tree in a fresh process, and raises AssertionError on violation.

# bench/properties.py import sys def p_end_is_inclusive(tree): sys.path.insert(0, tree) from ranges import parse_range assert list(parse_range("1-3")) == [1, 2, 3], list(parse_range("1-3")) def p_single_value_range(tree): sys.path.insert(0, tree) from ranges import parse_range assert list(parse_range("4-4")) == [4] def p_returns_a_value(tree): sys.path.insert(0, tree) from ranges import parse_range assert parse_range("1-2") is not None # passes on every mutant ever written 

p_returns_a_value is deliberately a tautology. If the bench cannot flag it, the bench is broken.

3. The calibration bench

Two files. The worker runs exactly one property against exactly one tree, in its own process, so imports never leak between variants and a hang is bounded by a timeout.

# bench/worker.py import importlib.util, json, sys, time from pathlib import Path def load(name): path = Path(__file__).with_name("properties.py") spec = importlib.util.spec_from_file_location("properties", path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return getattr(mod, name) prop, tree = sys.argv[1], sys.argv[2] fn = load(prop) t0 = time.perf_counter() try: fn(tree) outcome = "pass" except AssertionError: outcome = "fail" except Exception as exc: outcome = f"error:{type(exc).__name__}" print(json.dumps({"outcome": outcome, "seconds": round(time.perf_counter() - t0, 4)})) 

The driver repeats each property on the clean tree, then runs it once against every mutant.

# bench/calibrate.py import argparse, csv, json, re, statistics, subprocess, sys from pathlib import Path ROOT = Path(__file__).resolve().parent PROPS = ROOT / "properties.py" MUTANTS = sorted(p.name for p in (ROOT / "trees").iterdir() if p.is_dir() and p.name != "clean") def run(prop, tree, timeout): cmd = [sys.executable, str(ROOT / "worker.py"), prop, str(ROOT / "trees" / tree)] try: out = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) return json.loads(out.stdout.strip().splitlines()[-1]) except subprocess.TimeoutExpired: return {"outcome": "timeout", "seconds": timeout} except (json.JSONDecodeError, IndexError): return {"outcome": "error:crash", "seconds": timeout} def classify(stability, kills, cost_ms, budget_ms): if stability < 1.0: return "QUARANTINE" # flaky: freeze with expiry, count yield as zero  if kills == 0: return "TAUTOLOGY" # passes every known-bad tree  if cost_ms > budget_ms: return "DEFER" # real detection, wrong price  return "KEEP" def main(): ap = argparse.ArgumentParser() ap.add_argument("--repeats", type=int, default=5) ap.add_argument("--budget-ms", type=float, default=2000) ap.add_argument("--timeout", type=float, default=30) a = ap.parse_args() names = re.findall(r"^def (p_\w+)\(", PROPS.read_text(), flags=re.M) rows = [] for prop in names: clean = [run(prop, "clean", a.timeout) for _ in range(a.repeats)] stability = sum(r["outcome"] == "pass" for r in clean) / a.repeats cost_ms = 1000 * statistics.median(r["seconds"] for r in clean) killed = [m for m in MUTANTS if run(prop, m, a.timeout)["outcome"] in ("fail", "timeout")] rows.append({ "property": prop, "stability": round(stability, 2), "kills": len(killed), "killed_by": ";".join(killed), "cost_ms": round(cost_ms, 1), "verdict": classify(stability, len(killed), cost_ms, a.budget_ms), }) rows.sort(key=lambda r: (r["verdict"] != "KEEP", -r["kills"], r["cost_ms"])) w = csv.DictWriter(sys.stdout, fieldnames=list(rows[0])) w.writeheader() w.writerows(rows) if __name__ == "__main__": main() 

Run it with the budget your PR gate can actually afford, not the one you wish you had:

python bench/calibrate.py --repeats 5 --budget-ms 2000 > bench/yield.csv column -s, -t < bench/yield.csv 

Sample shape of the output (illustrative — replace every number with your own run; this is a schema, not a result):

property stability kills killed_by cost_ms verdict p_end_is_inclusive 1.00 2 mut_exclusive;… __.__ KEEP p_single_value_range 1.00 1 mut_exclusive __.__ KEEP p_returns_a_value 1.00 0 - __.__ TAUTOLOGY 

4. Reading the verdicts

Four verdicts, four actions. Do not soften them into warnings.

The ranking is the deliverable. When someone wants to add an eleventh property, the first question is which of the current ten it outranks.

5. Where free model access fits — and where it does not

Generating candidate properties is the cheap half of this workflow; deciding which candidates survive is the expensive half. That asymmetry is exactly what a free model tier is good for: draft invariants from a diff, then let calibration throw most of them away.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A prompt contract that keeps proposals checkable:

• Name only symbols that appear in the patch or its imports.
• One assertion per function, no mocks, no network, no clock reads.
• Return AssertionError on violation, accept a single tree-path argument.

The constraints matter more than the model. A proposal that imports a symbol that does not exist lands as error:ImportError in the bench and gets dropped on the first pass — which is the behavior you want. Note the limitation: model-proposed properties cluster around the examples in the diff, so they rarely cover boundary classes your mutant corpus does not already contain. Calibration will not invent coverage the corpus lacks.

6. Where the free server option fits

The repeat pass is the slow, parallel, intellectually boring part: n properties × n repeats × n trees, each in a subprocess. That is a good fit for the free server option — you want the stability numbers without pinning your laptop for an hour during a review cycle. Keep the mutant corpus and the budget threshold on the same machine as the run, because a stability figure measured on different hardware than the gate is not a stability figure.

7. The freeze ledger

A QUARANTINE verdict is never a retry. Flakiness on the clean tree means the property's signal is unknown, and an unknown signal must not vote on a merge. Move it to a freeze file with an owner and an expiry date.

# bench/freezes/p_large_input_settles.yaml property: p_large_input_settles reason: fails intermittently on the clean tree; suspected timing dependence owner: repo-maintainer opened: 2026-11-09 expires: 2026-12-09 yield: 0 

Then guard it, so a freeze cannot silently become permanent:

# bench/check_freezes.py import datetime, pathlib, sys, yaml today = datetime.date.today() expired = [p.name for p in pathlib.Path("bench/freezes").glob("*.yaml") if yaml.safe_load(p.read_text())["expires"] <= today.isoformat()] sys.exit(f"expired freezes: {expired}" if expired else 0) 

Run it as a required check. An expired freeze is a decision someone skipped, and the build should say so out loud.

8. Limitations and who should not use this

• The corpus bounds the conclusion. A property marked TAUTOLOGY is only tautological with respect to your mutants. Three weak mutants will produce a confident, wrong prune.
• Cost is machine-dependent. Compare properties within one run, never across runs on different hardware.
• Timeouts are verdicts, not errors. A property that exceeds the timeout on a mutant is counted as a kill; verify by hand before you trust that row.
• This is not a coverage tool. It ranks detection per unit of runtime and says nothing about untested inputs.

Skip this if your suite runs in seconds, if you have no known-bad variants to test against, or if nobody owns the PR gate's wall-clock budget. In those cases, the ranking exercise produces a table nobody acts on.

9. What to do on Monday

Copy ranges.py into a bench/trees/clean/ directory, add two deliberately broken copies, and run the driver once. The output is small enough to read in a terminal and specific enough to argue about. That argument — which property outranks which — is the actual review, and it is the one most suites never have.

If you want to draft candidates before committing to a paid plan, the free model tier is enough to propose them; calibration, not the model, decides what survives the cut.