Stop Writing Test Strategies for Agent Patches. Write a Contract Your CI Can Enforce.

A test strategy for agent patches dies the moment it meets a real diff. It lives in a wiki page, the agent never reads it, and the reviewer under time pressure approves a green build. The fix is not a better document. It is a small machine-readable contract that ships in the patch and a linter that rejects the patch when the contract is thin.

Below is the workflow I use: one JSON contract per patch, four linter rules, and a blast-radius table that decides how much checking the patch has earned. The linter is a reference implementation, stdlib only, roughly 60 lines. Run it before you trust it.

Why strategies decay and contracts do not

A strategy says "add property tests where appropriate." Appropriate is a judgment call, and an agent optimizes for closing the ticket. A contract says property_checks must be non-empty and each id must appear as a marker in an added test line. That is a string comparison, and string comparisons do not get tired.

The trade-off is real: contracts encode less than prose. You lose nuance about design intent. You gain a gate that fires without a human in the loop, which is exactly what an agent-generated patch needs before a human looks at it.

1. The contract file

Keep it in the patch, next to the code. JSON because the linter should not drag in a YAML dependency; convert if your repo already ships one.

{ "patch": "fix-retry-backoff", "owner": "finley", "blast_radius": "medium", "property_checks": [ { "id": "backoff-monotonic", "invariant": "delay(n+1) >= delay(n) for all n in [0, 8)", "seeds": [0, 7, 42] } ], "fixtures_locked": ["tests/fixtures/clock_frozen.json"], "flaky_frozen": ["tests/test_upload.py::test_eventual_consistency"], "frozen_until": "2026-11-21", "override": null } 

Three fields carry the weight. property_checks[].invariant is prose the linter cannot verify, so a human must own that line — not the agent that wrote the patch. seeds makes the property run reproducible; an unseeded random loop is an unrepeatable test with extra steps. frozen_until gives the freeze an expiry date, so the contract degrades into a loud failure instead of permanent debt.

2. Four rules the linter enforces

Reference implementation, Python 3.11+, no third-party imports:

#!/usr/bin/env python3 """tools/contract_lint.py - reference implementation, stdlib only.""" import json, re, subprocess, sys, pathlib, argparse ROOT = pathlib.Path(__file__).resolve().parents[1] def git(*args): return subprocess.run(["git", *args], cwd=ROOT, capture_output=True, text=True, check=True).stdout def changed_files(base): return [p for p in git("diff", "--name-only", f"{base}...HEAD").splitlines() if p] def added_lines(base): out = {} for path in changed_files(base): diff = git("diff", "-U0", f"{base}...HEAD", "--", path) out[path] = [l[1:] for l in diff.splitlines() if l.startswith("+") and not l.startswith("+++")] return out def rule_frozen_touch(c, changed, today): frozen_files = {t.split("::")[0] for t in c.get("flaky_frozen", [])} guarded = frozen_files | set(c.get("fixtures_locked", [])) ov = c.get("override") or {} valid = bool(ov.get("reason")) and ov.get("expires", "1970-01-01") >= today return [f"frozen-touch: {p} is frozen or locked without a valid override" for p in changed if p in guarded and not valid] MARKER = re.compile(r"#\s*property:\s*([\w.-]+)") def rule_properties(c, added): seen = {m.group(1) for p, lines in added.items() if p.startswith("tests/") for line in lines for m in [MARKER.search(line)] if m} problems = [] for pc in c.get("property_checks", []): if pc["id"] not in seen: problems.append(f"property-missing: no added test tagged '# property: {pc['id']}'") if not pc.get("seeds"): problems.append(f"property-unseeded: {pc['id']} declares no seeds") return problems LIT = re.compile(r'"(?:[^"\\]|\\.)*"|-?\d+(?:\.\d+)?') ASSERT = re.compile(r"assert\s+.*==\s*(\"(?:[^\"\\]|\\.)*\"|-?\d+(?:\.\d+)?)") def rule_mirror(c, added): """Heuristic: a test literal copied out of the patch proves nothing.""" src = {lit for p, lines in added.items() if not p.startswith("tests/") for line in lines for lit in LIT.findall(line)} problems = [] for p, lines in added.items(): if not p.startswith("tests/"): continue for line in lines: if "literal-ok:" in line: continue for lit in ASSERT.findall(line): if lit in src: problems.append(f"mirror-suspect: {p} asserts {lit}, also added in src") return problems def rule_quarantine_leak(c, added): frozen = {t.split("::")[0] for t in c.get("flaky_frozen", [])} stems = {pathlib.Path(f).stem for f in frozen} problems = [] for p, lines in added.items(): if p in frozen: continue for line in lines: if "import" in line or "subprocess" in line: problems += [f"quarantine-leak: {p} pulls in frozen module {s}" for s in stems if re.search(rf"\b{re.escape(s)}\b", line)] return problems def main(): ap = argparse.ArgumentParser() ap.add_argument("--base", default="origin/main") ap.add_argument("--contract", default="patch.contract.json") ap.add_argument("--today", default=None) a = ap.parse_args() import datetime today = a.today or datetime.date.today().isoformat() c = json.loads((ROOT / a.contract).read_text()) changed, added = changed_files(a.base), added_lines(a.base) found = (rule_frozen_touch(c, changed, today) + rule_properties(c, added) + rule_mirror(c, added) + rule_quarantine_leak(c, added)) if c.get("frozen_until", "9999-12-31") < today: found.insert(0, f"contract-expired: frozen_until {c['frozen_until']} has passed") strict = c.get("blast_radius") in {"medium", "high"} blocking = [f for f in found if strict or not f.startswith("mirror-suspect")] print("\n".join(found) or "contract ok") return 1 if blocking else 0 if __name__ == "__main__": sys.exit(main()) 

Run it as a pre-CI step:

python tools/contract_lint.py --base origin/main --contract patch.contract.json 

The four rules, and what each one actually buys you:

1. frozen-touch — the patch cannot edit a frozen test or a locked fixture unless the override carries a reason and a future expiry. This is the freeze from your flaky-test policy, made physical instead of advisory.
2. property-missing / property-unseeded — every declared invariant needs an added test line tagged # property: <id>, and a seed list. The tag is cheap to write and impossible to fake with a passing run.
3. mirror-suspect — advisory only. If a new assertion compares against a literal that the patch itself introduced, the test is probably restating the implementation. Expect false positives; annotate the honest ones with # literal-ok: and keep the rule noisy rather than silent.
4. quarantine-leak — nothing outside the frozen file may import its module, so a quarantined test cannot become an accidental dependency of a passing one.

A minimal property test that satisfies rule 2:

# tests/properties/test_backoff.py import random from src.retry import backoff_delays # property: backoff-monotonic def test_backoff_is_monotonic(): for seed in (0, 7, 42): rng = random.Random(seed) for _ in range(200): base, attempts = rng.randint(50, 5000), rng.randint(1, 8) d = backoff_delays(base, attempts) assert all(b <= a for a, b in zip(d, d[1:])), (seed, base, attempts, d) 

3. Where free model access and a free server fit

Two parts of this workflow are tedious but low-judgment, and one part is CPU-bound.

Drafting the contract. Writing invariant strings from a diff is a summarization task. Paste the changed function signatures plus git diff --stat into a free model and ask for a candidate contract, then edit the invariant text by hand. Do not skip the edit: an agent that authored the patch should never author the sentence that certifies the patch. That is self-certification with extra steps.

Running the seeded loop. Seed sweeps multiply fast — 3 seeds × 200 iterations × several properties over a shared module. That is the kind of parallel, disposable work the free server option is useful for, so CI minutes stay for the suites that need your real environment. Both availability claims here are operator-supplied; confirm current terms and limits yourself before depending on them, and do not assume anything about hardware, quotas, or persistence.

What not to hand to a model. The linter itself. It is deterministic, fast, and cheap, and a model reviewing a contract it just helped write adds a failure mode rather than removing one. Keep the gate boring.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

4. Sizing the contract by blast radius

Not every patch deserves the same contract. These are policy defaults to start from, not measured detection rates — tune them against your own merged history.

The point of the table is to stop the argument at review time. Either the contract matches the tier, or the patch waits.

Limitations, and who should skip this

• The diff parser is fragile. Renames, binary files, and merge commits will produce wrong changed_files output. Add --find-renames handling or fail closed before you rely on it.
• mirror-suspect is a heuristic with a high false-positive rate. It is a prompt for the reviewer, not a proof of a tautological test.
• Frozen lists rot. If frozen_until keeps getting bumped, the contract is documenting debt, not paying it down.
• Self-authored contracts are the main failure mode. Require a human owner on property_checks text, even if the model drafted it.

Skip this if you are a solo maintainer on a small repo — the contract is more overhead than the diff. Skip it too if your default branch has no stable merge base for origin/main...HEAD, or if a monorepo makes the changed-file list too large to review. The workflow assumes a bounded patch and a shared branch.

Before adding a fifth rule, run this linter against your last five merged agent patches. The violations you find on already-approved work tell you more about your gate than any new rule will.