Rerun Until Green Is Selection Bias: A Sequential Test for Agent-Patch Triage

A gate that reruns a red test until it goes green is not verifying anything. It is sampling, and it is sampling with a thumb on the scale. The core conclusion of this post: one green rerun is never evidence, and the honest stopping rule can be computed rather than argued. Under reasonable flake and regression rates, a sequential likelihood test says two independent greens is the line — and a single red costs you exactly one more rerun, nothing more.

The strategy below has three layers. Fixtures buy independence, property checks buy the things a rerun cannot launder, and a sequential test buys the verdict. This article is about the decision rule. The freeze mechanism is a consequence of it, not the subject.

1. The arithmetic behind "it passed on retry"

Suppose a genuine regression makes a test fail 90% of the time. Rerun it k times and ask for at least one green. That probability is 1 - 0.9^k.

Three retries wave through 27% of real regressions. If the regression only fails half the time, three retries wave through 87.5%. Those numbers are not a defect of your agent — they are the cost of a rule that stops at the first green and never names its error budget.

2. Step 1: measure the baseline failure rate first

You cannot test a hypothesis you have not parameterised. Run the target test on the untouched pre-patch tree and record the failure rate p0. Do not use the rate from the broken run; that is the thing under test.

cd baseline-worktree for i in $(seq 1 20); do python -m pytest -q tests/test_upload.py -p no:randomly >/dev/null 2>&1 done 

Record pass/fail per run, not just the exit code. Twenty runs is enough to separate p0 ≈ 0.05 from p0 ≈ 0.4; it is not enough to estimate p0 to two decimals, and you do not need that.

3. Step 2: run a sequential probability ratio test

Model each run as Bernoulli. H0 (flake) has failure rate p0. H1 (regression) has failure rate p1, the rate you require a real break to exhibit — 0.9 is a sane default. Wald's SPRT accumulates a log-likelihood ratio and stops when it crosses a bound.

# sprt_gate.py - stdlib only, Python 3.10+ import math from dataclasses import dataclass, field @dataclass class SPRT: p_flake: float = 0.30 # p0, measured on the baseline tree  p_regression: float = 0.90 # p1, minimum rate a real break must show  alpha: float = 0.05 # tolerated false "regression" verdicts  beta: float = 0.05 # tolerated false "flake" verdicts  llr: float = 0.0 trace: list = field(default_factory=list) @property def upper(self) -> float: return math.log((1 - self.beta) / self.alpha) # ln(19) ~ +2.944  @property def lower(self) -> float: return math.log(self.beta / (1 - self.alpha)) # ln(1/19) ~ -2.944  def observe(self, passed: bool) -> str: if passed: step = math.log(1 - self.p_regression) - math.log(1 - self.p_flake) else: step = math.log(self.p_regression) - math.log(self.p_flake) self.llr += step self.trace.append((passed, round(step, 3), round(self.llr, 3))) if self.llr >= self.upper: return "REGRESSION" if self.llr <= self.lower: return "FLAKE" return "CONTINUE" 

With p0 = 0.3 and p1 = 0.9, a pass contributes -1.946 and a fail contributes +1.099. Here is the trace a real gate produces:

Note what the red did: it bought one more rerun. It did not veto the patch, and it did not get ignored. That asymmetry — an expensive red, a cheap green — is the whole point, and a fixed "retry 3 times" policy cannot express it.

The robust part. For p1 = 0.9 and any p0 ≤ 0.5, a single green never reaches the lower bound (-1.946 > -2.944 at p0 = 0.3; -1.609 > -2.944 at p0 = 0.5), and two greens always do. So "two independent greens" is a defensible default that survives getting p0 roughly wrong. Push p0 toward p1 and the test loses its power — at p0 = 0.8 you need five greens. If your baseline is that flaky, stop gating and fix the test.

4. Step 3: independence is a precondition, not an assumption

The test above assumes independent runs. Two things break that, and both are fixture properties.

First, ordering. Shuffle collection and see whether the verdict survives.

# conftest.py - order probe import os, random def pytest_collection_modifyitems(config, items): random.Random(int(os.environ.get("ORDER_SEED", "0"))).shuffle(items) 

for seed in 0 1 2 3; do ORDER_SEED=$seed python -m pytest -q tests/test_upload.py | tail -1 done 

Second, carryover. A session-scoped mutable fixture, a reused cache directory, or a shared tmp_path_factory path lets run 2 inherit run 1's state. When that happens the LLR is computed over runs that are not independent draws, and the bounds mean nothing. Practical rule: if you replay the same outcome sequence twice and the SPRT verdict differs, you have a fixture problem, not a statistics problem.

5. Step 4: property checks a rerun cannot launder

Reruns only report on test outcomes. An agent patch can change what the tests are. Assert on the diff tree instead.

# properties.py import subprocess PROTECTED = ("tests/", "conftest.py", "pytest.ini") def changed_files(base="origin/main"): out = subprocess.run( ["git", "diff", "--name-only", f"{base}...HEAD"], capture_output=True, text=True, check=True).stdout return [line for line in out.splitlines() if line] def violations(files): return [f for f in files if f.startswith(PROTECTED)] 

Four properties worth enforcing before the SPRT runs at all:

1. No edits under tests/ or conftest.py unless declared in the patch description.
2. No new skip/xfail markers and no reduction in collected test count.
3. Every test that was red on the baseline still exists and still collects.
4. The frozen-freeze set is unchanged — a patch may not unfreeze a quarantined test to make its own run greener.

6. Step 5: freeze on evidence, release on evidence

When the SPRT returns FLAKE, write a ledger record. Bind the freeze to the failure signature and to the evidence, not to a date.

{ "signature": "sha256:9f2c...", "scope": ["tests/test_upload.py::test_retry_after_429"], "evidence": { "p_flake_baseline": 0.30, "p_regression": 0.90, "llr_trace": [[false, 1.099, 1.099], [true, -1.946, -0.847], [true, -1.946, -2.793], [true, -1.946, -4.739]], "decision": "FLAKE" }, "release_triggers": ["scope_file_changed", "signature_changed"] } 

Release the freeze when any file in the test's dependency set changes, when the normalised traceback hash changes, or when a replay of the recorded trace flips the verdict. A calendar expiry releases a freeze precisely when nobody is looking; a signature change releases it exactly when the underlying failure has moved.

7. Running the rerun budget

Each SPRT rerun wants a clean process. Reusing one warm interpreter between reruns quietly violates the independence precondition from step 3, and you will not see it in the pass counts.

This is where the free tier helped me prototype: I ran the rerun matrix on MonkeyCode's free server option and used its free model access to draft traceback-normalisation rules for the signature field. Both availability claims come from the operator, not from a benchmark of mine — I have not measured quotas, limits, or throughput, and I would not size a production gate on someone else's numbers. The model-drafted normalisation rules needed human review, and one of them mattered: a regex that stripped line numbers also merged two distinct bugs into a single freeze. Signature normalisation is a merge operation, and wrong merges are silent.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you adopt the SPRT gate, run it in parallel with your current rerun-once rule for a week before replacing anything. The list of patches where the two rules disagree is more informative than either rule in isolation.

8. Limits, and who should not use this

• Stationary failures only. If the failure rate drifts with load or time of day, p0 is not a constant and the bounds are fiction.
• Low-baseline suites gain little. At p0 < 0.05, the SPRT needs more runs than a developer will tolerate. Fix the test instead; the statistics are downstream of test quality.
• Not a substitute for a fix. A freeze is a debt record with a release condition. If nothing ever triggers the release condition, you have not built a quarantine — you have built a graveyard.
• Don't use it to adjudicate infra failures. A dead database is not a flake and not a regression. Route those out before the gate sees them.
• Small suites with expensive tests may not have the rerun budget. In that case use the property checks from step 5 and skip the sequential test entirely.

The transferable idea is not the specific bounds. It is that "how many reruns before I believe it" is a question with an answer you can compute and defend, and that the answer is not one.