“The tests pass” is an incomplete answer to “did this fix the bug?” An agent can change the implementation and the tests together, then hand you a green suite that never exercises the original failure.

For a compatible bug-fix regression test, ask for a comparison: run the same check against the old code and the proposed fix. The old code should fail for the behavior you meant to repair. The candidate should pass without moving the acceptance target.

Define the behavior before reviewing the fix

Consider a hypothetical amount field that incorrectly accepts blank input as zero. The requirement is small: reject blank input, and keep accepting a valid decimal amount such as 12.50.

That second condition matters. “Reject everything” would satisfy a badly written blank-input test while breaking the field.

Review those expectations before accepting generated implementation or test changes. Decide what rejection means at the boundary you're testing: a validation result, for example, rather than an unrelated exception. Keep the test on an existing interface both revisions expose.

The exercise below is proposed, not executed. It doesn't prescribe a validation library or how the application represents money.

Hold the regression test constant

Run the reviewed regression test against each revision's application code, holding its content and relevant fixtures constant. Running the base revision's old suite is a different check; it cannot tell you whether the new test catches the defect.

Check Expected on base Expected on candidate
Blank-input regression Assertion fails because blank input is accepted. Assertion passes because blank input is rejected.
Valid-amount control 12.50 remains accepted. 12.50 remains accepted.

These are expected outcomes, not recorded results.

Inspect the base failure. A missing import, absent new API, broken fixture or runner crash is not the red result you wanted. The check needs to reach the behavior and fail at the relevant assertion. A skipped test supplies no evidence either.

If both revisions pass the blank-input test, it hasn't demonstrated this regression. Check the fixture, execution path and assertion before celebrating.

This rule is narrow. Ordinary coverage tests may correctly pass on both revisions. A new feature may have no compatible base interface; use a separately reviewed control or record that the base comparison was not assessed. Don't manufacture a red run with a setup error.

Review changes to the checker

Test changes deserve their own review. Look for removed assertions, newly skipped cases, fixtures that avoid the failing input, suppressed type checks and runner configuration that stops selecting the relevant test.

Those changes aren't automatically dishonest. Requirements change; obsolete tests need replacing. But an acceptance change needs an explicit reason and review, followed by a fresh comparison against the approved target.

trustbut takes this approach in its documented workflow: rerun tests in a clean checkout, try changed tests against pre-change code, and flag patterns such as skipped tests and removed assertions. Its structured receipt can make verification easier to inspect without asking another model to judge the answer.

That's a useful workflow to evaluate, not evidence that the tool catches every way a test can become weaker.

Borrow the separation from formal verification

AlphaProof Code separates the target from the generated solution at a different level. Its workflow uses approved Lean specifications, proof checking and a comparator against a held-out specification, including dependent definitions.

The research concerns selected security properties of compiled libsignal in a CLI harness, with trusted machine semantics and cryptographic assumptions. It does not establish that all of Signal—or arbitrary software—is bug-free.

An ordinary regression test does much less: it observes selected cases. A kernel-checked theorem establishes a specified property under its formal model and assumptions. Neither process automatically makes the chosen requirement correct.

The practical lesson survives that gap: review the claim separately, then check the proposed solution against it. Another model agreeing with the implementation doesn't create that separation by itself.

A clean checkout still needs a controlled environment

A fresh worktree identifies source state. It doesn't make dependencies, test commands or credentials safe.

trustbut's documented workflow uses linked installed dependencies, and its hook runs tests outside the agent sandbox. Don't treat “clean checkout” as a hermetic environment or run untrusted branch code with personal credentials. Use a controlled, disposable runner with only the access the check needs.

Record runtime and dependency state. If the base cannot install dependencies or needs a compatibility adjustment, preserve that limitation. Silently changing the setup until something fails makes the comparison harder to interpret.

Request a receipt you can review

For this fix, keep a compact record:

  • Base and candidate revision IDs, plus the exact regression-test and fixture content or blob IDs.
  • Command, selected tests, runtime and dependency state.
  • Intended behavioral failure on the base and the actual failure location.
  • Actual results on both revisions, including the valid-input control, skips and setup failures.
  • Checks not run and acceptance changes approved during review.

Keep unrun checks and infrastructure failures out of the pass column. Run the broader candidate suite separately; this comparison doesn't cover every affected path.

Before merging, ask to see the original defect caught by the reviewed check. A green badge is easier to trust when you know exactly what would have made it red.

Source notes