Review-Driven Development: a method for the bugs a 95% mutation score can't see

Last week I wrote that Fuzzphony 0.11 had about 3,700 tests, PHPStan at max level, close to 100% line coverage and a 95% mutation score over roughly 11,900 mutants, and that an independent review of about 2,700 added lines still found three real bugs. One of them loaded another tenant's row.

I ended that post with one sentence: independent review is now part of the release process. A sentence is not a method. This post tries to make it one, with a definition, a loop, and numbers I can be held to. I call it review-driven development (RDD). The name is not new (see the prior art section), so I define what I mean by it.

1. Every automated check compares the code to something written down

Here is what each layer in my pipeline compares the code against, and what that makes it blind to. The evidence column is from my own two experiments: the architecture ruleset (12 hand-made rule violations against 7 checks) and the Fuzzphony 0.11 release.

Testing research calls this the oracle problem: a test needs something that says what the correct output is, and that something has to come from outside the code (Barr et al., 2015). Mutation testing (DeMillo et al., 1978; Jia and Harman, 2011) measures how well the tests use the oracle they have. It cannot produce a new one.

All three of my bugs were oracle gaps. The affected lines were covered and ran, the suite was green, and the requirement that was violated was asserted by no test, no rule and no type. The sharpest form of this is what Brian Marick called a fault of omission: the defect is code that should exist and doesn't, so no coverage-based technique can point at it. The cross-tenant load was exactly that, a filter that was never written.

A second number from the architecture experiment matters here. Of the 9 violations that were caught, 5 were caught by exactly one check. More than half of the detections depended on a single layer. Adding a layer of a different kind buys more than tightening an existing one.

Review is the only layer in the table whose reference is not an artifact the author produced. That is its value and the reason it is expensive. So the method has two goals: spend review only on what the other layers cannot see, and turn everything it finds into something they can.

2. What the research says about review

Review finds fewer functional defects than people expect. Formal inspections (Fagan, 1976) were designed to find defects. Modern tool-based review mostly does something else. Bacchelli and Bird (2013) observed and surveyed developers at Microsoft: finding defects was the top motivation, but the actual outcomes were mostly understanding, knowledge transfer and alternative solutions. Mäntylä and Lassenius (2009) classified review findings and put about 75% of them in the evolvability category, not functionality. Beller et al. (2014) found the same 75:25 split in open source.

The consequence: an unaimed review of a diff will mostly return naming and structure remarks. If I want it to find a tenant leak, I have to aim it.

How the reviewer reads matters more than whether there is a checklist. In the inspection experiments of the 1990s, readers who were given a specific scenario or fault class to hunt found more than readers with a generic checklist, and the generic checklist did no better than ad hoc reading (Porter, Votta and Basili, 1995). Perspective-based reading (Basili et al., 1996) built on that; replications were mixed (Regnell et al., 2000). These experiments were run on requirements documents, not on code, so applying them to code review is my extrapolation. I take one thing from this line of work: give the reviewer a target, not a list of good practices.

Mutation testing and review already meet, in one direction. Google surfaces surviving mutants inside the code review tool, on changed lines only (Petrović and Ivanković, 2018). Their follow-up study reports that developers exposed to this write more tests, and that mutants are coupled to real faults (Petrović et al., 2021). That is mutation feeding review. RDD adds the opposite arrow: review feeding the test suite.

In 2026 review is the bottleneck. LeadDev's 2026 report says 68% of respondents changed how they review because of AI, and quotes a CodeRabbit analysis of 470 pull requests with about 1.7 times more issues in AI-generated ones. I wrote most of Fuzzphony with Claude Code. When the code and its tests come out of the same session, I expect them to share blind spots, and a mutation score then only confirms that the two agree with each other.

Prior art. None of the parts is new. ComputerMinds used "review-driven development" for a peer-review habit. The "preview-driven development" gist (sleepyfox, 2020) moves review before coding. Several AI-era write-ups add an "adversarial review" step after generation. Fixing the class of a bug instead of the instance is what security teams call variant analysis. Planting known faults to measure a process is error seeding (Mills, 1972). What I add is the combination: review aimed at a ledger of invariants, a seed corpus that grows out of the project's own fixes, and four numbers to report per release.

3. The method

Review-driven development is a loop in which

1. the review is aimed at named invariants, not at the diff,
2. the reviewer is independent of the author's reasoning,
3. a finding is closed only by an executable artifact,
4. the review itself is tested.

NAME -> BUILD -> ATTACK -> ENCODE -> CALIBRATE ^ | +-----------------------------------------------+ 

Step 3 is what happened in Fuzzphony 0.11, followed by a fix for each finding. The ledger, class-level encoding, seeding and the metrics are what I am adding from 0.12 on. I have no data for step 5 yet.

Step 1. Name: an invariant ledger

An invariant is a sentence that must stay true across every feature, written so that it can be attacked. Fuzzphony has two that matter, named since 0.10. I keep them in one file with the question a reviewer should ask:

# INVARIANTS.md  ## INV-1 tenant-isolation For any actor of tenant A and any call that is not asSystem(): no id, row, count, facet or highlight derived from tenant B reaches the caller. Attack: find any path from a search result to data where the tenant is implied instead of checked. Enforced by: tests group inv:tenant-isolation ## INV-2 hidden-field-independence For an actor without the grant on field F: changing the content of F changes nothing in the response (ids, order, totals, facets, highlights, with()). Attack: find any second name, path or side channel that reaches F's data. Enforced by: tests group inv:hidden-field-independence 

Both are written as relations between two runs, not as expected values. That is deliberate. Metamorphic testing (Chen et al., 1998) uses such relations as oracles when the correct output is unknown, and a relation covers a class of inputs where an example covers one.

Step 2. Build: the cheap layers on every push

Nothing new here, only the order. Fast checks on every PR, the full mutation run once per release (mine takes 1 h 45 min):

vendor/bin/phpstan analyse vendor/bin/deptrac analyse vendor/bin/phpunit vendor/bin/infection --git-diff-lines --git-diff-base=origin/main \ --threads=max --min-covered-msi=90 --ignore-msi-with-no-mutations 

Surviving mutants on changed lines get a test or a written reason why they are equivalent. This step answers "are the tests I wrote any good". It says nothing about the tests I did not write.

Step 3. Attack: a review brief, not a pull request

The reviewer can be a colleague or a fresh LLM session. Three rules apply to both.

The reviewer gets the diff and the ledger, and nothing else. No PR description, no commit rationale, no walkthrough. Every explanation I give transfers my model of the system, and my model is the thing under test.

The reviewer reads and does not run the test suite. The review exists to find what the suite cannot. Step 5 also depends on this rule.

A finding has a fixed shape: the invariant violated, a concrete input, the observed result, the required result. "This looks risky" is not a finding.

The brief I will use from 0.12 on:

# Review brief (template) Scope: git diff <previous tag>..<release branch> -- src/ Targets: INV-1, INV-2 Not wanted: style, naming, structure Access: read-only, no test runs Finding format: invariant | input | observed | required 

The 0.11 diff was about 2,700 lines, far above the size where line-by-line reading works. Reading by invariant makes it tractable: the reviewer follows only the paths where a tenant or a grant flows, and ignores the rest.

One heuristic from my own data. Look at what the three bugs have in common:

Each bug sits in code that assumed a singular where the system allowed a plural. The first case is the clean one: 0.11 itself turned the 1 into an N, and a consumer written for the old world kept working on bare ids. The other two are relatives of it. So the brief gets one more line: list every place that assumes exactly one of something this release touches, and check each against the invariants. One clean case and two relatives from one project make a hypothesis, not a law. It is cheap to apply.

Step 4. Encode: a finding is closed by a machine

A fix plus a regression test for the reported input closes the instance. RDD asks for more: encode the class, at the cheapest layer that can see it. In order of preference:

1. the language (types, readonly, enums),
2. a static rule (PHPStan custom rule, about twenty lines),
3. an architecture rule (Deptrac, phpat),
4. a test generated from configuration,
5. an example test,
6. a new line in the ledger, when nothing above fits yet.

For the alias leak, the class is "a second name reaches the same data", which is the attack line of INV-2. The test should enumerate every granted field from the index configuration, not the one field from the report. A sketch (helper names are mine, not Fuzzphony's API):

#[Group('inv:hidden-field-independence')] final class GrantAliasTest extends KernelTestCase { #[DataProvider('everyGrantedField')] public function testNoNameReachesAGrantedColumn( string $index, string $field, string $column, ): void { $actor = $this->actorWithoutGrantOn($index, $field); // outcome() returns 'refused' or the list of returned columns self::assertSame('refused', $this->outcome($index, $actor, with: $field)); self::assertSame('refused', $this->outcome($index, $actor, with: $column)); } public static function everyGrantedField(): iterable { // yields [index, field, column] for every granted field // of every configured index, so new fields are covered on arrival } } 

A field added next year is tested without anyone remembering this bug. That is the difference between encoding an instance and encoding a class.

Step 5. Calibrate: who reviews the reviewer?

My Infection article asked "who guards the guards". Tests guard the code, mutants guard the tests, review guards the requirements. That leaves the review unguarded. When a review comes back clean, I cannot tell "no bugs" from "a reviewer that finds nothing".

Mutation testing answers that question for tests by planting faults. Mills proposed the same for programs in 1972. Applied to review, the project's own history is a free source of realistic seeds.

Every fixed finding becomes a seed. The reverse patch of the fix is a real invariant violation, of a kind this codebase has actually produced. Store it:

review-seeds/ INV-1-wrong-kind-ids.patch INV-2-column-alias.patch INV-1-first-index-only.patch 

Before a release review, adapt two or three seeds to the current code and apply them to the review copy only (never to a branch that can merge). The reviewer is told neither which seeds nor how many, and nothing about seeds appears in the brief. This is why the reviewer must not run the tests: after step 4 the suite catches these seeds at once, and the point is to measure the reader. I keep the stop rule on my side:

• Seed recall = seeds found / seeds planted. Below 100%, a clean result is not evidence, and the review is repeated with a different reviewer.
• A rough estimate of what is left, after Mills: with S seeds planted, s found and r real findings, remaining ≈ r × (S/s − 1). With 3 planted, 2 found and 3 real findings that is about 1.5 bugs still in the code.

With numbers this small the estimate is an alarm, not a measurement. The same caution applies to the alternative, capture-recapture with two independent reviewers: the inspection literature found those estimates unreliable for small teams (El Emam and Laitenberger, 2001).

Seeds have a second use with LLM reviewers. Model, prompt and context all change between releases. Seed recall is the only number I know that says whether this release's reviewer is as good as the last one.

4. What to report

Next to the mutation score I will publish four numbers per release:

Repeat-class rate is the one that tells whether the method works. If step 4 is done properly, a class found once is caught by a machine afterwards, and a human should never have to find it again. A repeat means the encoding was too narrow. The target is zero.

Oracle-gap share tells where to invest. A low share means review is finding bugs in code the tests never reached, so the money belongs in step 2. Mine was 3 of 3, which is why this post exists.

5. When not to do this

The same places where I argued mutation testing does not pay: prototypes, plain CRUD, thin read-only endpoints. RDD needs invariants worth naming. If you cannot write one sentence that must hold across every feature, there is nothing to aim the review at, and an ordinary pull request review is the right tool.

It fits where one violated sentence is an incident: multi-tenant data, permissions, money, anything with a legal retention or disclosure rule.

6. Limits

• Three findings, one library, one author. The method is derived from a single release.
• The singular-assumption pattern was found after the fact. I have not yet used it to predict a bug.
• Oracle-gap share is defined on line coverage. A stricter version would count only lines with no surviving mutant.
• Seeds come from past findings, so seed recall measures sensitivity to known classes. A reviewer can score 100% and still miss a new kind of bug.
• Seed recall only works with a reviewer who reads and does not run the suite. A reviewer with CI access finds every seed for free.
• How independent a second LLM session is from the one that wrote the code is unmeasured. Seed recall is a partial answer.
• The Infection flags above are the ones I use; check them against your version.
• The cost is real: a brief, a reviewer, and a test per finding class. I have not measured hours.

If you run a review like this on a Symfony codebase, I would like to know your oracle-gap share. My guess is that it is high wherever the mutation score is high, because by then the easy bugs are gone.

References

1. Mills, H. D. (1972). On the statistical validation of computer programs. IBM FSC Report 72-6015.
2. Fagan, M. E. (1976). Design and code inspections to reduce errors in program development. IBM Systems Journal, 15(3).
3. DeMillo, R. A., Lipton, R. J., Sayward, F. G. (1978). Hints on test data selection. IEEE Computer, 11(4).
4. Porter, A., Votta, L., Basili, V. (1995). Comparing detection methods for software requirements inspections: a replicated experiment. IEEE TSE, 21(6).
5. Basili, V. et al. (1996). The empirical investigation of perspective-based reading. Empirical Software Engineering, 1(2).
6. Chen, T. Y., Cheung, S. C., Yiu, S. M. (1998). Metamorphic testing: a new approach for generating next test cases. HKUST-CS98-01.
7. Regnell, B., Runeson, P., Thelin, T. (2000). Are the perspectives really different? Empirical Software Engineering, 5(4). doi:10.1023/A:1009848320066
8. Marick, B. (2000). Faults of omission. Software Testing and Quality Engineering Magazine.
9. El Emam, K., Laitenberger, O. (2001). Evaluating capture-recapture models with two inspectors. IEEE TSE, 27(9).
10. Mäntylä, M., Lassenius, C. (2009). What types of defects are really discovered in code reviews? IEEE TSE, 35(3). doi:10.1109/TSE.2008.71
11. Jia, Y., Harman, M. (2011). An analysis and survey of the development of mutation testing. IEEE TSE, 37(5).
12. Bacchelli, A., Bird, C. (2013). Expectations, outcomes, and challenges of modern code review. ICSE 2013. doi:10.1109/ICSE.2013.6606617
13. Beller, M., Bacchelli, A., Zaidman, A., Juergens, E. (2014). Modern code reviews in open-source projects: which problems do they fix? MSR 2014.
14. Barr, E. et al. (2015). The oracle problem in software testing: a survey. IEEE TSE, 41(5).
15. Petrović, G., Ivanković, M. (2018). State of mutation testing at Google. ICSE-SEIP 2018. doi:10.1145/3183519.3183521
16. Petrović, G., Ivanković, M., Fraser, G., Just, R. (2021). Does mutation testing improve testing practices? arXiv:2103.07189
17. LeadDev (2026). Use of AI has us creating more code than we can review. April 13, 2026.
18. ComputerMinds. Review driven development? (blog post).
19. sleepyfox (2020). Preview-driven development.
20. Infection: command line options.

I maintain Fuzzphony, a PostgreSQL search library for databases you are not allowed to change.