An example scenario, stated as a scenario
Example scenario (illustrative; not a measured case, and not drawn from a named project): a maintainer of a small library has five pull requests queued and access to a free model endpoint that will review diffs. On the first pull request the endpoint returns six comments; three read plausibly, one references a function that does not exist in the patched tree, and none explains what a complete review would have covered. Nothing in that run defines ground truth, so the maintainer cannot say whether the reviewer is useful or merely fluent.
Public bug-fix datasets solve this problem for their own repositories, not for yours, because they differ in language, defect mix and review conventions. A better-labelled corpus is usually sitting in the repository's own history, since every merged fix is a commit where the defect and its repair are both recorded. Reversing such a commit produces a patch that looks like an ordinary cleanup and quietly reintroduces a real bug.
The workflow below turns that idea into a small harness: build the corpus, freeze the prompt, score file and symbol recall, then decide what authority the reviewer has earned. Every script here is labelled illustrative because nothing was executed while writing this article; the commands are ordinary Git usage, but the numbers are yours to produce.
Step 1 — Build a corpus from your own merged fixes
The corpus builder walks recent fix commits and derives two artifacts per commit: a reverse patch that reintroduces the defect, and a ground-truth record of the files and removed identifiers that the fix touched. Adapt the commit filter to your project's conventions, since ^fix matches only commits that actually start with that word.
#!/usr/bin/env bash # build_corpus.sh — derive review tasks from merged fixes (illustrative) set -euo pipefail REPO="${1:?usage: build_corpus.sh <repo-path> [count]}" N="${2:-15}" OUT="$REPO/.review-audit" mkdir -p "$OUT/diffs" "$OUT/truth" git -C "$REPO" log --no-merges --grep='^fix' --format='%H' -n "$N" | while read -r fix; do parent="$(git -C "$REPO" rev-parse "${fix}^")" # fixed tree -> buggy parent: applying this patch removes the fix git -C "$REPO" diff --unified=3 "$fix" "$parent" > "$OUT/diffs/$fix.patch" # ground truth: touched paths git -C "$REPO" diff --name-only "$parent" "$fix" > "$OUT/truth/$fix.files" # ground truth: identifiers the fix removed (they come back in the reverse patch) git -C "$REPO" diff --unified=0 "$parent" "$fix" | grep -E '^-' | grep -v '^---' | grep -oE '[A-Za-z_][A-Za-z0-9_]{3,}' | sort -u > "$OUT/truth/$fix.symbols" done ls "$OUT/diffs" | wc -l The argument order in git diff "$fix" "$parent" is deliberate, because the diff reads as a change that takes the current tree backwards. When that patch is later presented to a reviewer, it must be described exactly like any other proposed change, with no hint that a defect is present. Line numbers are deliberately excluded from ground truth here, since the reverse patch's line numbers refer to the buggy tree and a naive comparison would score noise.
Validity gate: keep only defects with an observable failure
A reverse patch that no test detects is weak ground truth, because the audit then measures a judgement call rather than a known break. Apply each reverse patch in a scratch worktree, run the project's test command, and record the exit status next to the corpus entry.
OUT="$REPO/.review-audit" for fix in $(ls "$OUT/diffs" | sed 's/\.patch$//'); do git -C "$REPO" worktree add --detach "/tmp/audit-wt-$fix" "$fix" >/dev/null 2>&1 || continue if git -C "/tmp/audit-wt-$fix" apply "$OUT/diffs/$fix.patch"; then ( cd "/tmp/audit-wt-$fix" && ./run-tests.sh ) ; echo "$?" > "$OUT/truth/$fix.observed" else echo "apply_failed" > "$OUT/truth/$fix.observed" fi git -C "$REPO" worktree remove --force "/tmp/audit-wt-$fix" done grep -L '^0$' "$OUT"/truth/*.observed Entries whose observed file is 0 describe a bug your suite never caught, which is a legitimate finding but a poor recall target. Drop entries with apply_failed, and drop lockfile-only, formatting-only and vendored-code fixes, because they add review surface without adding defect signal.
Step 2 — Freeze the prompt and the output schema
Repeatability depends on an invariant request, so the prompt must be identical across models, repeats and days, and it must never reveal that a defect exists. The schema below asks for file:line claims because they make scoring mechanical, and it adds an explicit silence token so a reviewer that always writes prose can be told apart from one that always finds something.
You are reviewing a proposed change to a <language> repository. Report only issues you can locate in the changed lines. For each issue, emit exactly one line: <path>:<line> <low|med|high> <one-sentence claim> If you find no issue, emit exactly: NO_ISSUES No preamble, no summary, no praise. Two operational details decide whether the run means anything. First, log the model string you requested plus any model identifier the response carries, because free endpoints may route requests to different models over time. Second, keep temperature and any sampling parameter fixed in the request, and record them in the run log next to the model string.
Step 3 — Score file recall, symbol recall and silence rate
The scorer below reads a JSONL run log with keys task, model, repeat and text, and prints three numbers per model. File recall is the primary metric; symbol recall is a weaker tiebreaker, because removed identifiers are matched by substring and common words inflate it.
#!/usr/bin/env python3 """score_audit.py — illustrative; not executed for this article.""" import collections, json, pathlib, sys truth = pathlib.Path(sys.argv[1]) # .../.review-audit/truth runs = pathlib.Path(sys.argv[2]) # runs.jsonl def hit(text, needles): low = text.lower() return any(n.lower() in low for n in needles) scores = collections.defaultdict(lambda: {"n": 0, "file": 0, "sym": 0, "silent": 0}) for line in runs.read_text().splitlines(): row = json.loads(line) files = (truth / f"{row['task']}.files").read_text().split() syms = [s for s in (truth / f"{row['task']}.symbols").read_text().split("\n") if s] s = scores[row["model"]] s["n"] += 1 s["file"] += hit(row["text"], [f.split("/")[-1] for f in files]) s["sym"] += bool(syms) and hit(row["text"], syms) s["silent"] += "NO_ISSUES" in row["text"] for model, s in sorted(scores.items()): n = max(s["n"], 1) print(f"{model}: n={n} file_recall={s['file']/n:.2f} " f"symbol_recall={s['sym']/n:.2f} silent_rate={s['silent']/n:.2f}") Hand-inspect at least five scored runs before trusting any aggregate, because a reviewer that lists every file in the repository will score near-perfect recall and remain useless. The shape of a result table is what matters, not any number reproduced here: two models, three repeats, fifteen tasks, and per-model rows for file recall, symbol recall and silence rate. Post-process those rows into one labelled decision and stop there, since below roughly ten corpus entries the confidence interval swamps the difference between models.
Step 4 — Decide what authority the reviewer has earned
The table converts a measurement into an allowed use, and the forbidden column matters more than the permitted one. A reviewer with high recall and high noise is not a cheap human replacement; it is an unpriced interruption.
| Observed on your corpus | Permitted use | Explicitly not permitted |
|---|---|---|
| file recall below 0.3 | none; read the output as casual opinion only | gating a pull request, claiming coverage |
| recall 0.3–0.6, silence rate above 0.5 | triage: rank pull requests by distinct flagged files | any statement that a patch was reviewed |
| recall above 0.6, silence rate below 0.3 | second-pass attention list before a human reads the diff | approval, merge rights, security sign-off |
| any recall with flags on most files | nothing until the prompt is tightened | cost-savings estimates from reduced review time |
Track at least one defect class separately when the corpus supports it, because aggregate recall hides the classes a reviewer never notices. Null handling, off-by-one boundaries and resource cleanup fail in different ways, and a model strong on one class is routinely blind to another.
Step 5 — Where free model access and a free server change the arithmetic
One maintainer running fifteen tasks across three models and three repeats issues 135 review calls, which is too many to absorb as a recurring personal expense for a hobby repository. Free model access removes that budget negotiation, which matters precisely because the audit must happen before any spending decision rather than after it.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is the project whose operator described free model access and a free server option for this workflow, and, as stated at the time of writing, a free token allowance on the order of ten million tokens. Allowances and model availability change without notice, so confirm the current terms on the project's own pages before planning a corpus size around them. The harness itself is I/O-bound and mostly idle, so scheduling it as a batch run on a free server leaves the local machine for builds, tests and the debugger, which is the resource the audit competes with most.
Limitations, and who should skip this
- Corpus size dominates everything. Fewer than ten validated entries produce recall figures too unstable to compare across models, so report the count beside every percentage.
- A reversed fix is a synthetic regression, and reviewers may behave differently on a live pull request that carries discussion, CI status and reviewer history.
- The audit measures detection of defect classes already present in your history, and it says nothing about novel classes, design flaws or security properties.
- Repositories where fixes arrive as squashed upstream merges, dependency bumps or generated code will spend most of the harness on noise rather than on review signal.
- Free-tier terms, routing and model identity can change silently, so an audit is a snapshot with an expiry date rather than a certification; re-run it when the model string changes.
- Skip the approach entirely if your queue is small, your test suite is thin, or policy already requires line-by-line human sign-off, because no figure from this harness should ever grant approval authority.
A minimal cadence
Re-run the audit after any prompt edit, whenever the endpoint's model string changes, and once per quarter as a baseline check. Keep the corpus, the run log and the scorer under a path excluded from packaging, so the comparison is reproducible rather than remembered. Maintainers who want to test the shape of this on real history can start with the corpus builder and one model, then decide whether the free tier is worth wiring into a scheduled job.
