The cheapest explanation for a sudden, uniform eval shift is usually the scorer, not the model, so a grader you never audit is a silent regression source that no prompt diff will ever reveal. A grader change rewrites every historical number at once, and because it does so consistently, the resulting shift looks exactly like a model upgrade that helped. The fix is unglamorous: freeze a small human-labeled anchor set, score it with the current grader on every change, and compare per-label recall against a stored baseline before you trust any cross-run comparison. This article describes that loop, the arithmetic that keeps it from crying wolf, and the cases where it is the wrong tool.

Why an aggregate pass rate cannot see a grader change

A single pass rate collapses every label into one bucket, so a grader that loosens its warn rule while tightening its fail rule can leave the mean almost unchanged. The two errors cancel in aggregate and remain fully visible per label, which is why the anchor audit reports precision and recall for each label instead of a headline number. If your harness only stores passed: true/false, you have already thrown away the signal this method depends on, and the first step is to keep the raw grader output alongside the verdict.

The second reason aggregates mislead is that graders change for mundane reasons: a regex tightened to stop catching null, a rubric sentence reworded for clarity, a new JSON schema validator. None of these are model changes, yet all of them move the score, and a team watching a dashboard will attribute the movement to the prompt it shipped the same afternoon. An anchor set turns that attribution problem into a diff.

An anchor set is a fixture, not a test suite

An anchor set is a frozen list of inputs with human-assigned labels, and its only job is to measure the grader rather than the model. Because the inputs never change, you can re-score them from stored outputs without calling any model, which makes the audit cheap enough to run on every commit that touches grader code. Stratify the set by label so each class has enough members for a stable estimate, and deliberately include borderline items where a reasonable human hesitated, since those are the items a rubric edit flips first.

Keep two rules. Never edit an anchor item and its label in the same commit as a grader change, because that destroys the comparison; and never let the anchor set grow into a general regression suite, because a fixture that people keep "improving" stops being a fixed reference. Ambiguous items belong in a separate quarantine file until two annotators agree on them.

The audit script

The following code is illustrative and was not executed for this article; treat it as a starting point that you run and correct against your own label schema. It assumes rows of {"id": str, "label": str, "output": str} and a pure grader(output) -> str function, which is the shape most rubric graders can be wrapped into.

# anchor_audit.py — illustrative, not executed here import json, math from collections import defaultdict def wilson(k, n, z=1.96): if n == 0: return (0.0, 0.0) p = k / n d = 1 + z * z / n c = p + z * z / (2 * n) r = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) return ((c - r) / d, (c + r) / d) def audit(rows, grader): tp, fp, fn = defaultdict(int), defaultdict(int), defaultdict(int) for row in rows: gold, got = row["label"], grader(row["output"]) if gold == got: tp[gold] += 1 else: fp[got] += 1 fn[gold] += 1 report = {} for label in set(tp) | set(fp) | set(fn): p_den, r_den = tp[label] + fp[label], tp[label] + fn[label] report[label] = { "precision": tp[label] / p_den if p_den else None, "recall": tp[label] / r_den if r_den else None, "recall_ci": wilson(tp[label], r_den), "support": r_den, } return report def drift(baseline, current, min_delta=0.10, max_width=0.25): flags = [] for label, m in current.items(): base = baseline.get(label) if not base or m["recall"] is None: continue lo, hi = m["recall_ci"] if hi - lo > max_width: flags.append((label, "insufficient_n", round(hi - lo, 3))) elif abs(m["recall"] - base["recall"]) >= min_delta: flags.append((label, "recall_shift", round(m["recall"] - base["recall"], 3))) return flags if __name__ == "__main__": import argparse ap = argparse.ArgumentParser() ap.add_argument("--baseline", required=True) ap.add_argument("--current", required=True) args = ap.parse_args() base = json.load(open(args.baseline)) cur = json.load(open(args.current)) for row in drift(base, cur): print(*row)

Run it as python anchor_audit.py --baseline anchor_baseline.json --current anchor_now.json and treat any printed line as a stop-the-line event for grader changes only. The insufficient_n branch is deliberate: it reports that you cannot decide, which is a different outcome from a clean pass and should never be silently swallowed by CI.

Reading the output with intervals

A per-label recall of 0.90 over forty items carries a Wilson 95% interval of roughly 0.77 to 0.96, which is simple arithmetic, not a measurement from any system. A five-point recall shift therefore sits entirely inside that interval, so a min_delta of 0.05 on forty items will page you for noise, while 0.10 or more is defensible. If you need to resolve smaller shifts, the honest answer is more anchor items per label, and the interval width is how you decide how many rather than guessing.

Two practical rules follow. First, compare interval overlap before comparing point estimates, and second, store the baseline JSON in version control next to the grader code so the baseline travels with the change that produced it. When a flag fires, replay the stored anchor outputs through both the old and the new grader locally to confirm the diff is in the scorer, because a corrupted fixture file produces the same alarm for the wrong reason.

Where the loop runs

Because the audit should run on every grader edit and on a weekly cadence, execution cost rather than cleverness decides whether it survives contact with a real team. The outputs are stored, so scoring is free, but re-labeling new anchor candidates and smoke-testing candidate models against the anchor set does consume calls. The MonkeyCode project, an open-source offering, provides free model access and a free server option, which is enough to host the weekly job and to draft label candidates for human review; the operator describes the free tier as including ten million tokens, so confirm the current quota and terms on the official project page before you size anything against it, since capacity claims change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What this does not catch, and who should skip it

An anchor set says nothing about model quality on inputs it does not contain, so it will not detect a new failure mode that appears only in fresh traffic. It also assumes your labels are stable, and for genuinely subjective output, two annotators may disagree more than the drift threshold you want to set, which makes the whole comparison meaningless. If your grading is human-only at low volume, or if a single reviewer already reads every output, the audit adds ceremony without adding information.

Teams with no frozen human labels should not start here, because an anchor set built from model outputs grades the model against itself and will happily validate a broken rubric. Fix the labeling process first, and adopt the audit once at least two people can agree on the ambiguous cases.

A checklist before you trust the alarm

Before wiring this into CI, confirm four things in order: the grader is a pure function of the stored output, the baseline file matches the grader version it was recorded with, each label has enough support that the interval is narrower than your threshold, and the ambiguous quarantine file is excluded from scoring. Then run the audit once against an unchanged grader and confirm it prints nothing, which is the only cheap proof that your plumbing is not the source of the next alarm.

If you want to try the anchor loop without provisioning anything first, the free server and free model access are a reasonable place to start; bring your own labeled anchors, because no quota substitutes for the human labels the method depends on.