I ran the exact same prompt against the exact same 120 test cases, twice, with different seeds. Then I did it 30 times.
The pass rates differed by up to 13.3 points between two runs that changed nothing. My old CI rule, "block the build if the score drops more than 3 points below baseline", would have blocked 8 of those 30 builds. Eight red pipelines, zero real regressions.
That's the part of LLM evaluation nobody puts on the dashboard: one eval run is a sample, not a measurement. This post is about one narrow piece of the fix, the gate itself, and why "pass or fail" is the wrong shape for it.
Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete, tested project behind this post (a .NET 10 eval service with Clean Architecture, judge calibration, paired gates and 215 passing tests, running offline against a simulator) plus a step-by-step PDF guide is inside. Join Tech Skill Builder
About the numbers: all of them are my own local measurements with a **seeded simulator* playing the model and the judge. They show how the statistics behave, not how good any real model is.*
The gate most of us start with
// The naive gate: compare two averages, apply a threshold. bool ShouldBlock(double baselinePassRate, double candidatePassRate) => baselinePassRate - candidatePassRate > 0.03; It looks reasonable. It has three hidden problems:
- It ignores noise. Two averages from noisy runs differ even when nothing changed, and a fixed threshold doesn't know how noisy your runs are.
- It throws away pairing. Both runs used the same cases. A hard case is hard for both. Comparing two averages discards that, so the noise from case difficulty stays in.
- It only has two answers. When the evidence is weak, it still has to say "ship" or "block". So it guesses, and you can't tell a guess from a verdict.
Fix 1: Compare per case, not per run
Since both runs answered the same cases, compute the difference for each case and look at the distribution of those differences.
public sealed record CaseResult(string CaseId, int Passes, int Samples) { public double Rate => (double)Passes / Samples; } public static IReadOnlyList<double> PairedDiffs( IReadOnlyDictionary<string, CaseResult> baseline, IReadOnlyDictionary<string, CaseResult> candidate) { // Only cases present in both runs count. Missing cases are a bug, not a zero. var ids = baseline.Keys.Intersect(candidate.Keys).OrderBy(x => x, StringComparer.Ordinal); return ids.Select(id => candidate[id].Rate - baseline[id].Rate).ToList(); } A case that both runs fail contributes 0. Only the cases where the runs disagree move the result. That alone removes most of the "13 points apart" noise.
Fix 2: Put an interval around the mean difference
A single mean difference is still a point estimate. Bootstrap it: resample the cases (not the individual samples) many times and take the 2.5th and 97.5th percentiles.
public readonly record struct Interval(double Low, double Point, double High); public static Interval MeanCi(IReadOnlyList<double> diffs, int resamples, ulong seed) { var rng = new DeterministicRandom(seed); // seeded, so CI gets the same answer on every machine var stats = new double[resamples]; // ... resample diffs with replacement, store each mean, sort, read percentiles // full implementation (plus the clustered version for repeated samples) in the complete project throw new NotImplementedException(); } Two details matter more than the code:
- Seed it yourself. Don't feed
string.GetHashCode()into a seed; it's randomized per process in .NET, and your gate would give different answers on different agents. - The case is the unit of independence. 120 cases run 5 times is 600 results, but only 120 independent things. In my runs, treating them as 600 gave an interval 6.7 points wide; resampling by case gave 9.8 points. The narrow one was simply wrong.
Fix 3: Three answers, not two
Now the gate can say what the evidence actually supports. Pick a tolerance (how much drop you'll accept, say 3 points) and look at both ends of the interval:
public enum GateDecision { Ship, Block, NeedMoreData } public static GateDecision Decide(Interval diff, double tolerance) => diff.Low > -tolerance ? GateDecision.Ship // even the pessimistic end is acceptable : diff.High < -tolerance ? GateDecision.Block // even the optimistic end is too bad : GateDecision.NeedMoreData; // honestly undecided This is a non-inferiority check. You ship only when the worst plausible reading is fine, and you block only when the best plausible reading is still bad.
On those same 30 identical-config pairs, this gate blocked 0 of 30. And a clearly broken prompt (about 22 points worse) was still blocked straight away. Fewer false alarms, no missed disasters.
Fix 4: Make "NeedMoreData" actionable
"Undecided" is only useful if it tells you what would decide it. A standard sample-size estimate does that, using the spread of your own per-case differences:
// Two-sided 5% test, 80% power. A planning approximation, not a proof. public static int CasesNeeded(double sdOfDifference, double margin) { // n ≈ (z_alpha/2 + z_beta)^2 * sd^2 / margin^2 // full implementation with input validation in the complete project throw new NotImplementedException(); } In my simulator, a prompt that was genuinely a bit weaker showed -4.5 points, p = 0.0503 after 5 repeats per case. The old gate would have flipped a coin. The new one said: "About 695 cases are needed to settle it; you have 120." That's a conversation you can actually have with your team.
Rule of thumb: halve the margin and you need four times the cases.
Two traps that make the gate lie anyway
Different judges. If run A was graded by one LLM judge and run B by another (or by the same model with a new judge prompt), the scores aren't on the same scale. I refuse the comparison outright:
if (!string.Equals(a.JudgeId, b.JudgeId, StringComparison.Ordinal)) throw new DomainException(ErrorKind.Conflict, "judge_mismatch", "..."); // the API maps this to a 409 with a stable code, so a CI script can branch on it (And before any of this, the judge itself needs checking against human labels. A judge that agrees with humans 90% of the time can still be close to useless; that's its own topic.)
Adaptive sampling. It's tempting to stop sampling a case once its first 3 runs agree. In my test, that cut model calls by 38% and pushed the pass rate from 76.8% to 83.5%, because early stopping favours the easy cases. Cheap and wrong. Don't gate on a number from an adaptive run; re-run with fixed repeats.
Checklist before your eval gate blocks a deploy
- Both runs used the same case set, and you compare per case.
- The interval resamples cases, not individual samples.
- The bootstrap is seeded with your own deterministic RNG.
- The gate has three outcomes, and NeedMoreData comes with a case count.
- Both runs were graded by the same, calibrated judge.
- No adaptive or early-stopped runs feed the gate.
- A provider outage fails the run (502), it never shows up as a quality drop.
What's in the full version
The snippets above are skeletons: the shape of each piece, with the bodies left out. The complete project in Tech Skill Builder is a working .NET 10 eval service (Domain / Application / Infrastructure / Api, plus unit and integration tests, 215 passing) with:
- judge calibration with Cohen's kappa and a bootstrap interval, failing closed on unusable verdicts,
- Wilson and clustered bootstrap intervals,
- the paired Ship / Block / NeedMoreData gate and the case planner,
- the judge-mismatch and adaptive-run guards,
-
IChatClientadapters so OpenAI, Azure OpenAI or Ollama can plug in, and a seeded simulator so everything runs offline, - a PDF guide that walks through the architecture and every decision.
How does your team decide an eval drop is "real" today: a fixed threshold, a human looking at the diff, or something statistical? I'm curious what's held up in practice.

