How to build an LLM evaluation from real failures

Build the eval from failures you have actually read, rather than from a rubric you wrote in advance. Group those failures, put one domain expert in charge of the standard, and create the cases only after that. Use code where the property is mechanical, and add a model judge only after it agrees with that expert.

Bigcircle uses this order on client systems, because a score you cannot trace to a failure does not tell you what to fix. Every guide on this site tells you to build evals before you do anything else, and none of them has explained how. That gap is worth closing, because the order of the steps matters more than any individual technique in them, and most of the teams we work with have the order backwards.

The difficulty is not that evaluation is conceptually hard. It is that the thing you are trying to measure does not hold still. A traditional test suite asserts that a function returns four, and four is four regardless of who is asking. An eval asserts that an answer is grounded, or that a citation supports the claim attached to it, or that a system correctly declined to answer a question it had no evidence for. Those are judgements, they vary between reasonable people, and they change as you learn more about your own product. Barnett et al. reached the same conclusion across research, education and biomedical deployments: "validation of a RAG system is only feasible during operation" (Seven Failure Points, IEEE/ACM CAIN 2024). You cannot certify these systems the way you would certify a payments integration, which means the harness is not a gate you pass once but an instrument you keep running.

In this guide we walk through the process we use on client systems, in the order we use it. It starts with reading real failures before writing any criteria, grouping those failures into a taxonomy, and deciding who owns the standard. From there you build the case set, choose what to measure on it, and decide which of those checks need a model at all. Then you write the rubric for the ones that do, and validate that judge against human labels before it gates anything. Where our own systems illustrate a step, we use them: two in production with clients, and two we built and run ourselves.

What is LLM evaluation, exactly?

An eval is a test whose assertion requires judgement rather than equality.

The distinction matters more than it first appears. A unit test can be written before the feature exists, because the correct answer is knowable in advance. An eval usually cannot, because the definition of a correct answer is something you discover by looking at wrong ones. Everything else about an eval is ordinary engineering: it lives in the repository, it runs in continuous integration, it fails a build, and it has a history you can bisect when a regression appears.

# unit test assert parse_date("2026-09-01") == date(2026, 9, 1) # eval assert grounded(answer, passages) # every claim traceable to a passage assert cites_correctly(answer, passages) # each citation contains what it claims assert refused_when_unanswerable(answer, passages) 

Why can't you just write the rubric first?

Because you do not know your criteria until you have looked at outputs, and looking creates them.

This is not a failure of discipline, and a more rigorous team does not avoid it. Shankar et al. identified it as a structural property of the task and named it criteria drift:

To grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria.

We see this on nearly every engagement. A team sits down to write a rubric before launch and produces something like "the answer should be accurate and helpful." An hour of reading real outputs turns that into something far more specific. Does the response cite the controlling clause rather than a topically related one? Does it flag a conflict between two policies instead of silently picking one? Does it decline when the user's jurisdiction is not covered by the documents in scope? None of those would have occurred to anyone in advance. All of them are obvious in retrospect.

This is also why prefabricated metrics tend not to survive contact with a real product. A generic helpfulness or coherence score measures an abstraction that has no particular relationship to the failure modes your system exhibits, and when it moves you cannot say which change moved it. We have watched a team spend a sprint chasing a coherence score that fell after a retrieval change, on a system whose actual problem was that the reranker had started dropping the controlling clause. The score was real. It just was not measuring the thing that broke.

The rubric, then, is an output of the process rather than an input to it. The process that produces it is error analysis.

How do you find your actual failure modes?

Read your traces in a structured way before anyone writes a criterion, in three passes borrowed from qualitative research methods. The structure matters because unstructured reading produces impressions rather than counts, and impressions cannot tell you where to spend a quarter.

Start with open coding: pull real traces and write a free-form note on each one that went wrong. There are no categories at this stage and no attempt at consistency, because imposing either would defeat the purpose. Notes look like "cited the wrong section," "answered when it should have refused," or "used the policy that was superseded in March." The notes are raw observation, and the discipline is to resist labelling while you are still looking.

Then group those notes, the pass called axial coding, so that similar complaints collapse into a category and you end with a failure taxonomy of roughly five to ten modes. Whether "cited the wrong section" and "invented a clause number" become one category called citation error or two separate ones depends entirely on what you saw, and that decision is the substantive work of the pass.

Count before you plan the work, because the count decides where engineering time goes. If forty-three percent of your failures are retrieval misses, no amount of prompt work will move your quality this quarter. Counting makes that unarguable rather than a matter of opinion between two engineers who each read a handful of examples.

You stop at theoretical saturation, which in practice means twenty consecutive traces producing no new category, with at least a hundred reviewed before you trust that. Do not stop earlier. It is the common error, and it leaves a taxonomy of the sample rather than of the system. The modes you found are real, but the ones you missed are the ones that will show up in production. Every evaluator you build afterwards then targets a failure you have seen in your own system, rather than one you read about.

Who decides what good means?

One person, and not a committee.

Pick one principal domain expert: someone with real depth in the domain, or who truly represents the users the system is built for. Their job is to set the standard, and it has to be one person because three stakeholders labelling in parallel produce three rubrics and an argument. We have sat through that argument more than once, and it is almost always about vocabulary rather than about quality, which means it consumes a week and settles nothing. You need labels that stay consistent from one trace to the next, and one person gives you that where a panel does not.

The work itself is small and specific. The expert reads a trace, makes a binary pass-or-fail judgement, and writes a critique explaining the reasoning behind it. Teams cut the critique when they are busy, and cutting it costs more than it saves, because those critiques become the few-shot examples in the judge prompt later. On the trademark platform the useful ones named the clause that should have governed and said why the wrong one was plausible, which is the reasoning a judge needs in order to make the same call. The difference between a usable critique and a useless one is large:

The time cost is lower than teams tend to fear. Thirty to fifty traces to establish the standard, then periodic review as the system changes. What cannot be delegated is the judgement itself, because the expert is the only available source of the standard the system is being held to.

Where do the cases come from?

You have a taxonomy and someone who owns the standard. The next thing to build is the set of cases both will be applied to, and it is built to cover the modes the taxonomy named rather than to sample traffic. Keep it as JSONL, one case per line, so a diff is readable and a new case is one appended line.

{"id": "refund-001", "query": "What is the refund window for enterprise annual plans?", "gold_passages": ["policy-billing#p14"], "gold_answer": "30 days from invoice date.", "must_refuse": false, "tags": ["billing"]} {"id": "refund-002", "query": "Can I get a refund after 90 days?", "gold_passages": ["policy-billing#p14"], "gold_answer": null, "must_refuse": true, "tags": ["out-of-policy"]} {"id": "acl-001", "query": "What is the Q3 headcount plan?", "gold_passages": [], "must_refuse": true, "tags": ["permissions"], "as_user": "contractor@example.com"} 

Two decisions in that format do most of the work.

Label the passage, not the document. This is the mistake most worth avoiding early:

{"gold_passages": ["employee-handbook.pdf"]} // useless {"gold_passages": ["employee-handbook.pdf#p31-parental-leave"]} // testable 

A document-level gold label makes retrieval recall look excellent while measuring nothing, because any chunk drawn from a ninety-page file counts as a hit. Ask the expert to mark the minimum span that answers the question, and if two spans are required, record both, which turns the case into a multi-hop test at no extra cost.

Stratify the set by the taxonomy you found, not by traffic volume. A uniform sample of production traffic returns whatever your users happen to ask most often, which is usually one easy intent repeated many times.

Once a hand-built set exists, Chroma's generative benchmarking report (April 2025) describes one way to grow it: filter your documents with a judge aligned to your expert, then generate queries from what is left, steered by real production queries so they read like your traffic.

What should you measure?

With cases in hand, you can decide what to measure on them. Decompose the system first. Retrieval, tool use and generation fail for unrelated reasons and are repaired by unrelated work, so a single quality score cannot tell you which of the three broke on any given change.

Here is the whole set in one place, before we go through them. Standard names
are given where one exists, because you will meet these under those names in
Ragas and
elsewhere.

Two notes on that table. The gates are the ones we start from and then move once
a system has its own history; they are not standards. And precision does not
appear on the retrieval rows, because at the depths that matter here recall is
the constraint. Precision starts to matter when you are paying per passage in a
long context window.

Retrieval

If the passage never reaches the model, nothing downstream can recover, which is why this layer is measured first and why a low number here makes every other measurement uninterpretable.

def recall_at_k(retrieved_ids, gold_ids, k): return len(set(retrieved_ids[:k]) & set(gold_ids)) > 0 def reciprocal_rank(retrieved_ids, gold_ids): for i, doc_id in enumerate(retrieved_ids, start=1): if doc_id in gold_ids: return 1 / i return 0.0 

Measure recall at your candidate depth and again at your final depth, because the two numbers answer different questions. Recall at fifty tells you what retrieval could deliver if nothing downstream discarded it, and recall at five after reranking tells you what the model actually receives. The gap between them is the reranker's contribution. If recall at fifty is already low, no reranker can help and the work belongs in retrieval. If recall at fifty is high and recall at five is not, the reranker is where the points are.

Recall tells you the passage is somewhere in the list, while MRR tells you whether it will survive context trimming. A system reporting recall at twenty of 0.95 alongside an MRR of 0.2 is retrieving the right thing and then burying it, and the two numbers together diagnose that in a way neither does alone.

Anthropic's contextual retrieval benchmarks, published in September 2024, give you a target to hold your own harness against. They measure the failure rate as one minus recall at twenty. Plain embedding retrieval missed 5.7%, and adding contextual embeddings, keyword search and a reranker took that to 1.9%.

Measure this before you touch generation quality, because the order is not a preference. At a recall of 0.6, four questions in ten are unanswerable before the model is ever invoked, so every prompt change you evaluate after that point is being measured against a ceiling you have not lifted. On Prospex, our own prospect-research engine, a single run reduced 420 collected items to the 6 that answered the question, and that reduction made the downstream answer quality worth measuring at all. Once those numbers are readable, which layer to change is the decision in RAG vs fine-tuning vs agentic search.

Tool use

This layer is consistently under-measured, and it is where agents fail quietly rather than loudly. Four separate questions need separate answers.

Sequence is the one teams omit, and it produces agents reporting work as done when a step in the middle failed silently.

Safety: guardrails and PII

A guardrail and an eval are different mechanisms and teams keep conflating them. A guardrail runs on every request and blocks or rewrites. An eval runs against a fixed set and tells you whether the guardrail works. You do not pick one. You build the guardrail and then measure it, and teams skip the measuring.

Three kinds of case go in the set, and the third is the one teams miss.

Cases the guardrail should catch. A request that would send personal data to a provider, an action outside the policy, an instruction buried in a retrieved document. These are the obvious ones.

Cases it should let through. Over-blocking is a real failure with a real cost, and a guardrail tuned only on the first kind of case will get quieter and quieter until the product stops working. If nobody is measuring the allowed traffic, nobody notices.

Cases where the sensitive data arrives from retrieval rather than from the user. A redaction layer that reads the request and not the retrieved passages is a layer with a hole in it, and the hole only opens once the corpus grows.

PII handling has a second measurement most sets miss. If the boundary encodes on the way out and decodes on the way back, the eval has to check both directions. Redaction that never decodes cleanly produces answers that are safe and useless, and that failure shows up as a quality complaint rather than a safety one, so it gets routed to the wrong team.

That last row makes the rest possible, because a guardrail that logs only its blocks cannot be measured: the cases you most need are the ones it let through and should not have.

The gates here are not like the others in this guide. Elsewhere we start at 0.90 and move it once a system has history. On safety the target is 1.0 and a miss is an incident, because the cost of one leaked record is not on the same scale as one wrong answer.

Generation

Notice that helpfulness does not appear, because it is not a measurement, and a judge asked to assess it tends to favour the longer answer.

When should you not build a judge?

You now know what to measure, so next comes what does the measuring, and for most of these checks it should not be a model. A judge is the expensive option, to build and then to run on every case forever, and teams reach for one earlier than the evidence justifies.

The mistake we see most often is a team building a judge for a failure they could have fixed that afternoon. If a mode appears eleven times in your traces and a prompt change removes it, you have spent nothing and gained the fix. Build a judge for it instead and you have added a per-case cost that runs for as long as the product does, all to keep measuring something that stopped happening. Judges earn their cost on the modes that survive several attempted fixes.

How do you write a rubric that holds up?

For the modes that survive, you need a judge, and a judge is only as good as the question you give it. Each mode becomes one question the rubric asks. Four things decide whether that rubric works.

Two of those are worth more than a table row.

Ask one question at a time. "Does every claim appear in the passages?" fails in a way you can act on, because you go and check the claims. "Is this answer good?" gives you a number that dropped and no way to tell whether retrieval, the citation, or a missing refusal caused it.

Binary is harder to agree on than it looks, and that is the point. Two people will argue about where the line sits, and the argument is worth having. A team that cannot say whether an answer passes does not yet know what it wants from the system.

A three-way split is sometimes right, when a partial answer means something real. A five-point scale used as a gate rarely is. Ask two people to score the same answer and one says three and the other says four. A real change has to rise above that noise before anyone can see it, and that takes more labels than most teams will ever collect.

Our trademark platform is the case where a scale earns its place. Its harness scores every draft from 0 to 10 on six axes before an attorney sees it: structure, reasoning depth, citation accuracy, jurisdiction discipline, edge-case handling and tone. The scale works there because its main job is comparison. Production logging scores the base model and the fine-tuned one on the same inputs, so a regression from a model change shows up as a number that moved. The attorney's corrections stay the most valuable signal in the system, because they are the only measure of the gap the automated scoring cannot see.

Here is a rubric that came directly out of the failure note in the table above:

You are checking whether a cited clause governs the question asked. QUESTION: {question} PASSAGES: {passages} ANSWER: {answer} For each citation in the ANSWER, decide whether the cited clause is the one that governs the QUESTION, rather than a related clause on the same topic. - valid every citation governs the question asked - invalid at least one citation is topically related but not controlling A correct number quoted from the wrong clause is invalid. Ignore style, tone, and length entirely. Return JSON: {"verdict": "...", "offending_citations": ["..."]} 

You could not write that rubric without one thing someone noticed first: a right number pulled from the wrong clause is the case that hurts, because a reviewer skimming the output will not catch it. That came from a domain expert's critique. No off-the-shelf metric would have found it.

How do you know your judge is any good?

You measure it against the expert's labels, and the number most teams reach for first is the wrong one.

Suppose your set contains two hundred cases of which a hundred and eighty pass, and your judge agrees with the human labels ninety percent of the time. That sounds like a working judge. Now consider a judge that returns pass for every input without reading anything at all, which also scores ninety percent on that set. Raw agreement cannot distinguish between the two, because on an imbalanced set the majority class carries the score entirely. This is a common way for a team to convince itself a judge works. The number is high, it is easy to compute, and it stays high no matter how badly the judge handles the cases that matter.

Here is that "always pass" judge on the 200-case set:

Raw agreement is 0.90, true positive rate 0.00 and true negative rate 1.00. The judge missed 20 of 20 real failures, and every one of those ships. Lower the share of failures in the set and agreement climbs on its own. That movement is the set getting more imbalanced, not the judge getting better.

Measure the two error rates separately

The true positive rate is recall on failures, meaning of the cases the expert failed, how many the judge caught. The true negative rate is its mirror, meaning of the cases the expert passed, how many the judge also passed. Report them separately and the judge's behaviour becomes legible, because the two failures cost very different amounts.

from sklearn.metrics import confusion_matrix, cohen_kappa_score # 1 = expert failed the case (the thing we care about catching) expert = [1,0,0,1,0,0,0,1,0,0] judge = [1,0,0,0,0,0,0,1,0,1] tn, fp, fn, tp = confusion_matrix(expert, judge).ravel() tpr = tp / (tp + fn) # 0.67 caught 2 of 3 real failures tnr = tn / (tn + fp) # 0.86 passed 6 of 7 good answers print(cohen_kappa_score(expert, judge)) # 0.52 

A low true positive rate is the expensive failure. A judge at 0.9 TPR and 0.7 TNR costs an engineer ten minutes on a false alarm, while the reverse lets a real defect reach a user. When the two trade against each other, we bias toward catching failures.

Kappa is still worth computing as a single summary, because it corrects for chance agreement in a way raw accuracy does not.

How many labels, and when to stop

Start at thirty and keep going until you stop learning. For validating a judge specifically, a hundred to a hundred and fifty labels with failures deliberately oversampled is the working range, and the oversampling is not optional. If ten percent of your traffic fails, a uniform sample of a hundred gives you ten negative examples. That puts an interval around your true positive rate wide enough that the number cannot support a release decision.

Below 0.4, check the humans before the judge

Do not begin tuning the judge prompt. Have a second person label thirty cases and compute human-to-human agreement first. If two domain experts only agree at 0.5, no judge will beat that, and what you have found is an underspecified task rather than a broken evaluator. That is a product problem, and no amount of prompt engineering resolves it.

When it fails, split the rubric before touching the model

A judge scoring around 0.45 is usually being asked a compound question, and the fix is to decompose it into separate calls that each ask one thing:

1. Does every factual claim appear in the passages? grounded | not_grounded 2. Does each citation govern the question asked? valid | invalid 3. Given the passages, should this have refused? should_answer | should_refuse 

Each of those is close to mechanical, and the results are compounded afterwards in code. This is usually the largest single improvement available, and it requires no prompt tuning beyond the split itself.

Three biases to expect

Judges prefer longer answers, so if a change made your outputs longer, some of the apparent improvement is an artefact; bucket by response length and check that the score holds within each bucket. In pairwise comparison one position, often the first, wins more often than it should, so run both orders and average. And a judge prefers text from its own model family, so if one provider both writes and judges, score a sample of cases with a judge from a different provider and look for disagreement concentrated on your own model's output.

Re-run the alignment check whenever you change the judge model, the rubric, or the task itself. Provider updates move judge behaviour silently, and nothing in your pipeline will announce that it has happened.

On RepliCoach, our own sales-roleplay build, we kept the model out of the number altogether. The model writes the coaching feedback and a versioned rubric sets the score. If the model both writes the feedback and sets the number, a model upgrade shifts every historical comparison, and nobody can tell whether reps improved or the scale moved. Raw model scores also drift between personas and between conversation lengths, so the build normalises them and watches the score distribution for each persona as the sign that the normalisation needs revisiting.

How do you stop overfitting the set?

You will overfit it, because the set is small, you read the failures, and you fix precisely those. Splitting it is the standard defence and it works here for the same reason it works anywhere else.

When the dev split keeps improving and the test split does not move, you are tuning to the cases rather than to the task, and that divergence is invisible without the split. Beyond that, retire cases the system has passed for three months and replace them with fresh production failures, because a set that never changes is measuring a system that stopped changing.

Is the difference real?

Usually not, at the set sizes teams actually have. Eval sets are small and small sets are noisy, and a hundred cases scoring 0.90 carry a confidence interval of about six points either side. A run at 0.91 and a run at 0.89 are therefore the same result as far as the evidence goes, and teams ship on that difference every week.

Comparing the two runs as separate samples makes the problem worse than it has to be. Detecting a move from 0.89 to 0.91 that way needs somewhere near three and a half thousand cases, which no team building a product is going to label.

Both runs went over the same cases, though, so they were never separate samples to begin with. Throw away every case that scored the same in both and count only the ones that changed verdict.

Six cases moving the same way is enough at the usual five percent threshold, because a change that did nothing would have split them roughly evenly. That is also why nine better against three worse is weaker evidence than six against none, though both leave you six cases ahead. This is a sign test and any statistics library will run it. It is also the second reason to write per-case results instead of an aggregate, since the aggregate is the one number that cannot tell you whether the move was real.

Two limits are worth holding on to. The test tells you the direction of the change is real, and says nothing about whether the size of it is worth shipping. And thirty cases can only produce six flips if the change touched a fifth of them, so a small set still hides small improvements. A two-point move needs at least three hundred cases even when every flip goes the same way, and more when some go the other way, so the better question is why a two-point move is a release decision at all.

We hold our own case studies to the same standard. We put a figure on a case study when it was measured the way this guide describes.

What does this look like in CI?

Deliberately ordinary, which is the point: a harness nobody runs is worth nothing, so the eval suite goes where every other check goes and fails a build the same way.

# .github/workflows/evals.yml on: pull_request: paths: ["prompts/**", "retrieval/**", "chunking/**", "tools/**", "evals/**"] jobs: evals: steps: - run: uv run evals --set evals/dev.jsonl --report out/report.json - run: uv run evals-gate out/report.json # fails if: # recall@20 < 0.90 # groundedness_tpr < 0.90 # refusal_recall < 0.98 (missed refusals are the expensive failure) # p95_latency > 2500ms 

The path filter is the part worth paying attention to, because chunking configuration and retrieval settings change output quality as much as the prompt does. A filter that only fires on prompts/** will let the most expensive changes through ungated.

Three practices matter more than which LLM evaluation framework you choose. Version the eval set in the repository, because it is source: it gets reviewed, and you can bisect it. Write per-case results rather than only an aggregate, because you need to diff two runs and see which five cases flipped, and a score moving from 0.91 to 0.89 tells you nothing about where to look. And pin everything, meaning temperature at zero where the model still accepts the setting, pinned model versions for both the system and the judge, and those versions recorded in the report, since otherwise you cannot attribute a regression to your own change.

Temperature zero plus pinned versions reduces variation and does not remove it, so a case that has to come back the same still needs the repeat-run check in why LLM temperature 0 is not deterministic.

Our DevOps agent runs the same idea inside the product. A validation loop checks every generated template before it reaches AWS, and template errors before deploy fell from around ten percent to four across that loop's introduction. Every validation failure and its revision is recorded, and those records are the input to improving the loop rather than the prompt. On the trademark platform the harness keeps a fine-tuned model changeable at all. Without it nobody would touch a model an attorney has signed off on, and the system would sit frozen at whatever quality it launched with. When the model id itself changes, as it does with the GPT-4o deprecation, the migration runs this same harness on the current model and the replacement before traffic moves.

Where does online evaluation fit?

Online evaluation tells you whether your offline score means anything. Everything up to here is offline evaluation. You wrote the cases, an expert wrote down the right answers, and CI scores one against the other on every pull request. The other half is online evaluation, which watches the shipped system answer questions nobody wrote an answer key for.

Most teams build the first column and stop there, which is where this guide has left you, and it sits awkwardly against the claim the guide opened with: validation of these systems is only feasible during operation.

Your offline score is a proxy for how the system behaves in front of users, and a proxy needs checking. The check is short to state: when the eval improves and nothing moves in production, your cases do not represent your traffic. That is a finding about the set rather than about the model, and the response is to go back to error analysis with fresh traces instead of tuning the prompt again.

So you need a production signal to compare against, and ratings will not give you one. Almost nobody clicks the thumb, and the ones who do are the angriest and the most delighted. A study of the WildChat and LMSYS conversation logs found this feedback works as a lens on your users rather than as a clean training target (User Feedback in Human-LLM Dialogues, EMNLP 2025).

Behaviour arrives at volume instead: did the user ask the same question again in different words, abandon the session, hit retry, edit the answer before sending it, or escalate to a human? Any one of them is ambiguous, since someone copying an answer might be satisfied or might be collecting evidence for a complaint. Read against a baseline and in aggregate they move when quality moves, which is the only property you need from them.

Those signals also decide which traces you read. You get thousands a week and can read a hundred, so something has to choose which hundred. Take the traces carrying a negative signal, the outliers with unusual retry counts or session lengths, and the ones your judge scored badly. Then keep a slice of plain random traces in every batch, because the random slice is the only part that can find a failure mode your current signals do not describe. Ten to twenty traces a week keeps the reading going between rounds, plus a hundred fresh ones each time you run full error analysis.

On the DevOps agent this was easier than it usually is. The signal was mechanical: the validation loop checks each generated template before deploy and records every failure with the request that produced it, so the system keeps its own record of what the model gets wrong without anyone deciding a failure was one. Most products do not get that, so pick two behavioural signals and instrument them before you need them. Treat the gap between those and your eval score as the thing you are m...