You changed the system prompt. Is the bot better or worse? "I skimmed a few chats, and it looked fine" is not an answer.

This tutorial gives you a 1-file evaluation harness in plain Python. No evaluation framework, no extra dependencies, nothing to install. When you finish, 3 commands give you a before-and-after report you can paste into a pull request.

python eval_rag.py run v1 # edit your system prompt python eval_rag.py run v2 python eval_rag.py diff v1 v2

Requirements. Python 3.11 or newer. The code uses the match statement, which arrived in Python 3.10 (PEP 634), and the dict merge operator |=, which arrived earlier in Python 3.9 (PEP 584). Python 3.10 reached end of life on 1 October 2026, so 3.11 is the oldest version still receiving security fixes, and 3.14 or 3.15 is a better choice for new work.

What you will build

  • A golden set: 20 questions in a JSONL file, each with the facts a good answer must contain
  • A runner: sends every question to your bot and saves the replies
  • Deterministic checks: fact matching, forbidden phrases, and source citation
  • An optional judge model: catches what string matching misses, and flags answers for human review
  • A diff report: shows what a prompt change did, question by question

No dashboards. No vendor SDK.

Step 1. Write the golden set

A golden set is a list of questions with agreed answers. Write it with the person who owns the content — a support lead, a product owner, whoever maintains your help center. Do not write it alone with the developer who built the bot.

Each line in golden.jsonlis 1 item:

{"id": "refund-01", "question": "How long do I have to request a refund?", "expected_facts": ["14 days", "original payment method"], "must_not_contain": ["30 days", "store credit only"], "source_id": "kb-billing-refunds"} {"id": "sso-01", "question": "Does the Starter plan support SSO?", "expected_facts": ["Business plan", "not available on Starter"], "must_not_contain": ["yes, Starter includes SSO"], "source_id": "kb-plans-sso"} {"id": "export-01", "question": "Can I export my data as CSV?", "expected_facts": ["Settings", "Export", "CSV"], "must_not_contain": ["contact sales to export"], "source_id": "kb-data-export"}

That is sample data. Replace it with the answers for your own product.

The fields

Every field except id and question is optional, and the harness skips the checks you leave out.

Keep expected_facts short. "14 days" matches literally. "You have fourteen calendar days from purchase to request a refund" matches only itself.

What to put in your 20 items

  • The questions your users actually ask most
  • Refusal cases, where the bot should decline or pass the user to a person
  • Known traps: Questions where a wrong answer is still circulating, like retired pricing or a removed feature
  • Messy phrasing, because real users do not type clean questions

Step 2. The runner and the checks

Save this as eval_rag.py:

mport json import os import sys from pathlib import Path GOLDEN = Path("golden.jsonl") RUNS = Path("runs") USE_JUDGE = os.environ.get("USE_JUDGE") == "1" def load_golden(path: Path = GOLDEN) -> list[dict]: with path.open(encoding="utf-8") as f: return [json.loads(line) for line in f if line.strip()] def ask_bot(question: str) -> dict: """Placeholder. Replace with a call to your RAG bot. Must return {"answer": str, "sources": list[str]}.""" return {"answer": "I'm not sure, please contact support.", "sources": []} def check(item: dict, reply: dict) -> dict: answer = reply.get("answer", "").lower() sources = reply.get("sources", []) facts = item.get("expected_facts", []) forbidden_list = item.get("must_not_contain", []) source_id = item.get("source_id") found = [f for f in facts if f.lower() in answer] missing = [f for f in facts if f.lower() not in answer] forbidden = [p for p in forbidden_list if p.lower() in answer] cited = bool(source_id) and source_id in sources # A forbidden phrase is a hard fail, not a lost point.  if forbidden: return {"score": 0.0, "missing_facts": missing, "forbidden_hits": forbidden, "source_cited": cited} checks = [f in found for f in facts] if source_id: checks.append(cited) score = round(sum(checks) / len(checks), 2) if checks else 0.0 return { "score": score, "missing_facts": missing, "forbidden_hits": forbidden, "source_cited": cited, }

3 deterministic checks

  • Fact matching. Each expected fact found in the answer scores a point.
  • Forbidden phrases. Any hit scores the whole item 0. A forbidden phrase means the answer is wrong, not slightly worse.
  • Source citation. 1 point if the retrieved sources include the expected document.

The score runs from 0 to 1 and is the share of checks that passed. It is crude, but it is fast, free, and returns the same result every time, which is what a comparison needs.

The source check matters because it catches answers that are right but came from the wrong document, or from no document at all. That kind of correctness is luck, and the next prompt change will lose it.

Step 3. Add an optional judge

String matching cannot tell you whether an answer is complete, or whether the bot was curt with an angry customer. A second model can.

Add this to the same file:

JUDGE_PROMPT = """You grade answers from a customer support bot. Question: {question} Facts a correct answer must include: {facts} Bot answer: {answer} Rate the answer from 1 (wrong or unhelpful) to 5 (correct, complete, polite).""" JUDGE_SCHEMA = { "type": "object", "properties": { "score": {"type": "integer", "minimum": 1, "maximum": 5}, "reason": {"type": "string"}, }, "required": ["score", "reason"], "additionalProperties": False, } def call_judge(prompt: str, schema: dict) -> str: """Placeholder. Replace with a call to your LLM client, using its structured-output mode with `schema`. Return the raw JSON text.""" return '{"score": 3, "reason": "Judge not configured."}' def judge(item: dict, answer: str) -> dict: raw = call_judge( JUDGE_PROMPT.format( question=item["question"], facts=", ".join(item.get("expected_facts", [])) or "(none)", answer=answer, ), JUDGE_SCHEMA, ) try: data = json.loads(raw) score = int(data["score"]) except (json.JSONDecodeError, KeyError, TypeError, ValueError): return {"judge_score": None, "judge_reason": "unparseable", "needs_review": True} if not 1 <= score <= 5: return {"judge_score": None, "judge_reason": f"out of range: {score}", "needs_review": True} return { "judge_score": score, "judge_reason": data.get("reason", ""), "needs_review": score <= 3, }

Use your provider's structured output mode rather than asking for JSON in the prompt. All 3 major vendors now constrain the model to a schema during decoding: OpenAI calls it Structured Outputs, Anthropic calls it structured outputs, Google calls it response_format. Anthropic's documentation is blunt about the alternative, noting that even with careful prompting you can hit parsing errors, missing fields, and schema violations.

That is why call_judge takes a schema. Keep the try/except anyway — OpenAI's own documentation notes that structured outputs "can still contain mistakes", and Google advises validating values in your application regardless.

Why needs_review exists

A judge is still a model. It makes mistakes, it runs lenient, and it can be wrong while sounding certain.

The harness never treats its score as a decision. Anything scoring 3 or below, anything unparseable, and anything outside the 1 to 5 range gets flagged for a person.

The judge tells you where to look. A human decides.

Step 4. Run it and compare

Add the runner and diff commands at the bottom of the file:

def run(label: str) -> None: RUNS.mkdir(parents=True, exist_ok=True) out = RUNS / f"{label}.jsonl" with out.open("w", encoding="utf-8") as f: for item in load_golden(): reply = ask_bot(item["question"]) result = {"id": item["id"], "question": item["question"], "answer": reply.get("answer", ""), **check(item, reply)} if USE_JUDGE: result |= judge(item, reply.get("answer", "")) f.write(json.dumps(result, ensure_ascii=False) + "\n") print(f"Saved {out}") def load_run(label: str) -> dict[str, dict]: path = RUNS / f"{label}.jsonl" if not path.exists(): sys.exit(f"No run file at {path}. Run: python eval_rag.py run {label}") with path.open(encoding="utf-8") as f: return {r["id"]: r for r in map(json.loads, f)} def average(run_data: dict[str, dict]) -> float: if not run_data: return 0.0 return sum(r["score"] for r in run_data.values()) / len(run_data) def diff(a: str, b: str) -> None: v1, v2 = load_run(a), load_run(b) shared = [qid for qid in v1 if qid in v2] if not shared: sys.exit("No question IDs in common between the two runs.") print(f"| id | question | {a} | {b} | changed |") print("|---|---|---|---|---|") for qid in shared: r1, r2 = v1[qid], v2[qid] delta = r2["score"] - r1["score"] changed = "same" if delta == 0 else ("up" if delta > 0 else "DOWN") question = r1["question"].replace("|", "\\|") print(f"| {qid} | {question} | {r1['score']} | {r2['score']} | {changed} |") print(f"\nAverage: {average(v1):.2f} -> {average(v2):.2f}") only_1 = set(v1) - set(v2) only_2 = set(v2) - set(v1) if only_1: print(f"Only in {a}: {', '.join(sorted(only_1))}") if only_2: print(f"Only in {b}: {', '.join(sorted(only_2))}") regressed = [q for q in shared if v2[q]["score"] < v1[q]["score"]] if regressed: print(f"REGRESSIONS: {', '.join(regressed)}") sys.exit(1) if __name__ == "__main__": match sys.argv[1:]: case ["run", label]: run(label) case ["diff", a, b]: diff(a, b) case _: sys.exit("Usage: python eval_rag.py run <label> | diff <label_a> <label_b>")

To include judge scores, set the environment variable:

USE_JUDGE=1 python eval_rag.py run v2

diff exists with status 1 if any question scored lower than before, so you can run it in CI and have the pipeline fail on a regression.

Step 5. Read the diff

The diff prints a Markdown table you can paste into a pull request:

| id | question | v1 | v2 | changed | |---|---|---|---|---| | refund-01 | How long do I have to request a refund? | 0.67 | 1.0 | up | | sso-01 | Does the Starter plan support SSO? | 1.0 | 0.6 | DOWN | | export-01 | Can I export my data as CSV? | 1.0 | 1.0 | same | Average: 0.89 -> 0.87 REGRESSIONS: sso-01

The average gives you direction. The rows give you the story.

If the average rises but 1 item drops from 1.0 to 0.6, investigate anyway. Open runs/v2.jsonl, find the item, and read missing_facts and forbidden_hits. The cause is usually obvious: the new prompt shortened answers, and the bot dropped the menu path to the export setting.

Habits that keep the harness useful

  • Commit golden.jsonl to the repository, so changes to expected answers get reviewed like code.
  • Commit the run files for releases, so you keep a baseline.
  • Never edit the golden set to make a run pass. Edit it when the product changes.
  • Re-run on every prompt change, model change and knowledge base update. All 3 move answer quality without breaking a single unit test.

Known limits

This harness is small on purpose.

  • Substring matching is literal and blind to negation. "two weeks" will not match "14 days", so keep facts short and add accepted variants as separate entries. Worse, "You do not have 14 days" will match "14 days" and score as correct. Put the dangerous negations in must_not_contain.
  • It is single-turn only. It will not catch a bot that loses context after 3 messages.
  • Refusal cases need care. Give them an expected_facts entry containing the refusal wording you want, such as "cannot share". An item with no checks at all scores 0.
  • 20 items is a smoke test, not coverage. Grow the set as real questions arrive.
  • The judge needs calibration. Review its passes as well as its flags.

When you outgrow it

Eventually you will want multi-turn scenarios, per-criterion scores and better reporting. Our comparison of LLM evaluation frameworks covers the options and what each costs to set up.

If you would rather have a team build the golden set with your domain experts and run the evaluation as part of your releases, look at our LLM evaluation services.

Until then: 20 questions, 1 file, 2 runs. That already beats "it looks fine".