The problem
Every team I have seen wire an LLM into a production service eventually ships the same bug: somebody improves a prompt, the demo looks great, and a downstream consumer quietly breaks. The summarizer stops including the one field the ticket parser needs. The classifier starts answering "The sentiment is positive" instead of "positive", and the exact-match routing downstream falls over. The extraction step returns JSON wrapped in markdown fences, and json.loads throws at 2 a.m.
None of these are model-quality problems. They are contract problems. The model team changed the shape of the output, and nothing in the pipeline checked the shape.
The usual responses are both wrong-sized. Eyeballing a spreadsheet of outputs does not survive the second prompt change. Adopting a full evaluation platform, with its own config format, storage, and opinions, is a week of work for a service that has five prompts. What most services actually need sits in between: a small suite of deterministic checks, written in the same language as the service, that runs in CI and fails the build when an output contract breaks. This article builds one in about 150 lines of dependency-free Python.
The approach
Three design decisions keep the harness small and honest.
First, a check is just a function. It takes the model's response text and returns (passed, detail): a boolean and a human-readable explanation. No classes to subclass, no plugin registry. If you can write an assert, you can write a check.
Second, checks never raise. A malformed response is a failed check with a useful message, not a crashed test run. This matters more than it sounds like, and the gotchas section comes back to it.
Third, the harness knows nothing about model providers. It takes a callable, prompt -> response_text, and runs cases against it. In production that callable wraps your API client. In tests it is a stub with canned responses, which is how the example in this article runs with no API key and no network.
What this harness deliberately does not do is judge whether an answer is good. Deterministic checks verify contracts: required content, forbidden content, format, length, parseability. Semantic quality needs a different tool (human review, or a model-graded rubric with all its own caveats). Conflating the two is how eval suites turn into noise nobody reads.
Step 1: The check signature and the two simplest checks
Most production failures I have debugged were content failures: something required went missing, or something forbidden showed up. Two checks cover a surprising amount of ground.
CheckFn = Callable[[str], tuple[bool, str]] def contains_all(*needles: str) -> CheckFn: """Pass when every needle appears in the response (case-insensitive).""" def check(text: str) -> tuple[bool, str]: lowered = text.lower() missing = [n for n in needles if n.lower() not in lowered] if missing: return False, f"missing: {missing}" return True, "all present" check.label = f"contains_all{list(needles)}" return check def not_contains(*needles: str) -> CheckFn: """Pass when none of the needles appear (case-insensitive).""" def check(text: str) -> tuple[bool, str]: lowered = text.lower() found = [n for n in needles if n.lower() in lowered] if found: return False, f"found forbidden: {found}" return True, "none present" check.label = f"not_contains{list(needles)}" return check Each factory returns a closure with a label attribute, so failure reports can name the check that failed instead of printing a memory address. Matching is case-insensitive by default because models are inconsistent about capitalization and your contract usually is not.
Step 2: Format checks, where the real failures live
Format drift is the highest-value thing to catch, because format consumers are code, and code is unforgiving. Two checks cover the common cases: a full-match regex for label-style outputs, and a JSON check for extraction outputs.
def matches_regex(pattern: str) -> CheckFn: def check(text: str) -> tuple[bool, str]: if re.fullmatch(pattern, text.strip(), flags=re.DOTALL): return True, "full match" return False, f"no full match for {pattern!r}: {text.strip()[:60]!r}" check.label = f"matches_regex({pattern!r})" return check def valid_json_with_keys(*keys: str) -> CheckFn: def check(text: str) -> tuple[bool, str]: try: data = json.loads(_strip_code_fences(text)) except (json.JSONDecodeError, ValueError) as exc: return False, f"invalid JSON: {exc}" if not isinstance(data, dict): return False, f"expected JSON object, got {type(data).__name__}" missing = [k for k in keys if k not in data] if missing: return False, f"missing keys: {missing}" return True, "valid JSON with all keys" check.label = f"valid_json_with_keys{list(keys)}" return check Two details here are load-bearing. matches_regex uses fullmatch, not search: for a classifier, "positive" buried in a sentence is a failure, not a pass. And the JSON check strips markdown code fences before parsing, via a small _strip_code_fences helper. Whether you should strip fences depends on your consumer (more in the gotchas), but the harness should make that policy explicit rather than accidental. A max_words check, same shape, rounds out the set for length budgets.
Step 3: Cases and the runner
A case is a name, a prompt, and a list of checks. The runner calls the model once per case and collects failures instead of stopping at the first one, because in CI you want the full damage report from a prompt change, not a single red test.
@dataclass class Case: name: str prompt: str checks: list[CheckFn] = field(default_factory=list) def run_eval(model_fn, cases): results = [] for case in cases: response = model_fn(case.prompt) failures = [] for check_fn in case.checks: passed, detail = check_fn(response) if not passed: failures.append(f"{getattr(check_fn, 'label', 'check')}: {detail}") results.append(CaseResult(name=case.name, passed=not failures, failures=failures)) return results Step 4: A suite, a stub model, and a CI-friendly exit code
The suite below tests a small support-automation service: a summarizer, an extractor, a sentiment classifier, a factual answerer, and an explainer with a length budget. The stub model returns canned responses, three of which contain the exact failure modes this suite exists to catch. In a real deployment you would replace stub_model with a thin wrapper around your provider's client and keep everything else.
CASES = [ Case( name="summary_keeps_key_facts", prompt="Summarize this incident report in under 60 words: ...", checks=[contains_all("database", "rollback"), max_words(60)], ), Case( name="extraction_is_valid_json", prompt="Extract the contact from this signature block as JSON: ...", checks=[valid_json_with_keys("name", "email")], ), Case( name="sentiment_is_bare_label", prompt="Classify the sentiment of this review as one word: ...", checks=[matches_regex(r"positive|negative|neutral")], ), Case( name="facts_have_no_invented_links", prompt="When was the Eiffel Tower completed?", checks=[not_contains("http")], ), Case( name="explanation_respects_length_cap", prompt="Explain idempotency in at most 25 words.", checks=[max_words(25)], ), ] The main() function prints one line per case plus the failure details, then returns exit code 1 if anything failed. That exit code is the entire CI integration: run the suite on any pull request that touches a prompt, a model version, or decoding parameters, and let the build go red.
Step 5: Run it
Here is the actual output of the suite above (full code in eval_harness.py and run_evals.py):
[PASS] summary_keeps_key_facts [PASS] extraction_is_valid_json [FAIL] sentiment_is_bare_label matches_regex('positive|negative|neutral'): no full match for 'positive|negative|neutral': 'The sentiment is positive.' [FAIL] facts_have_no_invented_links not_contains['http']: found forbidden: ['http'] [FAIL] explanation_respects_length_cap max_words(25): 51 words > 25 2/5 cases passed Read the three failures the way you would in a real regression. The classifier drifted from a bare label to a sentence; every exact-match consumer of that label is now broken. The factual answerer invented a citation URL on a domain that does not exist; if your UI renders links, you are now sending users to a hallucinated site. The explainer blew its length budget by 2x, which is how a tidy UI card becomes a wall of text. Each failure names the check and shows the evidence, so the fix conversation starts from facts instead of vibes.
Real gotchas
Fenced JSON will flood you with false alarms if you let it. Models wrap JSON in code fences constantly, even when the prompt says not to, and even at temperature zero. If your consumer strips fences before parsing, your check should too; if your consumer does a bare json.loads, then the fenced output is a genuine failure and the check should be strict. The mistake is a harness whose policy differs from the production parser's. Mirror the consumer, exactly.
A crashed harness is worse than a red one. Early versions of suites like this tend to die on the first truly malformed output, a JSONDecodeError escaping out of a check, an index error on an empty response. In CI that reads as an infrastructure failure, and infrastructure failures get retried and ignored. Wrap every parse, treat empty responses as ordinary failures, and make the harness's own exit path boring. A suite people route around is worse than no suite.
Deterministic checks pass wrong answers happily. contains_all("1889") passes whether the sentence around it is right, wrong, or incoherent. For factual cases where the answer space is small, prefer exact-match or golden-answer checks. For open-ended cases, accept the limit: this harness guards contracts, and correctness sampling (a human reading twenty random outputs a week) is a separate, necessary habit.
One run proves little about a stochastic model. At nonzero temperature, a case that passes today is a coin you flipped once. For the checks that guard expensive failures, run each case three to five times and report a pass rate, failing the build below a threshold you choose. Keep the repetition count small and the case list shorter; a suite that takes twenty minutes will be the first thing deleted under deadline pressure.
Keep responses cached during development. When you are iterating on the checks themselves, you do not want to pay for fresh completions on every run. Cache responses keyed by model name, prompt, and decoding parameters, and add a flag to bust the cache when the point of the run is fresh model behavior.
Wrap-up
You now have a harness that catches the failures that actually page people: missing facts, forbidden content, format drift, invalid JSON, and blown length budgets, in about 150 lines with no dependencies. Graduate to a heavier evaluation platform when you need semantic scoring at scale, dataset versioning across teams, or statistical comparisons between model versions. Until then, the highest-leverage habit is procedural, not technical: every prompt change runs the suite, and a red suite blocks the merge. Contracts first, quality second, in that order, because the contract failures are the ones your users notice before you do.
