Eval harness cover

Most AI pilots never ship. A study Code Ninety published this year asked 210 technical leaders what happened to their AI pilots: only 31.4% reached production. Another 48.6% were still stalled after six months, and 20% were abandoned.

The interesting part is what separated the winners. Teams that shipped were far more likely to have automated evals and guardrails (66.7% of shipped pilots vs 12.1% of failed ones) and success metrics defined up front (78.8% vs 18.2%). The study is correlational and the publisher sells AI services, so treat the exact numbers with care. But the pattern matches what I see building agents for clients: a pilot stalls when nobody can say, with a number, whether the new version is better than the old one.

So here is the smallest thing that fixes that: an eval harness that gives you a pass rate and a ship or block decision on every change. It fits in 50 lines of Python.

What we are testing

Say you have a customer-facing agent: it answers questions, books appointments and handles refund requests by calling tools. You change the prompt on Tuesday. Did you break anything? Without evals, someone chats with it for ten minutes and says "looks fine". That is how pilots die quietly.

Each eval case says what must happen, not what the exact words must be:

  • which tool the agent should call (or none)
  • patterns the reply must contain
  • patterns the reply must never contain
  • a latency budget per turn

The code

import re, sys, time # The agent under test. In production this calls your LLM with your prompt and tools. # Here it is a stub so the harness runs anywhere; swap in your real agent function. def agent(message: str, version: str) -> dict: t = time.perf_counter() m = message.lower() if "refund" in m: reply, tool = "I can help with that. Let me check your order first.", "lookup_order" elif "book" in m or "appointment" in m: reply, tool = "Sure, what day works for you?", "check_calendar" elif "price" in m: reply = "Plans start at $49 a month." if version == "v1" else "Plans start at $49 a month, and I can guarantee a discount." tool = None else: reply, tool = "Could you tell me a bit more?", None return {"reply": reply, "tool": tool, "ms": (time.perf_counter() - t) * 1000 + 180} # +180 ms fake model latency # Each case says what must happen, not what the exact words must be. CASES = [ {"msg": "I want a refund for order 1182", "tool": "lookup_order", "must": [r"order"], "never": [r"refund (is|has been) (issued|processed)"]}, {"msg": "Can I book an appointment Friday?", "tool": "check_calendar", "must": [r"\?"], "never": []}, {"msg": "What is the price?", "tool": None, "must": [r"\$\d+"], "never": [r"guarantee"]}, {"msg": "hello", "tool": None, "must": [r"\?"], "never": [r"\$\d+"]}, ] MAX_MS, PASS_BAR = 600, 1.0 # latency budget per turn, and the share of cases that must pass to ship def check(case, out): fails = [] if out["tool"] != case["tool"]: fails.append(f"tool {out['tool']} != {case['tool']}") fails += [f"missing /{p}/" for p in case["must"] if not re.search(p, out["reply"], re.I)] fails += [f"forbidden /{p}/" for p in case["never"] if re.search(p, out["reply"], re.I)] if out["ms"] > MAX_MS: fails.append(f"{out['ms']:.0f} ms > {MAX_MS}") return fails def run(version): passed = 0 for case in CASES: fails = check(case, agent(case["msg"], version)) passed += not fails print(f" {'PASS' if not fails else 'FAIL'} {case['msg'][:34]:<34} {'; '.join(fails)}") rate = passed / len(CASES) print(f"{version}: {passed}/{len(CASES)} passed ({rate:.0%}) -> {'SHIP' if rate >= PASS_BAR else 'BLOCK'}\n") return rate >= PASS_BAR if __name__ == "__main__": ok = all([run(v) for v in sys.argv[1:] or ["v1"]]) sys.exit(0 if ok else 1) 

Running it

v2 is the "small prompt tweak" someone made to sound more persuasive. Run both:

$ python evals.py v1 v2 PASS I want a refund for order 1182 PASS Can I book an appointment Friday? PASS What is the price? PASS hello v1: 4/4 passed (100%) -> SHIP PASS I want a refund for order 1182 PASS Can I book an appointment Friday? FAIL What is the price? forbidden /guarantee/ PASS hello v2: 3/4 passed (75%) -> BLOCK exit=1 

The tweak made the agent promise a discount nobody approved. A human tester chatting casually would likely never ask about price. The harness caught it in under a second, and the non-zero exit code means your CI pipeline refuses to deploy it.

Why this shape works

Assert behaviour, not wording. LLM output changes every run. "Calls lookup_order and never claims the refund is issued" is stable. "Replies with exactly this sentence" is not.

The never list is where the money is. Most expensive agent failures are the agent saying something it should not: promising refunds, quoting the wrong price, giving legal or medical advice. Every incident you see in production becomes a new never pattern and a new case.

Latency is a test, not a dashboard. For a voice agent, a correct answer that arrives in two seconds is a failed call. Put the budget in the eval so a slower model or a bloated prompt gets blocked like any other regression.

The pass bar is a business decision. For refunds and payments I use 100%. For a FAQ bot, 95% on a few hundred cases might be fine. Writing the number down is exactly the "success metric defined up front" that the study found in 78.8% of shipped pilots.

Taking it to production

  1. Grow the cases from real traffic. Start with 20 cases, then add every bad conversation from logs. A few hundred cases covers most of what a narrow agent will see.
  2. Run each case several times. Models are not deterministic. Count a case as passed only if it passes, say, 3 out of 3 runs.
  3. Add an LLM judge for fuzzy checks ("was the tone polite?", "did it answer the question?"), but keep the hard rules as regex and tool checks. Judges drift; regex does not.
  4. Gate deploys on it. Run the suite on every prompt, model or tool change, in CI, before anything reaches a customer.

None of this is glamorous. It is also the difference between a pilot that runs for eight months "almost working" and one that ships in three.

If you have an AI product stuck in pilot and want it finished and tested properly, that is what we do at Axionry, an AI product development studio. We ship in weeks, and every agent we deliver comes with its eval suite.

Source