Our nightly evaluation gate went 0/55 three nights in a row. The third time, the product was completely fine and the harness was the one lying. Telling those apart is the hardest testing problem we have hit this year, and the discipline that solved it is simple to state: no fix gets pushed until the red run has a named cause with evidence.
The setup. We sell conversational AI employees, so the thing that must work is a conversation. The nightly harness provisions a real employee through MCP for each of 55 cases across 6 suites (booking, student FAQs, email orders, ERP lookups, log review, YouTube backoffice), replays the conversation against production, and asserts two things per case: the right tools ran, and a judge model scores the reply against a rubric. The gate fails below 80% on product-backed cases. This is evaluation-driven development: the cases are written from the use case's intent, and the nightly is the reference curve, not the target.
Night one: 0/55, cause named. Every conversation died with a 401 on turn one. The client sent the standard X-Api-Key header; the auth code only matched a converted-underscores variant (x_api_key) because of how the framework lowercases parameter names. One customer-facing symptom, one line of auth code, every external client broken. Fixed, and a regression test now covers all three accepted spellings.
Night two: 0/55, a different cause. The policy engine's gate hook read arguments from a function signature that had been refactored away. Every reply became "name 'kwargs' is not defined". Two things made this one instructive. First, the error was invisible in the product: users just saw the fallback reply. Second, we had a day earlier merged a fix whose tests were green, because no test imported the module that broke. The suite now has a static check that fails when a handler references variables its own signature does not define.
Night three: 0/55, and the product was fine. Probing by hand, a booking conversation worked end to end: the agent checked the calendar, offered slots, booked, confirmed. But the gate still said 0.304. Reading the transcripts line by line found three harness defects stacked on top of each other. The fixtures (calendar events, client catalogs) were injected into the system prompt, so the agent never needed to call a tool, and tool-expecting cases failed with zero calls. A skill assignment wrote one table while activation read another, so assigned skills were silently inert. And the response contract changed shape, so the gate was reading evidence from an empty field. None of these were product bugs. All of them made the product look dead.
The discipline that paid. Each red run got a named cause with evidence before any fix was pushed: an API probe that reproduces the failure in isolation, or a transcript line that proves the claim. The alternative is what we did the first day: two theories, two pushed fixes, both wrong, four CI cycles wasted. Reading the actual log before theorizing sounds obvious. Under a red nightly at midnight it is not obvious at all; it is the only thing that works.
What the gate measures now. With the harness fixed, the number finally means what it says: the 0.304 pass rate measured a real gap, the gap between what the product serves today (booking, email handoff, lead capture) and what the case suites contract (ERP flows, invoicing, scheduled reports). That is not overfitting, that is a roadmap with a number on it. The gate only blocks on the product-backed suites, and the full 55 keep running as the curve.
If you are building anything conversational, the takeaway is one sentence: separate the judge from the judged, and never let either of them skip a named cause. The product behind this one is Optimly, and the nightly curve is how it earns the right to ship on a Tuesday afternoon.
