SokoFlow AI Build Log, Week 5 of 6: building an eval harness, getting rate-limited by my own test, and what a 90% really means

I built an exam for SokoFlow's AI layer, ran it, and scored 71%. The target was 85%.

So I did what you're supposed to do: I read the failures before touching any prompt. And some of the failures weren't the model's fault. The answer key expected a product UUID for stock questions, while the app looks products up by name. A shopkeeper says "sugar", not product_id=42. Part of my exam was wrong.

That's the week in one paragraph: an evaluation harness is software, and it has bugs too.

TL;DR

  • Week 5 built an eval harness for the intelligence layer only: the LLM, the Orchestrator and the executor's validation. Not tool execution, not the database.
  • The design calls: a case is a whole interaction, not one message; non-execution is judged on the decision, not the prose; tool selection, argument accuracy and overall pass rate are scored separately; each case carries labels on several independent dimensions instead of one category.
  • First clean run: 71.11%. After fixing the answer key, the tool descriptions and the date handling: 90% (27/30) on three runs in a row, at about $0.000069 per query against a $0.004 ceiling.
  • The caveats matter as much as the number: I tuned against the same 30 turns, the repair path never fired, and the margin over the 85% target is one turn.

Where we are

SokoFlow is a headless WhatsApp ERP for Kenyan SMEs. I've been building it for about five months. There's no app and no UI to learn; for a shopkeeper, the conversation is the interface. Phase 3 adds one AI feature: natural-language shop analytics. A shopkeeper asks "which products are running low?" or "how does this week compare to last week?", and an LLM, using function calling against SokoFlow's existing services, figures out which query to run.

Last week was robustness and guardrails, driven by one question: what happens when the system can't safely give the user what they asked for? Weeks 1–4 were mostly about making the system behave safely and predictably. Week 5 is about whether the intelligence layer actually behaves correctly. The principle for the week:

A system that executes tools successfully is not necessarily a system that understands what the merchant asked for.

The evaluation boundary

The spec set an eval pass rate of at least 85%. My first question: what exactly are we proving when we say that?

Imagine a merchant sends: "How much did I make in sales last week?" The AI should propose get_sales_summary with the correct date range for last week. But a lot happens on the way: the LLM interprets the question, the ToolExecutor validates the proposal, the backend runs the tool, and the formatter writes the reply.

Do I have to test all of that? No. This eval measures the intelligence layer, not tool execution or database uptime. That was the first boundary I drew:

EVALUATION ENTRY Merchant query + trusted evaluation context ↓ LLMClient interpret merchant intent, propose a tool call ↓ Orchestrator tool selection, argument validation flow, bounded repair ↓ EVALUATION EXIT ToolExecutor validation "Does the final proposal resolve to an allowed tool and satisfy its argument contract?" OUTSIDE PRIMARY EVALUATION Tool execution → Services → Database → Formatting → WhatsApp delivery 

What is an evaluation case?

Next, the unit of measurement. SokoFlow handles multi-turn conversations, which made this hard. Consider:

  1. "How much did I sell today?"
  2. "How much did I sell yesterday?"

Are those two independent cases, or is the second a continuation of the first? I chose to treat a case as a complete interaction with whatever conversational context is needed to interpret it. One interaction may contain several merchant messages, and each message is an evaluation turn with its own expected behaviour.

The reason: if I evaluate message 2 on its own, I'm testing something different from production. In production the system has to understand that the merchant is continuing an analytics conversation, that the previous successful call provides context, that "yesterday" modifies the previous request, and that the new arguments must reflect yesterday, not today. That's a legitimate part of the intelligence layer.

Evaluate the decision, not the prose

A pass for a tool-calling case is simple: the selected tool and arguments match the expected ones. But SokoFlow also has a non-execution path. Imagine a merchant asks "What is the weather today?" That isn't something SokoFlow exposes, so the correct outcome is no tool call.

There's a catch. The model could return no tool call and an empty or malformed response. Or it could return no tool call and a sensible explanation that weather isn't supported. Should both get credit just because neither called a tool?

My first idea was to use the raw_response field on the LLMResponse schema to tell those two apart. I got skeptical fast. That field holds the raw provider response, for debugging and logging. Making evaluation depend on it would tie the eval to provider-specific structures and undermine the provider-independent LLMClient contract from Week 2. This is one more place where measuring AI quality is harder than checking whether a function returns the expected value.

For the weather example, expected behaviour is "no analytics tool call", and there are four possible outcomes:

Model behaviour Interpretation
No tool + relevant explanation Correct
No tool + generic but meaningful explanation Potentially correct
No tool + empty or malformed response Incorrect
Analytics tool call Incorrect

The hard part is deciding whether an explanation is relevant. One fix is a semantic classifier to judge arbitrary natural-language replies, but that could become a project of its own. So the decision was simple: evaluate the decision, not the prose. For expected non-execution cases, a pass needs:

  1. No tool call.
  2. A non-empty, structurally usable normalized response.

No semantic classifier. raw_response stays diagnostic material, not an application-level dependency.

Separate scoring criteria

I also saw a chance to get diagnostic data, not just a score. Suppose the expected tool is get_sales_summary, and the model picks get_sales_summary but supplies the wrong date range. Is that a successful tool selection with failed argument generation, or a complete failure? If I collapse it into one number, I can't tell where the model goes wrong.

So there are three metrics: tool-selection accuracy, argument accuracy and overall pass rate. The week's KPI is the last one:

PassRate = FullyPassedCases / TotalEvaluationCases × 100 
  • Tool-selection and argument metrics are diagnostic.
  • Overall pass rate is the authoritative Week 5 KPI.
  • Partial successes don't count as overall passes.

Repair attempts

SokoFlow has a bounded repair: at most one extra model attempt for specific errors (the retry rules from Weeks 1 and 4). What if the model proposes invalid arguments, then returns the right tool and arguments after the Orchestrator allows one repair? Is that a pass?

Yes. I want to evaluate the system as designed, and repair is part of the production architecture. If it recovers from an invalid first proposal within its limit, the system behaved correctly. But for diagnostics I record first-attempt success and repair-recovered success separately, so I can see how often the model needs the safety net.

A dataset the model can't grade itself on

I already had one source of cases: the InteractionLogger from Week 4. But I needed 25–40, so I needed more. And that raised a question: what makes a dataset independent?

Imagine asking an LLM to generate 30 merchant queries with expected tool calls, then evaluating the same model on them. The same model is author, judge and jury, and the score gets inflated. So I settled on a hybrid:

  1. A different AI family (Gemini 3 Pro) to draft cases, plus human review.
  2. Real interactions from the InteractionLogger.
  3. Human-authored expected outcomes.

Representative coverage

A representative dataset isn't one with many different questions. It's one that exposes the important ways the system can succeed or fail in realistic use. I focused on three things:

  1. Capability coverage: does it exercise what the analytics system is supposed to do?
  2. Failure-mode coverage: does it challenge the places where the architecture can break?
  3. Usage representativeness: does it look like how a real Kenyan SME merchant writes to SokoFlow?

Balancing those guards against an eval that gives reassuring scores while leaving big parts untested. A 100% pass rate doesn't necessarily mean a good system. It might mean I wrote an easy exam.

Coverage vs frequency

Representative coverage doesn't mean copying real traffic proportions. Suppose (hypothetically) 70% of real queries were sales summaries. I wouldn't give sales summaries 70% of the eval. This is an engineering evaluation: enough cases for common usage, and deliberate space for rarer capabilities and failure modes.

  • Production traffic tells you what merchants commonly ask.
  • The eval dataset tells you whether the system handles the things it's expected to handle.

Classification: a row, not a box

This was the hardest concept of the week. I wanted to classify each case along three dimensions:

  1. Business capability: sales summary, stock level, top products, trends, and so on.
  2. Linguistic complexity: straightforward, relative or explicit dates, multi-turn follow-ups.
  3. Expected behaviour: successful tool call (before or after repair), invalid-argument proposal, unsupported request, and so on.

What I got wrong first: I treated these as mutually exclusive categories, so each case went in one box. My reason was interpretability: when I inspect results, I should see which category is doing badly. The goal was right; the design was wrong.

Here's the mental model that fixed it. Imagine a database of people and the question "who in our company is struggling?" You wouldn't ask for each person either their age or their city or their job title. A person is 30 and lives in Nairobi and is an engineer. Those aren't competing choices, they're different questions about the same person.

A test case is the same. It has independent properties at once:

  • What business area does it test? → Sales summary
  • How tricky is the language? → A relative date ("this week")
  • What should the system do? → Call the tool successfully
  • How many turns? → Single turn

If I filed "How much did I make this week?" under only "Sales summary", then when failures spike I could never ask "are the failures concentrated in relative-date questions?", because I'd have thrown that label away when I filed it. Filing each case under one box lets me slice results one way and loses information. Giving each case a value in every column lets me slice any way I want.

The lesson: a classification dimension tells you where to look. The intersection of dimensions helps you form hypotheses about why something is failing.

Capability coverage

The seven-tool registry is the functional scope. Not every tool deserves identical representation, but every registered tool deserves coverage. Production traffic would have let me weight tools by real merchant usage, but I don't have it, so I used deliberate engineering coverage instead of pretending to know the distribution. Three layers:

  • Layer 1, individual capability (foundation): every tool gets at least one straightforward case testing basic selection and arguments.
  • Layer 2, interpretation (depth): variations in wording, relative periods, explicit ranges, product references and other argument challenges.
  • Layer 3, interaction (integration): combinations, multi-turn context, unsupported requests and repair scenarios.
Category Cases
get_sales_summary 8
get_top_products 4
get_product_price 4
get_stock_level 3
get_low_stock_items 2
get_sales_trend 2
get_slow_moving_items 2
Non-execution (no tool) 5
Total turns 30

Linguistic complexity

Consider three merchant messages:

  1. "Show me my sales today."
  2. "How much have I sold since Monday?"
  3. "Compare this month's sales with the previous month."

All three are about sales, but they make different demands. The first is straightforward, the second needs a relative time expression interpreted, and the third needs a comparison and its two periods. How do I represent that without turning the dataset into artificial grammar exercises? I used one filter for every candidate:

"Would a real merchant plausibly type this, and would getting it wrong hurt them?"

If yes, it's a candidate. If it's just a clever way to stress a parser, I left it out for this MVP.

Single-turn vs multi-turn

For interaction structure, I considered a conversation like this:

Merchant: How much did I sell yesterday? (turn 1) SokoFlow: Your sales yesterday were KSh 12,500. (previous system response) Merchant: What about the day before? (turn 2) Merchant: And this week? (turn 3) 

Turns 2 and 3 depend on context. The risk is building multi-turn cases that accidentally hand the model information it wouldn't have in production. The solution: construct every turn so the model sees exactly what production would show it at that moment. Raw messages, raw (even wrong) prior answers, production-shaped state, and never a cleaned-up or repaired transcript.

Making the evaluation trustworthy

Now I moved from "what is a correct answer" to "why should anyone trust this evaluation?" The core question: what makes it reproducible, so that a run today and a run tomorrow, with nothing changed, can be compared meaningfully? I listed what needs to stay constant:

Dimension What needs control
Dataset Exact evaluation cases and versions
Expected outcomes A frozen, independently derived answer key
Model Provider, model identifier and version where available
Prompt Exact system and developer instructions, and their versions
Tools Registry definitions, schemas and descriptions
Orchestration Repair policy, retry limits and execution configuration
Runtime Relevant application and dependency versions
Measurement Timing boundaries, cost calculation and aggregation rules

Model APIs can also be nondeterministic. Even with identical inputs and configuration I accept that outputs may differ, so reproducibility doesn't mean identical outputs. It means being able to reconstruct the experiment, measure the variation and explain what happened.

Another risk: the provider silently updating the model between runs. If that happens, a score shift might come from the provider rather than my prompts or code. So model identity is part of the evaluation baseline. To maximize predictability, temperature was set to 0.0.

What a run should record

I want to be able to open an old report six months from now, see that the pass rate fell from 90% to 75%, and reconstruct enough to investigate. So I record granular results, not just a total. A 75% tells you something went wrong, not what. An evaluation report should be an evidence trail, not merely a scoreboard. It covers three groups:

  1. Inputs: what entered the evaluation (dataset version or hash, application git commit, system prompts, inference parameters).
  2. Execution: what happened (exact messages and context sent to the model, the model request, the proposed tool call, the validation outcome).
  3. Outputs: what it produced (expected vs actual outcome, estimated cost, argument correctness).

Interpreting reliability

I added three measurements: mean pass rate (average observed performance), minimum and maximum (best and worst run), and standard deviation (how widely results fluctuate). I also track per-turn failure frequency, which separates persistent weaknesses from intermittent ones. A turn that fails three times in a row deserves different attention from one that fails once and passes twice.

Measuring latency

I separated user-perceived latency from component-level latency. For a shopkeeper, the total time across both inference attempts (first and repair) is what counts. For an engineer debugging, the initial model call matters too. And for elapsed time I use a monotonic clock, because it only moves forward at a steady rate and isn't affected by system time adjustments.

Cost

Another Week 5 target was cost per query at expected volume. First, the cost of one evaluation turn. Three options:

  1. Measure only the initial inference cost.
  2. Add up the cost of every inference attempt in that turn.
  3. Average across all evaluation turns, including turns that never needed a repair.

I chose 2 and 3. Option 1 only measures the happy path and hides the cost of a prompt that failed validation and had to be repaired.

C_turn = sum over i = 1..n of C_inference_attempt(i) C_average = sum of all C_turn / N_evaluation_turns 

The denominator includes turns that needed no repair. Repair is part of the system's real operating cost, not an optional expense to hide.

What if token usage is unavailable or unreliable? I record where each number came from:

Classification Meaning
Provider-reported Usage returned by the provider
Locally estimated Calculated from request and response data with a compatible tokenizer
Unavailable Not enough reliable information to calculate usage

A locally estimated token count isn't automatically an accurate billing estimate. Several articles I read note that different tokenizers count the same text differently.

Expected production volume

Now we leave the dataset and think commercially. The trap I nearly fell into: picking one number and treating it as fact. I first went with 100 queries a day:

100 queries × $0.0001 × 30 days = $0.30 per shop per month 

That's the temptation: turning an arbitrary assumption into an engineering requirement. A better approach is several usage scenarios, grounded in real product behaviour where possible:

Scenario Interpretation
Conservative Lower expected usage among active shops
Expected Most plausible usage based on available evidence
Power-user High usage from particularly active shops

I don't have production telemetry yet, so the inputs are intended merchant workflows, explicit usage assumptions, and (later) controlled pilot observations. Once real merchants use analytics, I'll replace assumptions with observed data: queries per shop per day, active shops, cost per query, repair frequency. Monthly model spend:

C = S × Q × D × c̄ S = active shops Q = daily queries per active shop D = operating days c̄ = average cost per logical query 

The budget ceiling

How do I set a defensible maximum cost per query? I borrowed from unit economics: expected revenue per shop, target gross margin, other service-delivery costs, merchant willingness to pay, and an explicit analytics budget.
But an average cost per query isn't enough for operational protection. That's where Week 4's limits come in, and each control has a different job:

  • Cost ceiling: what we can economically afford.
  • Rate limit: how quickly a shop can consume resources.
  • Daily query limit: how much a shop can consume in a period.
  • Token ceiling: how expensive a single inference can get.

Dev set and frozen headline set

This led me down a rabbit hole that mirrors the classic machine learning train/validation/test split. Suppose I run the headline evaluation and get 80%. Then I try different temperatures, prompt improvements, model changes and repair settings, and eventually find a configuration scoring 93%.

The danger: if I keep using the frozen headline dataset to pick the best configuration, I'm gradually optimizing against it. That 93% might measure how well I've learned those particular 30 cases, not how SokoFlow handles unseen merchant requests. This is evaluation-set contamination.

The fix is to split the data into a development set and a held-out test set, and enforce a gatekeeper rule: the test set runs only after a candidate configuration has already been chosen.

Putting the harness to the test

With the harness built, it was time to find out how SokoFlow actually performed. My target was an overall pass rate of at least 85%. The first runs reminded me that building an evaluation and running one reliably are different problems.

The first obstacle wasn't the model

It was Groq's rate limiting. I started getting HTTP 429 Too Many Requests. The evaluation was sending requests faster than my token budget allowed, and the retries weren't robust enough. Rather than treating it as a model-performance problem, I stepped back and made the evaluation infrastructure reliable:

  • Token-budget enforcement. Before each request, the client estimates token usage from the input character count and the configured max_tokens, to avoid exceeding the available budget.
  • A shared TokenBucket. Each LLMClient instance creates a shared token bucket, so requests through that client respect one common rate limit instead of each consuming capacity independently.
  • A proper 429 retry policy. _post_payload now detects 429s. It first reads the Retry-After header. If that's missing or invalid, it falls back to exponential backoff starting at evaluation_initial_backoff (1 second), doubling each attempt and adding 10% jitter. Retries are capped by evaluation_max_retries (default 3), and each one is logged with its attempt number and wait time.

I wasn't adding retries to make errors disappear. I wanted the client to respect the provider's limits and behave predictably when it hits them. This 429 retry is for evaluation only. But in future, I will be considering moving this feature to production too. Automatic retries for provider transport failures.

71%, and reading the failures

With the 429s handled, the harness completed its runs, but the results weren't what I'd hoped. SokoFlow scored around 71%, below the 85% target:

================================================================ EVALUATION RESULTS SUMMARY ================================================================ Mean Pass Rate : 71.11% (Target: ≥ 85.0%) Pass Rate Min / Max : 70.00% / 73.33% Standard Deviation : 1.92% KPI Status : [NOT MET] Average Cost / Query : $0.000051 (Ceiling: $0.0040) Budget Ceiling : [MET] Tool Selection Acc. : 76.00% Argument Acc. : 64.00% Non-Execution Acc. : 100.00% Mean Turn Latency : 15420.2 ms P95 Latency : 17871.4 ms ---------------------------------------------------------------- 

The infrastructure was now reliable, but the tool-calling behaviour clearly needed work. I went through the failed interactions looking for patterns, instead of randomly changing prompts. What I found:

  • Some failures were my exam, not the model. Stock questions expected a product UUID, while the application supports name-based lookup.
  • Unclear instructions. The model was getting ambiguous guidance on ranking dimensions, sales trends, date arguments and what "last week" means.
  • Misclassified failures. An executor schema error isn't the same thing as failing to produce a tool call, and the evaluation needed to tell them apart.

I corrected the relevant expectations, clarified the tool descriptions and prompts, aligned the date-handling logic and scoring, and added focused tests around the retry and error-handling paths. I also fixed a missing import in the rate-limit retry path and improved its jitter behaviour.

The final result

Then came another set of live runs. Across three runs of 30 evaluation turns each, SokoFlow scored a 90% mean pass rate, with every run exactly 90% (27 out of 30):

================================================================ EVALUATION RESULTS SUMMARY ================================================================ Mean Pass Rate : 90.00% (Target: ≥ 85.0%) Pass Rate Min / Max : 90.00% / 90.00% Standard Deviation : 0.00% KPI Status : [ACHIEVED] Average Cost / Query : $0.000069 (Ceiling: $0.0040) Budget Ceiling : [MET] Tool Selection Acc. : 100.00% Argument Acc. : 92.00% Non-Execution Acc. : 80.00% Mean Turn Latency : 19553.8 ms P95 Latency : 21712.0 ms ================================================================ 

That's a real improvement from the 71.11% I recorded in v1.0.0 (the set of test queries, expected tools/arguments, and scoring expectations used for the run. If the dataset’s contents change, its version should be bumped. The hash can also reveal content changes even if someone forgets to update the version). and it was consistent across all three runs.
Average cost was within the $0.004 ceiling.

What still failed, and what the number doesn't prove

Three interactions failed in every run. Two were mismatches involving unexpected arguments (ranking_dimension and comparison_period), and one was an empty model response on a chitchat message. I documented them instead of pretending the system was perfect.

A few honest limits on the 90%:

  • It's first-attempt performance. The report shows no repair attempts were exercised, so the result doesn't demonstrate recovery through the repair loop. The repair path is designed and unit-tested, but this evaluation hasn't proven it. An issue to investigate with the harness itself.
  • I tuned against the same 30 turns. The failures I fixed came from this dataset, which is exactly the contamination risk I described above. 90% is a post-tuning number, and an unseen set would likely score lower. That is a future improvement in mind.
  • The margin is one turn. At 30 turns, 85% means 26 passes. I got 27. One more failure still passes, but two more would have missed.
  • Identical runs show repeatability, not generalization. Three runs at exactly 90% tell me the system behaves consistently on this set, not how it handles merchants' messages I haven't seen.
  • Latency. Mean turn latency was 19.5 s, with a p95 of 21.7 s. This is considerably higher than the Week 3 measurement, which recorded a p50 of 1.20 s and a p95 of 2.27 s. However, these measurements do not use identical timing boundaries. The Week 5 evaluation measures wall-clock time around LLMClient.generate, including local token-bucket waits and any HTTP 429 retry/backoff sleeps. Week 3 measured a different path: from conversation entry through to the formatted reply, whereas the evaluation harness uses a direct orchestrator path with a mock database. The measurements were also taken nearly a month apart, so differences in provider conditions may have contributed.

I investigated whether local rate limiting explained the increase. Although token-bucket waits are included in the evaluation timing, the harness runs requests serially, leaving time for the bucket to refill between calls. This makes repeated long waits from the local token bucket an unlikely explanation for the entire gap. I therefore treat the two figures as measurements from different execution paths, not as a direct performance regression comparison. The higher evaluation latency remains something to investigate separately.

What started as a run blocked by rate limits turned into a useful exercise in debugging infrastructure, reading model behaviour, tightening tool contracts and improving the evaluation itself.

What I'd do differently

Three things, from this week's own mistakes:

  1. Validate the answer key before the first run. The UUID-vs-name mismatch was a bug in the key that I could have caught by checking it against the tool schemas. Instead I found it by scoring 71%.
  2. Design the labels as a table from the start. I lost time filing cases into single categories before realizing each case needs a value on every dimension.
  3. Make the repair path testable. If no real case triggers a repair, I need cases where the first proposal is deliberately invalid, or I'll keep reporting results that never touch it.

What's next: Week 6, polish, docs and demo

Week 6 is the final stretch. Everything from the last six weeks has to come together into a finished, demonstrable project. The focus: a final pass over chaos and edge cases, bringing the build log up to date, recording a Loom demo, writing the Phase 3 blog post, and tagging the release.

The finish line:

  • Feature merged into main.
  • Demo recorded.
  • Blog post published.
  • Build log current through Phase 3.

Six weeks of building, debugging, testing and refining, and now it's time to bring it together. Stay locked in for the next build log.