The migration looked like an obvious win on economics. In one EvalShift run, the candidate model was:

  • 58.8% cheaper
  • 75.2% faster
  • 0 failed or truncated calls

Run economics table: source and target each made 16 calls with no failures or truncations; the target cost $0.0741 against $0.1799 and averaged 773 ms against 3.1 s.

Run economics. Same 16 calls on both sides, no failures, and every number a finance reviewer cares about moved the right way.

Then the migration report returned:

FAIL

Not because the model crashed. Not because the API changed. Not because the output stopped parsing.

It changed what the agent actually did.

Migration verdict card: FAIL, 5 of 7 budgets within policy, 61.4% equivalent, 25.0% improved, 13.6% regressed, cost −58.8%, latency −75.2%.

The verdict card. Five of seven budgets passed; the two that did not are the two that describe behavior.

What failed

I replayed 16 real examples against the source and candidate models. Across the run, 13.6% of evaluated comparisons regressed.

The overall regression-rate budget actually passed.

Two other checks did not:

  • Tool-selection divergence hit 25%, against a 10% limit.
  • The overall equivalence rate fell to 61.4%, below the required 75%.

Report verdict and findings: the target breached max_tool_divergence at 25.0% against a 10% ceiling and fell below min_equivalence_rate at 61.4% against 75%; findings list searching instead of answering, confirming actions it never performed, and routing to the wrong tool.

The written verdict and its findings. The recommendation is not to migrate until the tool-selection and semantic regressions are fixed.

That distinction matters. Looking only at latency, cost, successful requests, or even a few manually inspected outputs would have made this migration look pretty attractive.

The behavioral diff told a different story.

The failures weren't cosmetic

One example asked for an opinion on working late.

The source model answered the question directly.

The candidate instead issued a search_web call and returned no text.

Per-example diff for the working-late question: the source called no tools and answered in two sentences; the target called search_web with the query "is working late worth it productivity well-being" and produced no final text. Flagged as TOOL_SELECTION_DRIFT.

The diff for that example. Source trace on the left, target trace on the right, and the reason it was flagged at the top.

Other cases had the same general problem in different forms:

  • an action was reported as completed even though the corresponding tool was never called;
  • a request was routed to a different tool;
  • an unnecessary tool call appeared where the source answered directly.

These are not necessarily signs that the candidate model is "bad." They show that changing the model changed the application.

For an agent, the model is part of the control flow.

Aggregate quality can hide this

The evaluator breakdown made the tradeoff clearer.

Routing arguments and routing conformance were broadly equivalent in this run, while tool-selection divergence, semantic similarity, and the pairwise equivalence judge showed regressions.

Per-evaluator table over 16 examples: routing divergence regressed (score delta −0.167, likely), routing args equivalent (+0.179, unclear), routing conformance equivalent (+0.047, unclear), the LLM equivalence judge regressed critically (−0.438, certain), and semantic similarity regressed (−0.214, likely).

Overall, by evaluator. Two rows say equivalent, three say regressed, and each comes with its own effect size, confidence interval and confidence label.

A single average score would flatten all of that into one number.

For a migration, I care more about the question:

What changed, on which examples, and is that change acceptable for this application?

This is why I built EvalShift

I kept running into model migrations that were tested roughly like this:

change model → try several prompts → outputs look fine → ship.

That works until the difference is something subtle like an extra tool call, a missing action, or different routing behavior.

EvalShift replays the same suite against the source and target model, compares the behavior, and produces the migration decision and individual diffs before the model string gets changed in production. The CLI is local-first and produces a self-contained HTML report; hosted upload is optional. How that differs from an observability tool or a prompt-eval runner is on the comparison pages.

The interesting result from this run wasn't that the candidate was worse.

It was that "59% cheaper and 75% faster" wasn't enough information to decide whether the migration was safe.