Confidence-Routed Agents: Cheap First, Frontier on Escalation

Originally published at webofmike.com on 2026-11-02. The demo repo and every command in it were run before publishing.

Escalating every uncertain agent decision to a frontier model sounds like the safe default. On a 240-case benchmark it was also the worse one: routing by confidence and escalating only when a cheap model said it wasn't sure cost half as much as escalating everything, and scored higher.

jev-benchmark measured TypeSafe AI's Jev against Claude on the same agent tool-call risk task. jev-calibration went further and measured whether Jev's own confidence score could be trusted to make a routing decision, and found a specific, usable threshold: a confidence of exactly 1.000 was right 133 times out of 134 on 240 labelled cases. Neither post built the thing that threshold is for. "Escalate when confidence is low" is not a model feature, it is a two-call routing problem, and routing problems belong at a gateway. This is that gateway, built and run for real: themsquared/confidence-routed-agents.

What was built

A router calls Jev directly for every task (Jev is a typed-decision API, not a chat model, so it isn't an LLM backend agentgateway knows how to proxy). When Jev's confidence clears the threshold, its answer is the answer. When it doesn't, the router escalates to Claude Sonnet 5, and that one call goes through agentgateway v1.5.0 instead of straight to Anthropic.

 ┌──────────────┐ confidence >= threshold ┌─────────────┐ task ──────▶ router.py ├───────────────────────────▶ answer │ │ (calls Jev │ │ (cheap │ │ directly) │ confidence < threshold │ path) │ └──────┬────────┘ └─────────────┘ │ │ escalate ▼ ┌─────────────────────┐ budget OK ┌───────────────┐ │ agentgateway ├───────────────▶ Anthropic API │ │ (holds the real │ budget │ claude-sonnet-5│ │ Anthropic key, │ exceeded └───────────────┘ │ per-key USD budget,│ ──▶ HTTP 429 │ access log) │ └─────────────────────┘ 

Why the gateway sits only on the escalation half: that's the half with a shared, billable credential. Jev's key is the router's own, scoped to a typed-decision API with no spend risk from a runaway loop in the same way a chat model has. The escalation key is exactly the kind of credential this campaign has argued belongs behind a gateway since secretless-ai-agents: the router never holds it, agentgateway injects the real x-api-key on the way out, and the budget and the audit trail live in one place instead of scattered across whatever called the model.

The task set is the same 240 hand-labelled agent tool calls from jev-calibration: classify each one as readonly, destructive, privileged, or exfiltration. Reusing it instead of sampling a new set means these numbers are comparable to that post's, not just another unrelated benchmark.

The result that mattered

Six configurations against the identical 240 tasks, one live Claude Sonnet 5 call per escalation:

Two things here are worth stating plainly instead of smoothing over. The 0.85 row scores a hair worse than both neighbors (92.1% against 92.9% on either side), which at n=240 is noise from a handful of cases crossing the cutoff, not a real dip. Read the 0.70-0.95 band as roughly 92-93%, not as a precise function of the threshold.

The real finding is the last two rows against the bottom one. Routing at jev-calibration's own recommended cutoff, escalate anything below exactly 1.000 confidence, cost half of what escalating everything costs ($0.0846 against an extrapolated $0.169 for 240 all-Claude calls) and scored better, not worse: 93.3% against 91.7%. The cheap path is not a compromise that trades accuracy for savings here. Jev's answers at exactly 1.000 confidence were already shown, in the prior post, to be right 133 times out of 134. Sending one of those cases to Claude anyway does not improve on that; Claude's terse, non-reasoning JSON classification call turned out to be the less reliable of the two on the cases Jev was already sure about. Escalating them just spends money confirming an answer you already had right.

Going from 0.99 to 1.00 is its own small lesson: it adds 27 more escalations and about $0.02 more cost for zero accuracy change (93.3% both). Past a certain point, a tighter threshold just buys more spend, not more correctness. The threshold that is "tight enough" is a measured fact on a specific task set, not a number to default to 1.0 on faith.

The budget actually blocks

A threshold set wrong, or an upstream regression that makes Jev return uncertain answers more often, turns an "occasional escalation" path into "most requests hit the expensive model." That is exactly the failure mode a gateway budget exists to cap. Verified with a temporary $0.0002/24h budget, not the real $1.00 one that ships in the repo's config, to prove the mechanism without burning hundreds of calls:

call 1 -> HTTP 200 call 2 -> HTTP 200 call 3 -> HTTP 200 call 4 -> HTTP 429 {"error":{"message":"Budget exceeded","type":"rate_limit_error","code":"budget_exceeded"}} 

agentgateway's own log line for the rejected call:

2026-10-01T14:39:20.199122Z warn budget API key budget exceeded connection.id=4 request.id=4 api_key="router" budget="daily-usd" used=0.000216000 limit_unit="USD" limit_amount=0.0002 exceeded=true 

used and limit_amount are both computed from the three real calls that preceded the rejection, priced from agentgateway's own modelCatalog rates for claude-sonnet-5, not round numbers chosen to make the demo work.

Jev being unreachable does not stop the agent

The router's --no-jev flag simulates the cheap path being down: every task falls through to the Claude escalation path with no special-casing in the code, because "escalate" is already the router's only other branch. A production version of this would want the inverse check too (what happens if the frontier path is the one that's down), but the asymmetry is deliberate here: Jev being unreachable should degrade to "more expensive, still correct," not "the agent stops answering."

A gotcha worth keeping

One case in an earlier full-240 run came back from Claude with no content block at all, just a finish_reason and nothing else, at max_tokens: 128. This is the identical failure the jev-calibration harness already hit at the same token ceiling: Claude's internal reasoning consumed the whole response budget before producing the JSON answer. Raising the cap to 256 did not reproduce it across all 240 cases, and router.py checks for a missing or empty content field explicitly rather than assuming the key is always there. A single silently-dropped answer in an agent pipeline is worse than a slow one; code that assumes the happy-path response shape will eventually hit this.

What this does not cover

No frontier-vs-frontier comparison; Claude Sonnet 5 is the only escalation backend tested here. No measurement of sustained escalation load over days, only a point-in-time budget trip. And the $1.00/24h budget shipped in the repo's config is a demo value, not a sizing recommendation, you'd size that from measured escalation rate and real model pricing for your own workload.

The harness, the task set, the raw per-call results at every threshold, and the agentgateway config are in themsquared/confidence-routed-agents.

Frequently asked questions

When should an AI agent escalate a decision to a frontier LLM instead of a cheap model?

When the cheap model's own confidence drops below a threshold you have actually measured, not a guess. On a 240-case agent tool-call risk benchmark, escalating only cases where TypeSafe AI's Jev returned less than its maximum confidence (1.000) cut cost in half against escalating everything, and improved accuracy from 91.7% to 93.3%.

Does escalating every agent decision to a frontier model improve accuracy?

Not on this task set. Routing every case to Claude Sonnet 5 with no reasoning budget scored 91.7% (55/60). Routing by Jev's own confidence and escalating only the uncertain 45.8% of cases scored 93.3% (224/240) for about half the cost. The cheap model was more reliable on the cases it was already sure about than Claude was on the same cases.

How do you stop an agent's escalation path from running up an unbounded LLM bill?

Put a hard budget on the API key that calls the frontier model, enforced where the call actually leaves your infrastructure. This demo puts a $1.00/24h USD budget on agentgateway's escalation key; a temporary $0.0002 budget set for verification returned HTTP 429 with a real dollar figure in the log line on the fourth call, not a simulated one.

Does the agent's router need to hold the frontier model's API key?

No. In this design the router's only credential is a gateway-scoped key with no model access of its own. agentgateway holds the real Anthropic key and injects it server-side on the escalation call, the same secretless pattern as direct agent-to-LLM traffic, applied to the second half of a two-model pipeline instead of the whole thing.

Canonical version, with machine-readable markdown at https://webofmike.com/confidence-routed-agents/index.md: https://webofmike.com/confidence-routed-agents/