Route by Slack, Not by Price
Free capacity is only cheap when your deadline is longer than its queue. That single comparison decides more of your bill than any per-token price. Everything below is a way to measure the queue and then choose a pool per request instead of per month.
Agent loops make this bite. One user question fans out into a planner call, three tool calls, a summarizer, and a repair pass when JSON comes back malformed. On a token sheet each call looks affordable. What actually arrives at the other end is wall-clock time, and someone is sitting there waiting for it.
Separate the queue from the work
Most dashboards collapse a request into one latency number. That number is useless for routing, because it hides whether you are paying in queue time or in generation time. A free pool with a long queue and fast generation and a committed pool with no queue and slow generation look identical once you average them.
So log four timestamps per call: submitted, accepted by the pool, first token, last token. The gap between submitted and accepted is your queue. The gap after that is the work.
# timestamps.py -- illustrative, not executed import time def timed_call(submit, deadline_monotonic): t0 = time.monotonic() handle = submit() # returns once the pool accepts the job t1 = time.monotonic() # accept / dequeue stream = handle.stream() first = next(stream) t2 = time.monotonic() # first token rest = first + "".join(stream) t3 = time.monotonic() # last token return rest, { "queue_s": t1 - t0, "first_token_s": t2 - t1, "stream_s": t3 - t2, "slack_at_entry_s": deadline_monotonic - t0, } Convert every deadline to the same monotonic clock at request entry, or the slack column will drift with clock adjustments and quietly lie to you.
Turn the numbers into a lane
A lane is a pool plus the numbers you measured for it. You do not need a scheduler. You need a comparison.
# lanes.py -- illustrative, not executed from dataclasses import dataclass @dataclass(frozen=True) class Lane: name: str queue_p95_s: float service_p95_s: float retries_per_call: float # measured, not guessed def expected_s(self, safety: float = 0.25) -> float: base = self.queue_p95_s + self.service_p95_s return base * (1.0 + self.retries_per_call) * (1.0 + safety) def admission(slack_s, free, committed): if slack_s >= free.expected_s(): return free if slack_s >= committed.expected_s(): return committed return None # shed or degrade: queueing here just spends the caller's deadline The third branch is the one people skip. If neither lane fits, waiting is not a strategy, it is a slow failure that also costs tokens. Shedding, returning a cheaper partial result, or telling the caller to poll later is honest. Silently adding four seconds is not.
Note the retry term inside the expectation. A retry inherits the remains of the original deadline, not a fresh one, so a call that failed once is a worse candidate for the slower pool than the same call was before it failed. That asymmetry is easy to miss when retries live in a wrapper nobody reads.
Check it before you believe it
The routing rule is only as good as the percentiles under it, so run the same prompt set twice: once sending everything to the slow lane, once routed by slack. Compare deadline miss rate per lane and total calls, not just average latency. Averages hide exactly the tail that breaks user trust.
# harness.py -- illustrative, not executed import asyncio, time async def one(prompt, deadline_s, route, lanes, submit): t0 = time.monotonic() lane = route(deadline_s, lanes["free"], lanes["committed"]) if lane is None: return {"lane": None, "miss": True, "calls": 0} text, usage = await submit(lane.name, prompt) return {"lane": lane.name, "miss": (time.monotonic() - t0) > deadline_s, "calls": usage.get("attempts", 1)} async def sweep(prompts, deadline_s, route, lanes, submit, n=200): results = await asyncio.gather( *(one(prompts[i % len(prompts)], deadline_s, route, lanes, submit) for i in range(n)) ) return results Run it with two flags and the same seed, and read the miss rate per lane rather than a combined score.
# illustrative invocations; substitute your own submit() implementation python harness.py --route all-free --n 200 --deadline 4 python harness.py --route by-slack --n 200 --deadline 4 If the routed run has the same miss rate as the committed-only run while pushing a real share of calls to the slow lane, you have earned the right to route. If the miss rate climbs, your percentiles were optimistic, and no amount of prompt tuning will fix a queue you did not measure.
The decision, written down
| Workload | Slack at entry | Route | Reason |
|---|---|---|---|
| Interactive turn, human typing | shorter than the fast lane's p95 | committed | the slow lane's queue alone eats the budget |
| Agent tool call, human still waiting | roughly twice the slow lane's p95 | slow lane | slack absorbs one tail spike |
| Nightly re-summarize of a backlog | hours | slow lane | the deadline is a maintenance window, not a person |
| Retry after a failed attempt | already halved by the first try | committed | the retry inherited a shorter deadline |
| Post-incident backfill | hours, very high volume | slow lane, throttled | volume is the risk, not latency |
Read the table as a template, not a recommendation. Your numbers belong in the middle column.
Where the free tier fits, and where it does not
This is the point in the workflow where an offer like the one around MonkeyCode becomes testable rather than aspirational. The operator-supplied terms are free model access and a free server option, with 10 million free tokens quoted for the free tier; the project is presented as open source. Treat those as claims to check on the project's official page, including the license, which models are covered, and whether anything expires.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The reason the free server option matters for a cost note is subtler than the token count. A free pool changes what you spend: you trade money for queue position. That trade is excellent for a batch job whose deadline is a maintenance window, and bad for a request a person is watching. Free capacity is a loan against your latency, and something has to repay it.
A one-paragraph version of when free is the wrong bet: whenever the measured slow-lane wait plus retry inflation exceeds the slack you actually have. Not the slack you wish you had. Not the average slack. The slack on the request in front of you.
Who should skip this
If your total spend is small enough that routing logic costs more engineering time than it saves, skip it. If you have no per-request telemetry, skip it until you do, because a router fed guessed percentiles is worse than no router. And if you need hard guarantees about availability, retention, or long-term free tiers, do not build a deadline class on top of something you have not read the terms of. Verify first, then route.
Limitations worth stating plainly: the snippets above are illustrative and were not executed for this article. Percentiles go stale when traffic shape changes, so re-measure on a schedule. A shared free pool can move under you in ways your own logs lag behind. And the strictest deadline class always wins the fast lane, which means the fast lane never gets cheaper no matter how much you route around it.
If you want your own slack numbers instead of mine, the free model access and free server option are a reasonable place to collect real queue-wait data cheaply. Measure first. Migrate second.
