Price the Retry, Not the Request

The sticker price of a request is the least interesting number in your cost model. What you actually pay is the request, plus the ones that failed before it worked, plus the minutes you sat queued behind other people's failures, plus whatever the deadline cost you when the budget ran out.

A free lane changes the first term of that sum to zero. It does nothing to the other three. That is the whole trap: free capacity is priced at zero and billed in retries, queue wait, and missed deadlines, and none of those show up on the page that told you it was free.

The retry multiplier is the real unit price

If an attempt succeeds independently with probability p, the expected number of attempts to your first accepted output is 1/p. At p = 0.6 you are paying 1.67 requests per result. At p = 0.3 you are paying 3.33. Nothing about that is new, and almost nobody puts it in the budget.

What makes it an ops problem rather than an arithmetic curiosity is that retries consume wall clock, and wall clock is the resource you cannot top up. A deadline converts retry cost into a hard feasibility question.

expected_attempts = 1 / p worst_case_spend = unit_cost * attempts_fitting_in_deadline attempts_fitting = floor((deadline - queue_wait) / attempt_latency) p_accept(attempts) = 1 - (1 - p) ** attempts effective_cost = worst_case_spend / p_accept(attempts) 

Note the split. expected_attempts tells you the average bill. attempts_fitting tells you whether you can finish at all. Those two numbers disagree often, and the disagreement is where outages live.

A gate you can run before you trust the lane

The script below is stdlib only, has no API keys in it, and runs offline. It takes your measured success rate and latency, computes the deadline-feasible attempt count, prints the effective cost per accepted output, and exits non-zero when the lane cannot meet your success threshold. The simulate function is the point: it checks the closed form against Bernoulli draws so you can tell a modeling error from a capacity problem.

#!/usr/bin/env python3 # budget.py - retry-and-deadline gate. stdlib only, no network calls. import argparse import math import random import sys def attempts_fitting(deadline_s: float, latency_s: float, queue_s: float) -> int: usable = deadline_s - queue_s if usable <= 0 or latency_s <= 0: return 0 return int(usable // latency_s) def p_accept(p: float, n: int) -> float: if n <= 0: return 0.0 return 1.0 - (1.0 - p) ** n def effective_cost(unit_cost: float, p: float, n: int, queue_s: float, engineer_rate_per_hour: float = 0.0) -> float: accepted = p_accept(p, n) if accepted == 0.0: return math.inf spend = unit_cost * n wait = (queue_s / 3600.0) * engineer_rate_per_hour return (spend + wait) / accepted def simulate(p: float, n: int, trials: int = 20000, seed: int = 7) -> float: rng = random.Random(seed) hits = 0 for _ in range(trials): for _ in range(n): if rng.random() < p: hits += 1 break return hits / trials def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--p", type=float, required=True, help="measured per-attempt success rate") ap.add_argument("--latency", type=float, required=True, help="seconds per attempt") ap.add_argument("--queue", type=float, default=0.0, help="observed queue wait, seconds") ap.add_argument("--deadline", type=float, required=True, help="total budget, seconds") ap.add_argument("--unit-cost", type=float, default=0.0) ap.add_argument("--engineer-rate", type=float, default=0.0, help="currency per hour") ap.add_argument("--min-success", type=float, default=0.95) args = ap.parse_args() n = attempts_fitting(args.deadline, args.latency, args.queue) closed = p_accept(args.p, n) empirical = simulate(args.p, n) if n > 0 else 0.0 cost = effective_cost(args.unit_cost, args.p, n, args.queue, args.engineer_rate) print(f"attempts fitting deadline : {n}") print(f"expected attempts : {1 / args.p:.2f}" if args.p > 0 else "expected attempts : inf") print(f"p_accept closed form : {closed:.4f}") print(f"p_accept simulated : {empirical:.4f}") print(f"effective cost / accepted : {cost:.6f}") if abs(closed - empirical) > 0.02: print("WARN: model and simulation disagree; re-measure p and latency", file=sys.stderr) if closed < args.min_success: print(f"FAIL: {closed:.4f} < {args.min_success:.4f}", file=sys.stderr) return 2 return 0 if __name__ == "__main__": raise SystemExit(main()) 

Run it against a lane you think is fine:

python3 budget.py --p 0.6 --latency 18 --queue 4 --deadline 90 \ --unit-cost 0.002 --min-success 0.95 

That input gives you four feasible attempts (86 usable seconds, 18 seconds each), a p_accept near 0.974, and an effective cost of roughly eight thousandths per accepted output — about four times the sticker, because you are budgeting the worst case rather than the average. Drop --queue to 0 and the attempt count becomes five; raise it to 40 and you get two attempts and a failed gate. Queue wait is the input that flips verdicts, and it is the input teams guess at.

In CI, keep the exit code:

python3 budget.py --p 0.6 --latency 18 --queue 4 --deadline 90 --min-success 0.95 || exit 1 

When the cheap lane is the wrong bet

The pattern in the right column is not about price. Free capacity is a spot market: its availability moves with everyone else's load, and queue wait is the signal that moves first. Under Little's law, wait time grows as arrival rate approaches service rate, so a lane that is snappy at 9am can be unusable at 2pm without a single thing changing in your code. Measure the queue, not the marketing page.

Where a free lane earns its place

A free lane is at its best as an experiment lane: the place where a failed attempt costs you nothing but clock, and where clock is not on the critical path. That is the honest use for MonkeyCode's free model access and free server option — running the simulator, replaying a batch that nobody is waiting on, validating a schema change against real prompts before you commit capacity to it. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free tier as operator-supplied availability, not as an SLA. Free quotas, limits, and hardware change without notice, so read the current terms before you build a dependency on them, and never let a free endpoint own a deadline you promised to someone else. The method above is deliberately product-agnostic: swap in your own measured p and latency and it grades any lane, paid or free.

Limits and who should skip this

This gate models independent retries. If your failures are correlated — a provider incident, a bad prompt, a poisoned input — p is not constant and the arithmetic flatters you. It also ignores token-level cost variation, so lanes that bill by token rather than by request need unit_cost recomputed per attempt, not per call.

Skip this if you have under a hundred calls a day and no deadline: the gate costs more attention than it saves. Skip it too if you cannot measure p honestly, because a guessed success rate produces a confident wrong answer.

If you want to test the gate before you trust it in production, run it against the cheap lane first, with a deadline you do not care about. That is the order that keeps the pager quiet.