I built a small gateway that pools free LLM tiers — NVIDIA NIM, Kilo, OpenRouter, a couple of
community endpoints — behind a single OpenAI-compatible URL. Thirty-three models, one API key,
no monthly bill. It ran real daily work for a month.
It also broke in six different ways, and every single break was something I would have avoided
if someone had written it down first. So I did.
The numbers first
| What | Measured |
|---|---|
| Free tier rate limit | 40 requests/min, sliding window |
| Burst probe result | 45 requests → 24 success, 20 rate-limited, 1 upstream 503, in 12.8s |
| Fast model, 95K context | 18.6s |
| Slow model, same context | 83s |
| Requests that hung and then failed | every one, at 178–194s |
That last row is the one that cost me the most sleep.
Break 1: the 180-second hang that was not my bug
Every request through one particular relay hung for exactly 178 to 194 seconds and then failed
with a connection error. My code was fine. The relay had a hard 180-second timeout of its own,
and it sat on the socket without sending a single byte until the deadline expired.
The fix is not a longer timeout. It is an inter-chunk idle deadline: if no SSE chunk arrives
within N seconds, abort and retry on a fresh connection. A total-duration timeout will not save
you, because a stalled connection still counts as in-flight.
Break 2: one 429 froze every model on the provider
Free tiers rate-limit at the channel level, not the model level. A single greedy request tripped
a 429 and the cooldown I wrote disabled every model behind that provider for two minutes —
including ones I was not even calling.
Two fixes: pace client-side at ~85% of the measured entitlement instead of bursting, and scope
any cooldown to the exact model that was limited.
Break 3: the retry stampede
Fifteen parallel workers hit the same failure and retried on the same second. One upstream
hiccup became an outage. Exponential backoff without full jitter is the bug — the workers
stay synchronized.
Break 4: a byte-order mark broke my config
A scheduled job started throwing Unexpected token ''. The JSON config was written by
PowerShell's Set-Content -Encoding UTF8, which emits a BOM. JSON.parse does not tolerate it.
The fix is one line, but finding it was not.
Break 5: the stream that connected and said nothing
Some proxies accept the connection and send zero bytes. No error, no data — the request just
sits there. This looks identical to a slow model until you instrument the difference between
connect-fail, idle-fail, and slow-success.
Break 6: I lost a batch at item 700
No checkpoints. A crash at item 700 of 1000 meant reprocessing everything, and the bill doubled.
An append-only checkpoint journal with content-hash dedupe keys makes any job resumable in
seconds.
What I actually built
The gateway is about 600 lines: a candidate selector that respects per-model cooldowns, a pacer
that keeps under the sliding window, a failover ladder (primary → fallback → local Ollama), and
one JSONL line per request so that a 3am question has an answer in the morning.
Three of the recipes — the pacer, the 15-second messaging window, and the postmortem template —
are MIT licensed in this repository: llm-ops-recipes.
The other 27 (JSON repair loops, provider failover ladders, model tiering, golden-set regression
gates, cost accounting, and two full postmortems) are in the
LLM Ops Cookbook.
The one thing I would tell myself a month ago
Measure the limit before you design around it. I spent two days building a cooldown system
before running a 45-request burst probe that told me, in 13 seconds, exactly what the real
constraint was.
If you'd rather not rebuild this from scratch, I packaged the exact scripts I use: LLM Ops Cookbook — 30 one-page recipes for running LLM workloads that don't fall over. There is also a free sample if you want to see the format first.
