The most interesting moment in my load test was when my proxy was doing nothing.

RPM 40/40 IN-FLIGHT 0 QUEUED 60 TOTAL FIRED 40 

Sixty requests were waiting. No requests were running. Why wasn't request 41 going anywhere?

Because the first 40 had finished, but the minute hadn't. Starting another request then would confuse two different limits: how many requests can run at once, and how many can start in a minute. That distinction is why I built UnlimitedLLM, a small Python proxy that sits between a coding agent and an OpenAI-compatible Chat Completions endpoint.

The semaphore that would pass the wrong test

Imagine putting asyncio.Semaphore(40) around an upstream call. Fire 100 requests at it and the first 40 get through. So far, that looks exactly right. Now imagine the upstream answers in 100 milliseconds. The semaphore releases, the next batch starts, and all 100 requests can hit the provider within a second. The semaphore kept its promise. It just wasn't the promise I needed.

UnlimitedLLM keeps two separate records. A window counter increments when a client request gets permission to start. An in-flight map tracks requests that haven't finished. Completing a stream removes it from the map; it doesn't subtract one from the counter.

The limiter uses a monotonic clock and resets the counter when the configured interval elapses. This is a fixed-window design: it can admit a burst near each side of a window boundary. It doesn't guarantee that every rolling 60-second interval contains at most 40 attempts. That's an important distinction when the provider owns the other clock.

Here's what happened as the 100-request burst drained. These are dispatch counts, not completed answers:

Snapshot Total fired Still queued
First window full 40 60
Next window 80 20
Final batch dispatched 100 0

Request 41 didn't need a faster model. It needed somewhere to wait.

Who gets the next slot?

An agent can have a tool loop running when I type a new question. If both requests have to wait a minute, I care which one goes first. The proxy accepts an X-Priority header: a client can mark an interactive request 1, ordinary work 3, and a large background request 5.

The queue entry in app/rate_limiter.py looks like this:

heapq.heappush(self.queue, (entry.priority, entry.arrival_ts, entry.seq, entry)) 

Python's heap pops the smallest tuple. Priority decides first, arrival time breaks ties, and a sequence number preserves ordering even if two arrivals get the same timestamp. Each waiting request sleeps on its own asyncio.Event. Granting a slot increments the counter and wakes that waiter under the same lock, so two coroutines can't both claim the last slot.

Two details keep the queue useful under load. A request that's waited long enough gets promoted, so a steady stream of interactive work doesn't leave it at the back forever. And the queue has a hard cap: at the default depth of 500, the next arrival gets a local 503 with Retry-After. At that point, telling the client to come back is better than accepting work I can't hold.

The hardest run was Gemini at 15 RPM

I ran the proxy separately against four providers, setting the admission limit for each run. These are my test settings, not advertised quotas. The client sent streaming requests with a small output budget to exercise request handling rather than benchmark model quality.

Upstream RPM setting Requests Client HTTP 200s Client HTTP 429s Avg total time
NVIDIA NIM 40 100 100 0 21,694 ms
Groq 30 100 100 0 35,721 ms
Gemini 15 100 100 0 95,653 ms
OpenRouter 20 40 40 0 3,355 ms

Four providers. 340/340 HTTP 200 responses. Zero client-facing HTTP 429 statuses.

The number I'd look at next is 95.7 seconds. That's the average total time for Gemini, including waiting. The upstream returned both 429 and 503 during that run. From the client's perspective, queuing traded an immediate rejection for a potentially long wait. And since the runs used different limits and request counts, this table isn't a ranking of model speed.

The current proxy retries upstream 429 and 503 responses with bounded exponential backoff: by default, one second, then two, then four. There's a wrinkle: retries happen inside an admitted request. The dashboard can say “40 fired” even when the provider has seen more than 40 HTTP attempts. If I want that number to explain an upstream rate-limit burst, I need two counters: requests admitted and attempts sent.

I gave the load tester a broken answer. It counted a success.

A streaming endpoint sends HTTP headers before the last token exists. If an error arrives later, the proxy can't go back and change 200 OK to 503. It can send an error event inside the stream. My tester read the body, but its success column counted the HTTP status.

To find out what that meant, I ran the same test function against five controlled responses:

What the client received What the tester counted
Content, finish_reason: "stop", then data: [DONE] HTTP 200
Partial content, then an early end with no [DONE] HTTP 200
An error event carrying code 429 HTTP 200
An error event carrying code 503 HTTP 200
An actual HTTP 503 response HTTP 503

The first four landed in the same column. In a separate live check, I asked for PONG and inspected the events: I got PONG, finish_reason: "stop", and [DONE]. That proves one complete reply. The four-provider run didn't record those markers per request, so its 340 successes are HTTP successes.

Try the burst without spending provider quota

Clone the repo and install requirements.txt. In the project root, copy .env.example to .env if you don't already have one. Set these values in .env for the local demo:

UPSTREAM_BASE_URL=http://127.0.0.1:9001/v1 MAX_RPM=40 DASHBOARD_ENABLED=true 

Start python mock_upstream.py in the first terminal. Run python main.py in the second; the proxy loads .env at startup. In a third terminal, run:

python load_test.py --total 100 --wave1 40 --wave2 60 

The second wave starts while the first is still active. Watch IN-FLIGHT and QUEUED separately. Zero in flight with a nonempty queue can be the correct behavior. That's the snapshot I'd put on the cover.

The next benchmark I want has three numbers side by side: requests admitted, attempts sent upstream, and streams completed. If your LLM gateway says “success,” which of those is it counting?