If your app talks to models through an OpenAI-compatible gateway (LiteLLM or one of its many peers), that gateway is doing a lot on every request. It re-streams the response chunk by chunk, recomputes token usage, works out what the request cost, and has to pass upstream failures back to you honestly.
That is stateful, timing-sensitive code sitting in the hot path of every call. The bugs it produces share one nasty property: they look like success.
The five failure modes
1. A dropped or reordered stream chunk. The answer arrives one word short. Nobody notices in a manual test, because you'd have to know what the full answer should have been.
2. Wrong token usage. prompt_tokens, completion_tokens or total_tokens is off, either in the final streamed chunk or the non-streamed body. Your usage dashboards are now fiction.
3. Wrong cost. The gateway still returns a number. You only find out it was wrong when the bill doesn't match the usage.
4. Cross-stream leakage. Under concurrency, content from one request shows up in another's stream. This is a real bug class in gateways that multiplex streams over shared internal state, and it is also a data-exposure problem.
5. A swallowed upstream error. The upstream fails mid-stream and the gateway reports finish_reason: "stop". To your application, that is indistinguishable from a complete, successful answer. This is the worst one.
Why you can't test this against a real model
The obvious approach is to point a test at the live gateway and check the output. The problem is ground truth: you don't know in advance what a real model will say, how many tokens it will use, or exactly when it will stream each chunk. So you can't tell whether the gateway changed anything on the way through.
The fix is to replace the model with something scripted.
The pattern: a deterministic mock upstream
- Start a mock upstream that speaks
/v1/chat/completionslike a real provider, but every response is scripted: exact content, exact usage, exact timing, and, when you want it, an exact failure. - Configure the gateway under test to use that mock as its upstream, the same way you'd normally point it at a provider.
- Send requests through the gateway and compare what comes back with what the mock was told to send.
checks ──requests──> [ gateway under test ] ──requests──> mock upstream (ground truth) <──responses── <──responses── Each failure mode becomes a check with a precise pass condition:
| Check | Passes when |
|---|---|
| streaming fidelity | the reassembled stream equals the non-streamed response to the same request |
| usage accuracy | reported token counts equal the mock's scripted usage |
| cost accuracy | the reported cost (e.g. LiteLLM's x-litellm-response-cost header) equals usage × your pricing table |
| concurrent isolation | parallel streams never contain each other's content |
| error propagation | a scripted mid-stream failure surfaces as an error, never as finish_reason: "stop" |
Keep the checks independent so a failure tells you exactly which promise broke.
Doing it with an existing tool
I packaged this pattern as an MIT-licensed TypeScript tool called gatewayproof. Configure your gateway's upstream as http://127.0.0.1:5055, start the gateway, then:
npx gatewayproof run --target http://localhost:4000 --mock-port 5055 A run against a gateway with a cost bug looks like this:
PASS streaming-fidelity -- Streamed content matches non-streamed content, with no drops, dupes, or reordering PASS usage-accuracy -- Reported token usage matches the upstream's actual usage FAIL cost-accuracy -- Reported cost matches usage x the supplied pricing table expected cost 0.00019999999999999998, got 0.000001 PASS concurrent-isolation -- Concurrent streamed requests never leak content across streams PASS error-propagation -- A mid-stream upstream failure surfaces as an error, never a silently-truncated success 4/5 checks passed. It exits 1 on any failure, so it works as a CI gate. There is also a GitHub Action:
- name: Start your gateway run: | my-gateway serve --port 4000 & # upstream = http://127.0.0.1:5055 sleep 2 - name: Run gatewayproof against it uses: jayblast-spec/gatewayproof@main with: target: http://localhost:4000 mock-port: '5055' If you are building a gateway yourself, use it as a library inside your own test suite:
import { startMockUpstream, runChecks } from "gatewayproof"; const mock = await startMockUpstream(5055); const gateway = await startMyGateway({ upstreamUrl: mock.url }); const report = await runChecks({ targetBaseUrl: gateway.url, mock }); console.log(report.passed, report.results); await gateway.stop(); await mock.stop(); What this does not cover
It is not a load test or a benchmark, and it is not a fuzzer: every check uses a fixed script. It also doesn't yet cover provider-specific features such as function calling or vision inputs. It covers the chat-completions streaming, usage and cost surface every OpenAI-compatible gateway shares, which is where the most expensive silent bugs tend to live.
Takeaway
Whatever tool you use, the principle carries over: don't test a gateway against a real model. Script the upstream, push requests through the gateway, and assert on the difference. That turns "it seemed fine when I tried it" into a pass/fail you can put in CI.
I wrote gatewayproof, so I'm biased; issues and PRs are welcome on GitHub. This article was drafted with AI assistance and checked against the tool's source and README by me.
