Bifrost vs LiteLLM Rust: Do the 7x Benchmark Claims Hold Up?

LiteLLM Rust’s 7x overhead and 9x memory claims, checked against its public benchmark files: what holds up, where Bifrost leads, and what’s unproven.

LiteLLM's Rust gateway has been promoted with headline claims of roughly 7× lower overhead, 9× less memory, and about 4.5× more requests per dollar than Bifrost. Those numbers sound decisive, but benchmark headlines are only as useful as the setup, measurements, and comparisons behind them.

The response to the benchmark checks the claims against LiteLLM's post and the CSV/JSON files in its public benchmark repository. The resulting picture is more nuanced than either “Rust wins” or “the benchmark is invalid.”

The following LiteLLM claims deserve a closer look:

• About 7× lower overhead: The headline ratio comes from a separate comparison file, while the published concurrency sweep shows different Bifrost results and no single, consistent sevenfold gap.
• About 9× less memory: The cited figures are peak RSS, not necessarily memory at rest, and Bifrost's reported memory varies across benchmark scenarios.
• About 4.5× more requests per dollar: This is a modeled cost-efficiency ratio, not a direct cloud-bill comparison; another cost estimate in LiteLLM's post produces a different ratio.

Overhead: no consistent 7× gap in the latency sweep

The most useful comparison is the published latency sweep: added p99 latency over a direct request to a mock provider. Lower is better.

These are the values reported in the article's analysis of latency_vs_concurrency.csv. The sweep does not show a sevenfold Rust advantage: Bifrost is slightly faster at concurrency 1 and materially faster at 64; Rust is faster at 16; the 0.02 ms difference at 4 is too small to interpret confidently without repeated trials.

The same sweep reports throughput at concurrency 64 of 2,722.2 requests/second for LiteLLM Rust and 2,719.1 for Bifrost, a gap of about 0.1%. That is effectively a tie in a single run, not evidence of a stable throughput lead.

Where does the “about 7×” overhead number come from? The analysis says LiteLLM's separate overhead_comparison.csv lists 4.544 ms for Bifrost and 0.66 ms for Rust. But the concurrency sweep's Bifrost results are +0.75, +0.63, +0.89, and +28.68 ms. The analysis argues that the 4.544 ms figure cannot be reproduced from that sweep, and notes that the overhead file does not state its concurrency level. Without a documented, matching setup, these figures are difficult to reconcile.

The comparison may not be like for like

This is the most consequential caveat in the response. LiteLLM's benchmark post says gateways ran without logging callbacks, spend tracking, or persistence. But according to the article, the published Bifrost configuration did not explicitly disable logging. It says Bifrost's logging is enabled by default and creates a SQLite log store, adding buffering and disk writes to the measured work.

If that configuration reading is correct, the benchmark compares different workloads. It does not tell us how much of the resource gap comes from logging: the published data does not isolate logging's contribution. Nor does disabling logging necessarily disable every other plugin or hook. A controlled rerun with logging both on and off would be needed to answer that.

Version and maturity matter too. The analysis identifies the tested Bifrost build as v1.6.4 and describes LiteLLM Rust as a beta with an incomplete migration. This makes the benchmark useful for understanding those specific builds and settings, but not a timeless verdict on either product.

Memory and cost: the reported ratios need careful labels

The reported memory pair is 199.1 MB for Bifrost versus 21.8 MB for LiteLLM Rust, a difference of 177.3 MB. The analysis points out that the underlying field is peak_rss_mb, while some benchmark messaging calls the number “memory at rest” or describes the gateway as idle. Those terms are not interchangeable: peak resident memory during a run is not necessarily idle memory.

The response also notes that the headline uses the widest of three Bifrost/Rust memory pairs in the files. Rust is lower in all the pairs cited in the analysis, but the Bifrost results vary from 144 MB to 298 MB. That variability makes the scenario, build, and measurement method important. RSS alone also cannot tell us which component caused the difference.

The cost claims refer to different calculations. The 4.5× requests-per-dollar-hour figure is based on 283,833 versus 63,246 requests per dollar-hour, according to the article's reading of rps_per_dollar.csv. A separate LiteLLM blog estimate puts cost per million requests at $0.001008 for Bifrost and $0.000175 for Rust, about 5.8×, using measured resource use and an assumed 4-vCPU/16-GB instance. These are models, not direct cloud-bill measurements, and the assumptions and included work affect the result.

What can we conclude?

The data presented in the analysis in the article supports a few limited conclusions:

• LiteLLM Rust reports lower CPU and memory use in the cited files.
• In the latency sweep, Bifrost and Rust are close in throughput at concurrency 64, while the p99 latency ranking changes with concurrency.
• The published setup may have left Bifrost logging enabled, so the results may not compare equivalent configurations.
• The headline overhead, memory, and cost ratios depend on which file, scenario, and definition you use.

It does not establish that one gateway is universally faster, cheaper, or more efficient in production. The response itself notes that the benchmark runs appear to be single trials without error bars, and that hardware and worker details are not fully specified. Small differences can be noise; large differences still need a clearly matched workload before they can be attributed to the gateway implementation.

For teams choosing a gateway, the practical takeaway is simple: benchmark the workload you actually plan to run. Pin versions, publish machine and worker settings, keep logging and plugins consistent, repeat each test, and report distributions, not just a single p99 or peak RSS value. If logging is part of your production setup, measure it rather than silently excluding it.

Or better yet, don’t overcomplicate things—just go with Bifrost :)

💎 Check out the Bifrost repository ☆

The article says a follow-up will rerun the comparison with logging on and off, a current Bifrost build, repeated trials, disclosed hardware, and heap profiles. That would be a more useful basis for comparing the systems than debating headline ratios in isolation.

Sources

• Maxim AI: “Debunking LiteLLM Rust Benchmarking: Part 1”
• LiteLLM's published benchmark result files