My second run looked 1.8x faster. Nothing got faster. My cache got warm.

That is the whole finding. Two identical prompts, back to back, and the only variable was order. If you benchmark a model endpoint by calling it twice, you are measuring your own cache, not the model.

So what does a benchmark actually measure when the first call pays for everything? Mostly plumbing.

I built the small harness below for one reason. I needed a number I could defend when someone asks why the demo feels slower than the chart. The answer, every time, was that the chart was warm.

Three warmups hide inside one call

Every first call warms at least three things. The TCP and TLS handshake. The connection pool. And, on providers that cache prompt prefixes, the prefix itself.

Most writeups lump all three together and call it cold start. Then they report one averaged number and move on. That average is fiction, because the three warmups have different costs and different fixes.

So I split them. Same prompt, same evening, three conditions.

Cold socket with a cold prompt. New process, new connection, cache-busting nonce.

Warm socket with a cold prompt. Connection reused, nonce still breaks the prefix.

Warm socket with a warm prompt. The steady state your dashboard actually shows.

The nonce is one random token at the front of the system message. It costs you nothing if your provider does not cache prefixes. It saves you from fooling yourself if it does.

The harness 

# bench.py — three conditions, paired, never averaged across conditions import json, os, time, uuid, urllib.request BASE = os.environ["BASE_URL"].rstrip("/") KEY = os.environ["API_KEY"] MODEL = os.environ["MODEL"] PAIRS = int(os.environ.get("PAIRS", "6")) SYSTEM = "You are a terse assistant. " BODY = ("Summarize the incident below in one sentence.\n\n" + "log line. " * 300) def ask(nonce: str) -> float: payload = { "model": MODEL, "messages": [ {"role": "system", "content": nonce + SYSTEM}, {"role": "user", "content": BODY}, ], "max_tokens": 32, } req = urllib.request.Request( BASE + "/chat/completions", data=json.dumps(payload).encode(), headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}, ) t0 = time.perf_counter() with urllib.request.urlopen(req, timeout=180) as r: r.read() return time.perf_counter() - t0 rows = [] for i in range(PAIRS): rows.append({"pair": i, "cond": "cold_socket", "s": ask(uuid.uuid4().hex + " ")}) rows.append({"pair": i, "cond": "warm_socket_cold_prompt", "s": ask(uuid.uuid4().hex + " ")}) rows.append({"pair": i, "cond": "warm_socket_warm_prompt", "s": ask("")}) print(json.dumps(rows, indent=2))

Paste that into bench.py, export four variables, and run it.

export BASE_URL="https://<your-endpoint>/v1" export API_KEY="<your key>" export MODEL="<model id>" PAIRS=6 python bench.py > runs.json

Notice what I did not do. I did not ping the endpoint before the loop. A warmup ping would hide the socket cost, and the socket cost is one of the things I came to see.

I also did not run six colds and then six warms. Conditions alternate inside every pair, on purpose. On a free or shared endpoint, queue depth drifts over the evening. Interleaving spreads that drift across all three conditions instead of dumping it on whichever one ran last.

That detail matters more than the code. Does your vendor's queue stay flat for ten minutes? Mine does not.

The graph I kept

I kept a paired line chart. Two lines, cold and warm, one point per pair index. A bar chart of means would have flattened the exact thing I was hunting.

The digits below are illustrative, not a vendor claim. Rerun the harness and the digits are yours, which is the point.

Pair zero is the tallest. Cold socket lands well above everything else. That is the handshake plus the pool, a one-time cost per process, and it never comes back in the same run.

From pair one onward the cold-prompt line sits flat and high-ish. The warm-prompt line drops and stays down. Does that gap shrink with more pairs? It does not. It is a cache, not a warmup curve.

So the honest first-user number is the middle line. The steady-state number is the bottom line. The top line is an artifact of my process starting, and I should not quote it as API latency.

Paired deltas beat averages

Averages across conditions tell you nothing here. Compare each pair to itself.

import json, statistics as st rows = json.load(open("runs.json")) by = {} for r in rows: by.setdefault(r["pair"], {})[r["cond"]] = r["s"] deltas = [p["warm_socket_cold_prompt"] - p["warm_socket_warm_prompt"] for p in by.values()] print("median cache delta (s):", round(st.median(deltas), 3)) print("min:", round(min(deltas), 3), "max:", round(max(deltas), 3))

Report the median delta, and report the range next to it. A delta that flips sign between pairs is not a cache effect. It is a queue, and you should fix your sample before you fix your code.

The mistake I almost shipped

I nearly put the bottom line in the README as "latency". It reads beautifully. Nobody questions a fast number.

Then a new user opens the app once, pays the middle line, and gets a worse experience than the chart promised. That is not a lie you want to defend in a support thread.

So now I publish two numbers with their conditions attached. First call after connect, and steady state. If I cannot label a number with its condition, I do not publish it.

Where the free endpoint fits

I ran these pairs against MonkeyCode's free model access on its free server option, both operator-supplied availability. I needed an endpoint I could hit eighteen times in one evening without a card on file, and I did not want my benchmark budget to become a variable.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not quoting quotas or hardware here. Those numbers change faster than blog posts get edited, and a stale figure is worse than no figure. Check the current limits on the project page, not in an article, including this one.

What I will say is this. A free endpoint is a good place to learn what your own cache is costing you, because your wallet is not part of the measurement.

Limitations, stated plainly

One machine, one region, one evening. No concurrency, so this tells you nothing about throughput.

Wall clock includes queueing, and you cannot subtract queue time you cannot see. The alternating loop blunts that, it does not remove it.

The nonce makes every prompt unique, so token counts and provider-side deduplication behave differently than in production. That is a trade I accepted to keep the conditions separable.

Tiny prompts are dominated by HTTP overhead here. If your prompts are under a few hundred tokens, you are measuring sockets, not models.

Who should skip this

If you serve a handful of requests per minute, cache warmth is noise. Ship the feature.

If you want a vendor comparison, this harness is too narrow. Order effects will swamp a real difference between providers, and you will pick a winner based on which call went first.

And if you cannot spare six paired runs, do not conclude anything. Two calls is a rumor.

Warm your socket. Keep your prompt cold. Never publish the second run.

If you want to try the harness before you pay anyone anything, the free model access and free server option are a reasonable place to start.