Streaming Feels Faster, but Your Model Isn't: A Wire-Level Timestamp FAQ

Does streaming make a model answer faster?

Every time I hear that, I ask a boring follow-up. Faster for whom, and measured where?

I keep seeing the same four claims repeated in review threads and README files. Each one is testable in about a minute. None of them survives contact with a clock.

Here is the FAQ, the probe I use, and the corrected mental model.

The claim that keeps coming back

A developer adds stream: true. The UI paints tokens sooner. Everyone agrees the endpoint got faster.

But the model did not change. The queue did not change. Only the delivery schedule changed.

So what is actually being measured? And what is being hidden?

Two runs, one clock

You need two numbers per run: time to first byte, and total wall clock.

You also need event timestamps, not network read timestamps. Those differ more than you think.

Here is the probe. Node 20+, no dependencies, one file.

// probe.mjs — two runs, one monotonic clock. import { performance } from 'node:perf_hooks'; const ENDPOINT = process.env.ENDPOINT; // full chat-completions URL const KEY = process.env.API_KEY; const PROMPT = 'List five two-word color names. No other text.'; async function run(stream) { const t0 = performance.now(); const res = await fetch(ENDPOINT, { method: 'POST', headers: { 'content-type': 'application/json', authorization: `Bearer ${KEY}`, accept: stream ? 'text/event-stream' : 'application/json', }, body: JSON.stringify({ stream, messages: [{ role: 'user', content: PROMPT }] }), }); if (!stream) { const text = await res.text(); const t1 = performance.now(); return { stream, status: res.status, ttfb: t1 - t0, total: t1 - t0, events: 1, bytes: text.length }; } const reader = res.body.getReader(); const dec = new TextDecoder(); const marks = []; let buf = ''; let bytes = 0; let firstByteAt = null; for (;;) { const { value, done } = await reader.read(); if (done) break; const now = performance.now(); firstByteAt ??= now; bytes += value.byteLength; buf += dec.decode(value, { stream: true }); let i; while ((i = buf.indexOf('\n\n')) !== -1) { const evt = buf.slice(0, i); buf = buf.slice(i + 2); if (!evt.startsWith(':')) marks.push(now - t0); // skip keep-alive comments } } const t1 = performance.now(); return { stream, status: res.status, ttfb: (firstByteAt ?? t1) - t0, total: t1 - t0, events: marks.length, bytes, marks }; } const batch = await run(false); const live = await run(true); console.table([batch, live].map(({ marks, ...rest }) => rest)); // Burst check: share of SSE events landing in the last 10% of the clock. if (live.marks?.length) { const span = live.total - live.ttfb; const tail = live.marks.filter((m) => m > live.ttfb + span * 0.9).length; console.log(`tail share: ${((tail / live.marks.length) * 100).toFixed(0)}%`); } 

Run it twice against the same prompt and the same endpoint. Keep the terminal output.

I have not run this against your endpoint. Treat it as the instrument, not the result. Run it, and the numbers below will be yours, not mine.

FAQ: four claims, four corrections

1. "Streaming makes generation faster."

It does not touch generation at all. It changes when bytes leave the server.

A streamed run often ends slightly later in wall clock, because every event carries framing and header overhead.

Corrected model: streaming trades total efficiency for earlier partial value. That is a UX decision, not a speed decision.

2. "Time to first byte is a model latency number."

TTFB is a pipeline number. It bundles DNS, TLS, queueing, admission control, prompt processing, tokenizer warm-up, and proxy buffering.

Compare two providers and you are comparing pipelines, not models.

Corrected model: label the metric honestly. Call it client-observed first-byte time, and keep the endpoint and region in the same note.

3. "If chunks are arriving, nothing is buffering."

Buffering can be bursty. A proxy flushes every N bytes, or on a timer.

Look at the tail share. A high value means most events landed in a narrow window near the end.

That is a buffering signature, not a slow model.

If you terminate on nginx, one response header usually settles it:

# Disable proxy buffering for the SSE route. proxy_buffering off; add_header X-Accel-Buffering no; 

Also check content-encoding. Compression on an event stream tends to delay flushes.

4. "Streaming is the safer default on a free tier."

Not always. A stream holds one connection open for the entire generation.

Long-held connections collide with idle timeouts and with whatever concurrency cap you were granted. Batch jobs usually prefer one request and one generous client timeout.

Corrected model: streaming is for humans watching output. Non-streaming is for queues, retries, and anything that runs at 3 a.m.

Cross-check in ten seconds

Before you trust the script, confirm with curl. Two tools disagreeing is information.

curl -sS -N -o /dev/null \ -w 'starttransfer=%{time_starttransfer} total=%{time_total}\n' \ -H "authorization: Bearer $API_KEY" \ -H 'content-type: application/json' \ -d '{"stream":true,"messages":[{"role":"user","content":"List five two-word color names."}]}' \ "$ENDPOINT" 

-N turns off curl's own output buffering, so you can watch events land. The -w timings are recorded either way.

If curl shows a fast first byte and your app shows a slow one, stop blaming the endpoint. Your client is buffering.

Decision table

Use the shape of the workload, not the shape of the demo.

Where this bites on a free tier

Free capacity makes the pipeline longer and less predictable. That is exactly when a bad metric hurts most.

I run these probes against the free model access and free server option that MonkeyCode provides, mostly because the pipeline in front of a free endpoint is the interesting part. The product is the environment here, not the lesson.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free environment changes is variance. Shared capacity means your TTFB depends on neighbours, not just your prompt. So record the time of day with every run.

What this test does not tell you

• It does not measure model quality. You need an eval set for that.
• It does not separate network RTT from server queueing. You need a second vantage point for that.
• One run is noise. Five runs minimum, ideally spread across an hour.
• It says nothing about cost. Free-to-paid transitions are a separate question.
• It cannot prove your proxy config. Only the tail share hints at it.

Who should skip this

If your users never see partial output, skip streaming entirely and skip this probe.

If you already log server-side timings per request, you likely have better data than any client probe.

And if you are choosing between providers on vibes, a timestamp test will not fix the decision process.

The model that survives the clock

Streaming is a delivery choice. It says nothing about model speed.

First-byte time is a pipeline fact, not a model fact. Name it that way.

Bursts near the end mean buffering, not a slow generator. Check the proxy before the prompt.

So: run the probe once. What is your tail share, and does it match what you assumed?

If the two numbers surprise you, that is the article doing its job.