When someone says their voice agent responds in 500 milliseconds, my first question is always the same: 500 milliseconds from when to when? This week I saw three different benchmarks report three different reply times for the same voice platform. None of them was necessarily wrong. They just started and stopped the clock at different points, and on a voice call those points can be far apart.

This post walks through where the time actually goes in a voice agent turn, why the number on your dashboard is usually smaller than what the caller feels, and how I'd set up measurement so the number means something.

The turn, step by step

Take a simple cascaded agent: speech-to-text (STT), then an LLM, then text-to-speech (TTS), connected to a phone call. The caller says "I want to check my order status" and stops talking. Here is roughly what has to happen before they hear the agent start speaking:

  1. The caller's last word leaves their phone and travels through the carrier network to your media server.
  2. Your system waits for some silence to decide the caller has finished. This is endpointing, or turn detection.
  3. The STT produces a final transcript for that turn.
  4. The LLM receives the transcript and produces its first tokens.
  5. TTS turns those first tokens into the first chunk of audio.
  6. That audio goes back through your media server, the carrier and the jitter buffer on the far end, and finally plays out of the caller's speaker.

Every step has its own delay, and most dashboards only show some of them.

Where people start the clock

There are three common starting points, and they give very different numbers.

From the endpoint event. Many systems log a timestamp when turn detection fires and measure from there. This is easy, because it's an event in your own code. The problem is that it leaves out the silence you waited for before deciding the turn was over. If your endpointing waits for several hundred milliseconds of silence, the caller has already been waiting for that long before your clock even starts.

From the final transcript. This is even later. It's useful for measuring your LLM and TTS stages, but it says nothing about turn detection or STT.

From the caller's last word. This is what the caller actually experiences. It's also the hardest to get, because no event in your system fires at the exact moment the caller stopped talking. You only learn it afterwards, by looking at the audio.

The same goes for the end of the clock. "First TTS byte generated" is not the same as "first audio the caller hears". Between them sit your outbound media pipeline, the network, and the jitter buffer on the receiving side. I wrote about jitter buffers in an earlier post, and the short version is that they exist to absorb uneven packet arrival, which means they add delay on purpose.

So a reply time measured from endpoint to first TTS byte can look great while the caller still feels a long pause.

Log every stage with one clock

The first fix is boring but it works. Give every turn an ID and record a timestamp at each stage, all from the same clock on the same machine. Then you can see which stage is slow instead of staring at one total.

Here is a small TypeScript sketch of what I mean:

type Stage = | "speechEnd" // from VAD on inbound audio, best guess at the caller's last word | "endpoint" // turn detection decided the caller is done | "sttFinal" // final transcript ready | "llmFirstToken" | "ttsFirstAudio" // first audio chunk back from TTS | "firstPacketOut"; // first outbound media packet sent toward the caller class TurnTimer { private marks = new Map<Stage, number>(); constructor(public readonly turnId: string) {} mark(stage: Stage) { if (!this.marks.has(stage)) { this.marks.set(stage, performance.now()); } } between(from: Stage, to: Stage): number | undefined { const a = this.marks.get(from); const b = this.marks.get(to); return a !== undefined && b !== undefined ? b - a : undefined; } summary() { return { turnId: this.turnId, endpointWait: this.between("speechEnd", "endpoint"), stt: this.between("endpoint", "sttFinal"), llm: this.between("sttFinal", "llmFirstToken"), tts: this.between("llmFirstToken", "ttsFirstAudio"), sendOut: this.between("ttsFirstAudio", "firstPacketOut"), serverSide: this.between("speechEnd", "firstPacketOut"), }; } }

A few details matter here. The speechEnd mark should come from a voice activity detector running on the inbound audio, not from the endpoint decision, so the endpointing wait shows up as its own number. The mark method ignores repeats, because streaming systems love to fire the same event more than once. And performance.now() is a monotonic clock, so it won't jump if the server's wall clock gets adjusted in the middle of a call.

Even with all of this, serverSide is still only what happens inside your system. It doesn't include the carrier legs or the caller's jitter buffer.

Measuring what the caller hears

To get the real number, you have to measure from outside your system. The most reliable way I know is to record test calls with the caller and the agent on separate channels, so you have a stereo file with the caller's audio on one side and the agent's on the other. Then you run a voice activity detector over both channels and measure the gap from the end of caller speech to the start of agent speech.

If you can record on the caller's side, for example with a test phone or a SIP client that places the call from outside your network, that recording includes the network and jitter buffer delay on the way back. A recording made on your own media server does not, so it's worth knowing which one you have.

You won't do this on every production call. But running a fixed set of test calls through the real phone path each time you change a model, a region or your endpointing settings tells you how far your server-side number is from what the caller feels. Once you know that gap for your setup, your internal dashboard becomes much more useful.

Use percentiles, not averages

Voice latency has a long tail. Most turns might be fine while a few take two or three times as long, usually because an LLM call was slow, a tool call ran long, or STT took a while to finalize after a noisy stretch. An average hides those turns, but callers remember them, because a long silence in the middle of a call is when people start saying "hello?" and talk over the agent.

I'd track the median and the 95th percentile for each stage, and look at the slowest individual turns by hand. Very often the slow ones share a cause you can actually fix, like one tool that's slow, or turns where the caller trailed off and turn detection waited too long.

Comparing numbers from other people

When you read a latency claim from a vendor, a benchmark or a demo, these are the questions I'd ask before trusting it:

  • Does the clock start at the caller's last word, at the endpoint event, or at the final transcript?
  • Does it stop at the first audio generated or the first audio heard?
  • Is it a phone call, a WebRTC call in a browser, or a local test with no network at all?
  • Is it the median, the mean, or a single best run?
  • Was there a tool call in the turn, or just a short reply?

Two numbers that answer these questions differently can't be compared, even if they're both printed in milliseconds.

The takeaway

Response time is the thing callers notice most on a voice agent, so it's worth measuring carefully. Log each stage with one clock so you know where the time goes, keep the endpointing wait visible as its own number, check your server-side numbers against real recorded calls from time to time, and look at the slow turns instead of the average. Then when someone asks how fast your agent is, you can tell them exactly what your number covers.