A production outbound voice agent was taking about three seconds to answer. We got it to about one. Almost none of that came from changing the model, the speech-to-text provider, or the text-to-speech voice.

The symptoms were the ordinary ones. Callers talked over the bot, because by the time it started speaking they had assumed the line was dead. Others just went quiet and hung up. Of the calls that were answered, only about 40% of callers actually engaged with the agent at all.

When a voice agent feels slow, the instinct is to go shopping. A faster LLM. A streaming TTS. A lower-latency ASR. We did some of that, and it moved the number very little.

The two seconds were somewhere else entirely, and the place they were hiding is not on anyone's list.

Stop measuring the turn end to end

A single end-to-end number is useless for debugging. "Three seconds" tells you nothing about which part of the turn owns them.

A voice turn is at least five stages, and each one can be timestamped separately:

  • End-of-speech detection — the caller stops talking, and your VAD decides they have finished. This is a guess, and the threshold you pick is a latency knob.
  • ASR finalization — the last partial transcript becomes a final one.
  • LLM first token — not the full completion. If you are waiting for the whole response before speaking, that is your bug, not your latency.
  • TTS first byte — again, first audio out, not the full synthesis.
  • Transport — getting that audio onto the call and into the caller's ear.
  • Log a timestamp at every boundary, on every turn, with the call ID. Then look at the distribution rather than the average, because the calls that lose you customers are in the tail.

What we expected to find was one fat stage. A slow model, or a TTS provider having a bad week.

That is not what the data showed. Every stage was a little slower than it should have been. Nothing was catastrophically wrong anywhere, and the whole thing was two seconds too slow. The delay was smeared evenly across stages that share nothing with each other except the machine they run on.

That pattern is the tell. When unrelated stages all degrade together, the problem is underneath all of them.

The machine was out of CPU

The hosts running the pipeline were sitting at 96% CPU.

Not during a load test. In normal operation, carrying ordinary call volume, on shared-core burstable instances that had been chosen the way most infrastructure gets chosen — early, cheaply, and never revisited.

We moved to dedicated-core VMs. On n2-standard-8, CPU settled at around 23%. End-to-end turn latency went from roughly three seconds to roughly one.

No model changed. No provider changed. No prompt changed.

Why saturation hits voice harder than it hits anything else
If you run a web API at 96% CPU, you get worse p99 latency and a few timeouts. Unpleasant, survivable.

Audio is different, for three reasons.

It is a continuous deadline, not a request. Audio arrives in frames, typically every 10 or 20 ms, and every frame needs to be handled before the next one lands. There is no queue you can fall behind on politely. A frame handled late is jitter, and jitter is audible.

Every stage is competing for the same starved cores. VAD, resampling, codec work, the websocket reads, the TTS playback loop. They all need the CPU at the same moment, which is exactly why the delay appeared everywhere at once instead of in one place.

Shared-core instances steal time invisibly. Burstable instance types are built to borrow CPU in short spikes and give it back. A voice call is not a spike — it is sustained load for the length of the call. Once the burst credits are gone you are throttled, and the throttling shows up in your application as mysterious jitter with no corresponding error in any log.

That last one is the trap. Nothing fails. No exception is raised, no provider returns an error, no dashboard goes red. The pipeline just gets uniformly, inexplicably sluggish, which is precisely the shape that sends you off looking for a faster model.

Fast is not the same as not-awkward

At one second, the agent was fast and callers were still hanging up.

Latency and turn-taking feel like the same problem from the outside. They are not. Latency is how long the silence is. Turn-taking is whether the silence means anything. A caller will forgive a slow agent that behaves predictably, and will hang up on a fast one that talks over them.

Three failures mattered more than the remaining milliseconds.

The agent cut people off. The VAD was declaring end-of-speech too eagerly, so a caller drawing breath mid-sentence got interrupted. We settled on a 0.3s silence threshold before treating speech as finished. That number is a tradeoff, not a best practice — shorter feels snappier and interrupts people, longer feels sluggish and lets them finish. It is worth tuning against your own call recordings rather than copying it from a blog post, including this one.

The agent went silent and nobody recovered. A dropped partial, a provider hiccup, a state machine in a corner it had no exit from — and the line just sat there. Callers do not wait through silence on a phone call; they assume the call is broken, because it is. We added an 8s silence-recovery prompt: if the agent has said nothing for eight seconds, it says something. Anything is better than dead air, and this single change did more for completed calls than several hundred milliseconds of latency did.

The agent lost its instructions when interrupted. In multi-tier bots, a user interruption mid-response was dropping the core system instructions from the context that got rebuilt afterwards. The agent carried on sounding fluent and stopped following its own rules, which is the worst failure mode available to you because it is invisible in your metrics and obvious to the customer. If you support barge-in, write a test that interrupts the agent and then asserts the system prompt survived.

There was also a duplicate-sentence bug, where the agent would say the same line twice. Less interesting technically, equally fatal to trust.

The combined result: callers engaging with the agent went from about 40% of answered calls to about 60%. Roughly twenty points more people actually having the conversation — which was always the point, and is a better number to be judged on than a latency figure.

Check these before you change providers

In rough order of how often they are the actual problem, and how cheap they are to rule out:

  • Host CPU during real calls. Not at idle, not during a synthetic load test — during live traffic at your busiest hour. If it is above 70%, stop reading and fix that first.
  • Your instance type. If it is shared-core or burstable, you are gambling on credits you cannot see. Realtime audio wants dedicated cores.
  • Whether you are streaming at all. Waiting for a full LLM completion before starting TTS, or full synthesis before playing audio, costs more than any provider choice. First token and first byte are the numbers that matter.
  • Your VAD threshold, against your own recordings. Listen to twenty calls. You will hear whether you are cutting people off.
  • What happens on silence. If your agent can go quiet with no recovery path, you are losing calls to a bug you have never seen in a log.
  • Interruption handling. Barge-in, then assert your system prompt is still there.
  • Only then go shopping for a faster model.

What this does not cover

Being honest about the edges of it.

This was a Python pipeline on GCP carrying telephony traffic, not a WebRTC browser product, and transport behaves differently there. The 3s-to-1s figure is end-to-end turn latency on our own calls under our own conditions — it is not a benchmark, and your baseline will differ.

We also never got the tail as tight as the median. A small number of calls stayed slow for reasons I never fully pinned down, likely a mix of carrier-side variance and provider cold starts. If you have solved that cleanly, I would genuinely like to hear how.

The broader point survives all of those caveats. In realtime voice, the model is the part everyone looks at and rarely the part that is costing you the seconds. Look underneath it first.

I build realtime voice agents in Python — streaming ASR → LLM → TTS. If you are working on this and hit something strange, I am happy to compare notes.