How OpenAI Made Voice AI Feel Like a Real Conversation: 8 Engineering Changes

When a voice agent feels slow, most people blame the model. In practice, the model is only one of several places where time is lost.

A voice call has a long path. The call has to connect. Audio has to travel. The system has to decide when you are done talking. Something has to think. Something has to speak. And all of this has to keep running for a huge number of calls at once.

OpenAI has written about changes at almost every step of that path. This post walks through them in the order a call actually happens:

connecting a call → starting it faster → scaling it → making the talk natural → handling hard tasks → saving CPU time → switching models → securing direct connections

Each section starts with a real problem, then shows the fix. Most sections have a small interactive diagram on the original post.

First: how a voice agent call works, in plain words

A voice agent is a program that talks to you on a call. It can live on a website, in an app, or behind a normal phone number. Every time you speak, the same loop runs:

Hear. Your voice is cut into tiny pieces of audio and sent over the network to a server.
Think. A model works out what to say. It may also look something up or use a tool.
Speak. The answer is turned into voice and sent back over the network.

The word you will see most in this post is latency. It just means delay: the time between you stopping and the agent starting to answer.

Why is this so strict for voice? In studies of conversations in many languages, the gap between one person finishing and the next one starting is usually around 200 milliseconds. That is a fifth of a second. An agent that takes one second to reply already feels slow. Several seconds of silence feels like a dropped call.

And the delay does not come from one place. It is lost in the connection setup, in the network trips, in waiting to be sure you finished, in the model thinking and in making the voice. That is why OpenAI worked on so many parts at once. Here is the map for the rest of the post:

Map: where each change sits on a call

The map has four boxes, in the order a call happens:

Connect the call (before anyone can talk): 2. WARP + Instant Connect, and 1. Phone: SIP + webhook
Edge and media (carry the audio packets): 3. Relay + Transceiver, and 6. Python to Go
The model (listen, think, speak): 4. Full-duplex, 5. Background reasoning, and 7. Model handoffs
Your app (how it connects and stays safe): 8. Ephemeral tokens

The numbers match the sections below. Each section zooms into one box.

Interactive version on my site

If a word is new to you, there is a small glossary at the end.

1. How WebRTC connects you to an AI voice agent

Before we talk about what changed, we need to know how a voice call reaches an AI agent in the first place.

Most browser voice agents use WebRTC. It is the same technology behind video calls in your browser. It is built for one job: move audio and video between two endpoints with very low delay. Instead of sending audio over a normal request and response channel, WebRTC sends it as small UDP packets, about 20 milliseconds of sound each.

In a voice agent, one endpoint is the user. The other endpoint is not a person. It is a server that runs a speech model.

How a call reaches an AI voice agent

There are two paths:

From a browser or app: User (browser or app) → WebRTC → AI voice agent (speech model on a server). The four steps are signaling, find a path, lock it, then talk. During signaling, the browser asks your server for a session. The two sides swap a short description of what they can do (called SDP). No audio yet.
From a real phone call: Caller (any phone) → phone network (the carrier) → SIP trunk (for example Twilio) → AI voice agent. The four steps are dial, invite, accept, then talk.

Names like ICE, SDP and SIP are only labels here. The WebRTC post on this blog goes deeper.

Interactive version on my site

On the browser side, a call goes through four steps before you hear anything:

Signaling. The two sides swap a short description of what they can do.
Find a path. They test network paths and pick one that works through routers and firewalls.
Lock it. They agree on encryption keys.
Talk. Audio flows both ways at the same time.

The names for these steps (SDP, ICE, DTLS) do not matter for this post. What matters is this: steps 1 to 3 happen before the agent can hear a single word. Keep that in mind for the next section.

What about real phone calls?

Many voice agents are not in a browser at all. A customer dials a normal phone number, and an AI answers. The journey is different:

The caller dials. The phone network routes the call to a SIP trunk provider (for example Twilio). This company turns phone calls into internet calls.
The trunk sends a SIP INVITE to the voice API. SIP is the "someone is calling" language of phone systems. OpenAI's Realtime API accepts SIP calls at a project address, and it sends your server a webhook called realtime.call.incoming.
Your server accepts the call through the API and sets the voice, the instructions and the tools. It can also open a WebSocket for that call to watch events and control it.
Audio flows as RTP packets, the same small packets as in the browser case. Phone lines are narrow, usually 8 kHz, so a phone call sounds thinner than a browser call.

The phone path is also in the interactive version on my site (the From a real phone call tab).

Why does this matter for this post? Because different changes live in different places:

Full-duplex, background reasoning and model handoffs live inside the model and the session. They help every call, however it arrived. They matter most on the phone, where callers say "uh-huh", talk over the agent and have no screen to look at.
WARP is about the WebRTC handshake, so it matters for browser and app clients. A phone call uses SIP for its first leg instead.
I could not find OpenAI saying exactly how phone calls use the relay and transceiver layer. Treat that part as unknown.

If you want the full story with NAT, STUN and TURN, I wrote about it in the WebRTC post. The packet side is in the voice jitter buffer post.

2. WARP: why starting a voice conversation took time

The problem. When you press the mic button, nothing can be heard until the setup is done. Each step of the setup needs messages to go to the server and come back. Every one of those trips is called a round trip. OpenAI counts six round trips to start media and the data channel in a normal WebRTC call. If your network round trip is 100 ms, that is 600 ms before the agent can listen.

Why so many? WebRTC was designed before QUIC, a newer protocol that was built to keep setup short. WebRTC stacks several protocols on top of each other. Each one does its own checks. Some of that work is repeated. For example, both the path check and the encryption handshake have their own way to protect the server from fake clients.

The fix. OpenAI made two related changes:

WARP changes the transport handshake itself. Media and data channel startup goes from six round trips to one. WARP is backward compatible. One write-up says it folds the encryption handshake into the path check and uses a newer version of DTLS.
Instant Connect takes signaling off the critical path. The settings are agreed before the call. The server can create the session when the first media packet arrives, so the client can start with a single UDP packet. Normal signaling stays as a fallback.

OpenAI is advancing WARP as open specifications through the IETF (the group that standardizes internet protocols), and support has already landed in libwebrtc and Pion, two widely used WebRTC libraries.

Starting a voice call: before and after WARP

Before: vanilla WebRTC. Six round trips: ICE checks the network path, then DTLS hello, cookie check, key exchange and finish, then the data channel opens and the agent listens.
After: WARP. One round trip for all of it.

OpenAI counts six round trips for media and data channel startup. How they split up in this picture is illustrative. Each double arrow is one round trip (a message out and a reply back). Signaling is separate, and Instant Connect takes it off the critical path.

If the network round trip is 100 ms, the wait before the agent listens is 600 ms in the before case.

Interactive version on my site

On the interactive version, you can play both versions and move the round trip slider. The number of round trips drops from six to one. What each round trip costs depends on your network, so the gain is biggest on slow mobile networks, which is where many real users are.

Why it matters: the first second of a voice call decides how it feels. If the agent is not listening yet, the user's first words are lost or the user waits in silence.

3. Relay + Transceiver: scaling millions of voice sessions

WARP makes one call start faster. This section is about a different problem: running a very large number of calls at once. They are related, but they solve different things.

The problem. Picture a call center building. In the classic setup, every call has its own front door facing the street. Every worker is on the public internet, with its own open UDP port, and the same worker also holds everything about the call. With more calls you need more workers, more open ports and more machines facing the internet.

That hurts in practice. Large public port ranges are hard to run safely, especially in Kubernetes. Some workers get busy while others sit idle. Rolling out a new version is risky, because a worker with live calls cannot just be replaced.

The fix. Add a reception desk and keep the real work inside the building. OpenAI split the job into two layers:

Relay. The reception desk. It sits on the public side with a small, fixed footprint. It receives packets and forwards them to the right room. It keeps almost no state.
Transceiver. The room where the call actually happens. It owns the whole WebRTC session: path checks, encryption handshake, encrypted media and session lifetime. It lives inside the cluster, not on the public internet.

How does the relay know the right room? The design reportedly puts a routing hint in the first packets of a session. The relay reads the hint, forwards, and does not need to understand the whole protocol.

OpenAI also looked at the usual alternatives and passed on them. A TURN server is a heavier middleman than needed, because most voice agent calls are one user and one model. An SFU is built for conference calls with many people. Here the model is just one endpoint, not a conference guest.

Where do the calls land? Workers on the internet vs relay + transceivers

Before: every worker faces the internet. Each caller talks straight to a worker. That worker needs its own open UDP port and also holds the session. Public UDP ports grow with the number of calls, and every worker holds WebRTC session state.
After: a relay sits on the public side with a small, fixed footprint and forwards packets to a transceiver inside the cluster. The transceiver owns the whole WebRTC session. A routing hint in the first packets of a session is how OpenAI describes the relay finding the right transceiver without keeping session state.

Illustrative.

Interactive version on my site

On the interactive version, you can click different callers and switch between the two tabs. Before, each caller needs its own open port, so the number grows with the calls. After, the public side stays small and fixed, and all the hard WebRTC state lives in one place, inside the cluster.

OpenAI describes the idea in one sentence: the best place to add complexity is in a thin routing layer.

Do not mix these up. WARP improves connection setup for one call. Relay + Transceiver improves infrastructure scaling for many calls.

4. Full-duplex: AI that listens while speaking

The problem. Think of the difference between a walkie-talkie and a phone call. On a walkie-talkie you press a button, talk, say "over", and wait. On a phone call you can interrupt, laugh, say "uh-huh" and the other person adjusts.

The classic voice agent is a walkie-talkie. It is a pipeline:

Two ways to build a voice agent

Old way: walkie-talkie. Speech to text (voice becomes words) → LLM (words in, words out) → Text to speech (words become voice). One step at a time. Each step waits for the one before it.
New way: phone call. You ⇄ one speech-to-speech model that listens and speaks at the same time.

Interactive version on my site

A small helper model, called a turn detector, listens for silence to guess when you have finished. Only then does the LLM start. And while the AI is speaking, it is slow to notice that you started talking.

That has a built-in trade-off. If the detector guesses too early, the agent cuts you off. If it guesses too late, you wait in silence. A quiet "mm-hm" looks like a new turn, so the agent stops and restarts. On a real phone call, with noise and people who talk over each other, this is where the robotic feeling comes from.

The fix. OpenAI's GPT-Live (the model behind ChatGPT Voice) is a native speech-to-speech model. Audio streams in and speech streams out at the same time, with no hard line between "your turn" and "my turn". The turn detector is removed from the audio path. This is called full-duplex: both sides can talk and listen at once, like a phone call between people.

A phone call with a pizza bot

The diagram compares a walkie-talkie agent and a phone call (full-duplex) agent in three moments: you ask a question, you interrupt, and you say "mm-hm". Timings are illustrative.

Interactive version on my site

Imagine you are calling a pizza shop run by an AI. On the interactive version, you can press play and try the three moments. Switch between the walkie-talkie and the phone call, and watch the AI ears and when the AI answers:

You ask a question. The old way leaves a pause. The new way answers right when you stop.
You interrupt. The old way talks over you for a moment and needs a new turn. The new way stops and answers.
You say "mm-hm". The old way stops and restarts. The new way understands it means "go on".

5. Background reasoning: thinking without stopping the conversation

The problem. A fast voice model is good at talking. It is not good at long reasoning or slow tool calls. If one model has to do everything, the user hears silence while it thinks. A real person would say "let me check" and keep talking.

The fix. Split the work across two models:

A fast voice model stays in the call. It talks and listens all the time.
A reasoning model (a frontier model, GPT-5.5 at the time of writing) works on a separate asynchronous path. The voice model hands it the hard task. The result is merged back into the live conversation without pausing it.

OpenAI counts the whole delegation loop against the response time budget: routing, prompt processing, model run and tool calls. To keep it fast, the app server creates the reasoning session when the call starts and fills it with the conversation so far, so the first delegation does not pay for a cold start. Session affinity (sending the same session to the same machine) and prompt caching keep later calls quick.

A fast voice model with a reasoning model in the background

The voice model (fast, always in the call) stays on the line. The reasoning model (slow, deep, uses tools) works on a separate path. In the example, the user says "Move my flight to tomorrow and email me the receipt." The voice model never stops talking and listening. The slow work happens on a separate path.

Interactive version on my site

On the interactive version, you can step through the example. Notice that the voice model keeps talking and even asks a follow-up question while the work runs.

6. Python to Go: improving real-time audio performance

This one is short on purpose.

The problem. Real-time audio is a stream of tiny frames that must arrive on schedule. The media frontend and the inference logic were running on Python asyncio. At large scale, small delays and unlucky scheduling add up. They show up as the slow frames, the tail latency, which users hear as cuts and glitches.

The fix. OpenAI moved the media frontend and inference logic to Go. A usual reason for this kind of move is that Python asyncio runs many streams on one event loop, so one slow task can delay the others, while Go spreads streams across CPU cores. That is my explanation, not a quote. The reported result is that the new system's p95 frame delivery time matches the old system's p50. In plain words: the slow frames of the new system are about as fast as the typical frames of the old one.

Frame delivery time, before and after the Go rewrite

The chart compares p50 and p95 frame delivery time for Python asyncio and for Go. Time grows to the right. p50 is the typical frame, p95 is the slow one in twenty. Illustrative scale. OpenAI reports a relative result only: the new p95 matches the old p50.

Interactive version on my site

OpenAI shared this as a relative result, so the chart is only a picture of the idea, not real milliseconds.

7. Seamless model handoffs: switching without interrupting the user

The problem. Long calls fill up the model's context window. Sometimes you also want to change the model that serves a call. If you just stop one model and start another, the user hears a gap, and the new model has to read the whole history first.

The fix. Treat it like a hot swap:

Warm up a new instance.
Prefill it with the current conversation.
Run both in parallel while the new one catches up.
Cut over when the new one is ready.

When the history gets too long, it is compacted (shortened), and a replacement instance is prepared in the background. The live call never waits for the new instance to rebuild its cache.

Switching models in the middle of a call

The timeline shows instance A serving the call, then a new instance B warming up and prefilling, then both running in parallel, then the cut over so A is retired and B is serving. The user hears continuous audio, with no gap. Illustrative timeline.

Interactive version on my site

There is one more finding worth keeping. For voice, the first limit is usually not the GPU. Open sessions send audio frames all the time, so the CPU-side stream handlers, queues and network paths fill up first. OpenAI also tested the new system with shadow traffic: a small share of real ChatGPT Voice sessions ran on it in read-only mode, without changing what users heard. That found problems short load tests missed, like memory pressure in long sessions, state recovery after reconnects and races during shutdown.

8. Grok: why ephemeral tokens matter

The last topic is not about OpenAI's servers. It is about how your app should connect to a voice API.

The problem. A browser app needs to talk to a voice model. The easy and unsafe way is to put the API key in the browser. Anyone can open dev tools, copy it and spend your money. The safe but heavy way is a backend relay: the browser sends audio to your server, and your server forwards it to the provider. That keeps the key safe, but every audio packet now passes through your backend. It adds a hop of delay and your backend must scale with every call.

The fix. Use an ephemeral token: a short-lived key for one session.

The browser asks your backend for a token.
Your backend calls the provider with the permanent key and gets a short-lived token.
The backend gives that token to the browser.
The browser connects directly to the provider with it.

Backend relay vs ephemeral token

Backend relays audio: Every audio packet passes through your backend. It is one more hop of delay, and your backend must scale with every call.
Ephemeral token: The backend only hands out a token. Audio goes straight from the browser to the voice provider. The permanent API key never leaves your server.

Grok Voice Agent is shown as the example provider. OpenAI Realtime supports the same pattern.

Interactive version on my site

xAI's Grok Voice Agent API works this way. Your server calls POST /v1/realtime/client_secrets with the API key to create the token. Browsers cannot set custom headers on a WebSocket, so the browser passes the token in the WebSocket subprotocol list with an xai-client-secret prefix when it connects to wss://api.x.ai/v1/realtime. The token expires after the time you set. xAI's docs say to always use ephemeral tokens for browser and mobile apps.

OpenAI supports the same pattern. Its Realtime API has an endpoint with the same name, POST /v1/realtime/client_secrets. Your server creates a short-lived key (it starts with ek_), and the browser uses it to open its WebRTC session. So the idea is not tied to one vendor. The benefit is the same: your key stays on your server, your backend is out of the audio path, and audio takes the shortest route.

The whole story in one picture

Step	Problem	Fix
Connect	Six round trips before the agent can listen	WARP (six to one) and Instant Connect (start with one packet)
Scale	Public ports and worker state grow with calls	Relay + Transceiver
Natural talk	Walkie-talkie turns, pauses, bad interruptions	Full-duplex speech-to-speech (a phone call, not a walkie-talkie)
Hard tasks	Thinking blocks the conversation	Fast voice model + background reasoning
Performance	Tail latency in audio frames	Python asyncio to Go
Long calls	Context limits, gaps when switching models	Parallel instances and compaction
Security	Keys in the browser or audio via your backend	Ephemeral tokens, direct connection

If you build voice agents, the lesson is that latency is not one number. It is the sum of connection setup, packet routing, turn-taking, reasoning, scheduling and model changes. A better model helps. A better system around the model helps just as much.

What can you use today? GPT-Live is described as rolling out in ChatGPT Voice first, with API access planned. Check OpenAI's current docs for what your API key can use. Parts of this story, like ephemeral tokens and SIP calling, are already available in the Realtime API.

If you build voice agents: my takeaways

These are my own conclusions, not OpenAI's.

Measure the whole gap. Time from "user stops talking" to "first audio is heard". Not only the model time.
Look at the slow end, not the average. A call is ruined by its slowest moments, so watch p95 as well as the typical case.
Test the human moments. Interrupt your agent. Say "mm-hm" while it talks. Stay silent for two seconds. These show the problems that a demo hides.
Do not leave silence while work runs. Let the agent say a short line while tools or a slower model work.
Never put a permanent API key in a browser. Use short-lived tokens, or run the call through your server on purpose.
Check phone and browser separately. They connect in different ways and sound different (narrow phone audio vs wide browser audio).

Words you will see

Word	Plain meaning
Latency	Delay. Here: the time before the agent answers.
Round trip	A message goes to the server and the reply comes back.
Packet	A tiny piece of data. Voice is sent as many packets, about 20 ms of sound each.
UDP	A fast way to send packets that does not wait for receipts. Good for live audio.
WebRTC	The technology browsers use for live voice and video.
SIP	The "someone is calling" language of internet phone systems.
SIP trunk	A company that connects normal phone numbers to internet calls.
RTP	The envelope format for live audio packets.
STT / TTS	Speech-to-text and text-to-speech.
LLM	The language model that decides what to say.
Turn detector	A helper that guesses when you have finished speaking.
Full-duplex	Both sides can talk and listen at the same time.
p50 / p95	The typical case, and the slow case (1 in 20 is slower than this).
Ephemeral token	A short-lived key for one session.

Sources

This post is based on the following. My tools could not open OpenAI's engineering pages directly, so some details come from technical write-ups and quotes of them, including the six-to-one round trip number. Check the originals for exact wording.