Language models aren't deterministic, so every leg of a voice agent conversation has to run several times before you trust it. One good call tells you the agent can work. It tells you nothing about whether it will.
That single idea changes how I'd approach a phone agent as a developer. Stop treating it as a prompt you tweak until the demo sounds right, and treat it like any other service you ship, with small integration points and versions you can test and roll back.
The spread is real, even at temperature 0
Thinking Machines Lab sent 1,000 identical requests to a model and got 80 distinct completions back. A voice stack piles more variance on top, because the speech-to-text model can mishear a name and callers interrupt whenever they like. The EVA-Bench study tested 12 voice agent systems, and none scored above 0.5 on both task accuracy and caller experience.
So "I called it and it worked" is a sample size of one.
Keep your own code at the edges
The hard real-time work, like streaming audio and deciding when the caller has finished talking, belongs to the orchestration layer. Your code only shows up at the edges of a call.
Before the call, your system starts it. In Dograh, the open-source AI voice agent platform we build, that's a POST to an API Trigger with the number to dial and an initial_context object the prompts can read. You can self-host it or run it on our cloud.
curl -X POST https://your-dograh-instance/api/v1/public/agent/{uuid} \ -H "Content-Type: application/json" \ -H "X-API-Key: dgr_your_key" \ -d '{ "phone_number": "+1415555XXXX", "initial_context": { "customer_name": "Jane", "order_id": "A-1042" } }' The reply carries a workflow_run_id. Store it against your own record, so a retry on your side never dials Jane twice.
While the call runs, the agent calls you. An HTTP API tool hits your endpoint to check a slot or write a record. Those tools time out after 5 seconds by default and don't retry, so the prompt has to say what the agent tells the caller when a request fails. A tool can also play a typed line or a recorded clip as it starts, which keeps a slow lookup from turning into dead air.
When the call ends, a Webhook node posts the results to you, with the extracted variables and QA results, plus links to the recording and transcript. Timeouts and 5xx replies get retried automatically, up to 5 attempts. A wrong URL or credential doesn't, so that's the failure to watch for in your logs.
That's the whole integration surface. Everything after it is testing and release.
Write the scenarios before you trust anything
Pick a fixed set of scenarios, including the awkward ones, like the caller who gives a wrong date of birth or asks for a person halfway through. Run each leg several times and count the passes. A pass rate per leg is far more honest than a feeling about one recording.
Test Chat in the editor lets you rerun a single agent turn, or edit it and rerun, without dialling again. Test values go in the Variables tab, and setting direction to outbound exercises the outbound path. QA scores each step of the call separately, with its own 1-10 score and tags, so you can compare versions node by node instead of guessing which change helped.
Finish with a real phone call on the line you'll actually use, since a browser test runs at 16 kHz while most phone calls still arrive at 8 kHz.
Ship versions, never live edits
This is the step teams skip. Every edit should land in a draft, and live callers should only ever reach a published version. In our case that's built in: edits save as drafts, Publish runs validation and records a new entry in Version History, and the API Trigger has a production URL for the published version plus a test URL that runs the latest draft.
Rollback gets boring, which is exactly what you want. Pause the campaign, publish the previous settings again, and you're back. A campaign can also point at one specific saved version, so a half-finished edit never reaches a caller.
One catch if a coding assistant helps you build. An agent created through the MCP server is published as version 1 straight away, so review it before it takes real calls. Edits to an existing agent save as drafts like everything else.
If I had to boil it down, keep your integration code at the edges and let pass rates, not demos, decide when a version ships.
How many runs per leg do you make before you call a voice agent ready? I'm curious where other teams land.
The security threats and the go-live checklist that didn't fit here are in our developer's map of a phone agent, from first API call to release.
