I Told Everyone to Tune Voice Agents Per Room. Nine of Mine Share One Config.

Adapted from a post on my own site. I reworked it for a dev audience and added the part where I check my own work against it.

A few months ago I wrote a long post about why voice agents talk over people, or refuse to stop talking. The single most useful habit in it, I said, was this: tune turn taking per agent, never globally. A garage and an estate agent can share a prompt and every integration and still need different interruption settings, because they get called from different rooms.

This week I diffed the exported configs of nine agents I built for US small businesses. HVAC, roofing, a dental practice, two law firms, a water treatment company, credit repair, debt settlement and a sales qualifier. Here are their turn taking settings:

{ "interruption_sensitivity": 0.9, "responsiveness": 1, "enable_backchannel": false, "ambient_sound": "call-center", "denoising_mode": "noise-and-background-speech-cancellation", "reminder_trigger_ms": 10000, "reminder_max_count": 1, "end_call_after_silence_ms": 60000, "allow_user_dtmf": true, "stt_mode": "accurate" } 

That is not one of them. That is all nine. Byte for byte. The only fields in that neighbourhood that differ anywhere are voice speed (1.1 on the sales agent) and max call duration.

So this post is the original advice, rewritten for engineers, with my own configs as the worked example of what not following it looks like.

Two decisions, made in opposite directions

Most "it sounds bad on real calls" complaints are not language problems. They are turn taking problems, and there are two separate decisions hiding in there.

Endpointing: has the caller finished? The agent watches for silence and decides the caller's turn is over. Too eager and it answers while someone is drawing breath in the middle of their postcode. Too patient and every turn ends in a dead pause that reads as a dropped call. In Retell this lives mostly in responsiveness, and I wrote about what it costs on the clock in A Faster Model Will Not Fix Your Slow Voice Agent.

Barge-in: has the caller started? The agent is speaking, audio arrives on the inbound leg, and the agent has to decide whether to stop. In Retell that is interruption_sensitivity, from 0 to 1, where lower means the caller has to say more before the agent yields.

Barge-in does not hear words, which is why noise breaks it far more than it breaks recognition. It has to decide before any transcription exists, on a few hundred milliseconds of waveform. A dog, an impact wrench, a tannoy and a passenger in the next seat are all audio energy on the inbound leg.

What the failure looks like

A noisy call degrades in a recognisable loop:

1. The agent starts a sentence. Something in the caller's room makes a noise. The agent stops mid word.
2. The agent resumes or rephrases, producing another fragment.
3. The caller hears fragments, starts talking to fix it, and triggers another stop.

Three turns in, two parties are interrupting each other and the transcript looks like the agent had a stroke. Nothing in the prompt is wrong. The agent never finished a sentence long enough to be judged on.

The mirror failure is an agent so deaf to barge-in that it finishes a twenty second turn while the caller says "hello, hello" into the void. That one is worse commercially, because the caller hangs up and tells the business the robot ignored them.

And here is the trap for anyone debugging this: a transcript of a badly interrupted call and a good one can look almost identical as text. Everyone opens transcripts because that is what the dashboard shows. This bug lives in the audio.

So what is wrong with my nine identical configs?

Read the list of businesses again with the room in mind:

• HVAC and roofing. Plausibly a lot of these callers are homeowners standing next to a broken outdoor unit, or someone on a job site, or a person on a ladder looking at a leak.
• Two law firms. Injury callers may well be calling from a car, a hospital, or a room with other people in it.
• Dental. Plenty of calls probably come from a quiet kitchen.

Those are not the same acoustic environment, and they all have the same 0.9 barge-in sensitivity, which is close to the most interruptible end of the scale. If I followed my own advice, the trades agents are the first ones I would test from a noisy place, and the first ones I would expect to need that number lowered.

I should be honest about what I do not know here. I have not gone back through recordings to prove any of these agents is actually misbehaving on noisy calls. The point is narrower: identical numbers across nine different rooms is evidence nobody tuned them, me included. It was a copied block that worked well enough in testing from a desk. A shared default is fine. A default nobody revisited is the problem.

What tuning actually looked like when I did it

I do have one agent where I clearly did iterate, a UK estate agent build from my time at Fortell AI, and the version history is in the exports:

Responsiveness climbed, the silence timeout halved, and backchannel ("mhm", "right", "lovely") switched on. Barge-in sensitivity sat at 0.7 the whole time, noticeably less interruptible than the 0.9 the US agents ended up with. That is what per agent tuning looks like in practice: a few knobs moving over several versions while the one you were happy with stays put.

A small platform gotcha from the same history: v1 used "ambient_sound": "office", which Retell rejected on import with a 400. call-center is accepted. If you are porting configs between platforms, expect enum fields like that to bite.

What I would change, in order

1. Treat turn taking settings as per agent config, not a template. If you generate agents from a base config (I do), make the turn taking block an explicit, per client input rather than something inherited silently. A default that nobody chose is not a decision.

2. Shorten the agent's turns before touching any slider. This one my prompts already do. All nine say some version of "ask one question per turn, keep responses to one to three sentences." A one sentence turn is barely worth interrupting, and if it gets cut off the caller already has the part that mattered.

3. Confirm identifiers, not conversation. Noise rarely destroys a whole call. It destroys names, numbers, addresses and dates. My agents spell emails back character by character and confirm addresses before booking, which is the right instinct.

4. Give unclear audio a ladder. This is the gap I found. Every one of the nine has allow_user_dtmf: true, but none of the prompts ever offers the keypad, and none has an explicit "I did not catch that" path. What I want is: first unclear turn, ask them to repeat. Second, ask a narrower question or offer the keypad. Third, stop and offer a callback or a human. Seven of the nine prompts also have a rule to end the call immediately if "repeating loops" are detected, aimed at jailbreak attempts. I now want to check a noisy fragment loop cannot trip that same rule, because ending the call on a caller standing next to a compressor is the worst possible outcome.

5. Cover known waits with speech. If the agent goes quiet during a calendar lookup, the caller assumes the line dropped, starts talking, triggers barge-in, and collides with the response arriving. One short line before a tool call removes a whole class of collision.

Test it in the room, then freeze it

You cannot evaluate turn taking by reading transcripts. So call the HVAC agent standing next to an outdoor unit. Call the law firm agent from a moving car on speakerphone. Then fold those calls into the scripted set you re-run before every change, because a prompt edit that lengthens the greeting by one sentence can undo a week of tuning. I went into why phone calls are so hard to reproduce in There Is No Repro for a Phone Call.

The other half of this habit is cheap and I recommend it to anyone running more than a handful of agents: export every agent config and diff them. When I did that for prompts I found nine agents sharing one giant prompt. Doing it for the audio settings found a block of numbers nobody, including the person who wrote the blog post telling you to tune them, had ever tuned.

The full original, with the four UK rooms I first ran into this in (a garage, a vet waiting room, a car on the motorway and a hospital corridor), is on my site.

If you run voice agents in production: do your agents share one turn taking config, and has a noisy room ever told you it should not?