A caller rings the Harborview Inn and asks what they need to know about checking in. The front desk starts the full rundown: check-in from 3 pm, photo ID and the card they booked with, a 100 dollar deposit, breakfast hours, the pool and the wifi. Four seconds in, the caller cuts across it with "Sorry, wait, is there parking?", and the client's log prints a single line: (barge-in: stopped the reply after 4.8 s).

That line is accurate, but it answers a different question. It records how far into the reply the client was when it decided to stop, which the server needs to trim its history. It does not record how long the caller heard the agent talking over them, and it cannot say whether the agent should have stopped at all, because the same handler runs for a question, a murmured "mm-hmm" or a television in the room whenever the server reports the sound as the caller.

In this guide, you will take the Harborview Inn agent in pmckeetx/barge-in-voice-agent and record scripted calls to PolyAI's Dialog-RSN-1 API as stereo WAV files, with the caller on the left channel and the agent on the right, both timed by when each sound actually played. You will then score them with bargein.py, a Python script that uses only the standard library and reads any 16-bit two-channel WAV, whichever stack produced it. For every moment the caller starts talking while the agent is speaking, it reports how long the agent kept talking and, given a label for what the caller meant, whether that was a stop, a missed barge-in, an ignored backchannel (a listener's "mm-hmm") or a false stop.

On 17 November 2026 I recorded 70 scripted calls against api.studio.poly.ai from the US Midwest. When the caller asked about parking, the agent's voice stopped a median 0.78 s after the caller started talking (0.76 to 0.86 s over 20 calls). If you hold your agent to a 300 ms budget, a figure some developers start from although no standard sets it, that misses by about half a second. Nearly all of that time passes before speech_started arrives, because while a reply is audible Dialog-RSN-1 confirms that a sound is the caller before it sends the event; the client then cut playback within one 20 ms tick. The same check let a murmured "mm-hmm" pass in 10 of 10 calls, but it mistook a newsreader on a nearby TV for the caller in 10 of 10, and the newsreader cut the reply every time.

Code: pmckeetx/barge-in-voice-agent (recorder.py, record.py and bargein.py)

What you'll build

You will add three files to a Python voice agent for the Harborview Inn. recorder.py writes each scripted call as a stereo WAV on the playback clock, with a JSON file beside it that says what the caller said, when, and what they meant. record.py runs a caller script repeatedly and prints the stop each recording shows next to the arrival of speech_started, so you can check the recording against the server's events. bargein.py scores any 16-bit two-channel WAV, and offline tests prove it on synthetic calls with known answers.

Here is the finished result, bargein.py scoring all 70 recordings from the run described below:

A terminal running uv run bargein.py runs/*.wav: per-call lines such as "parking (barge-in) stopped after 0.80 s [stopped]" and "tv (background) stopped after 1.86 s [false stop]", then a table with one row per script: parking stopped 20 of 20 with a median of 0.78 s, the phone-line run 10 of 10 at 0.90 s, Stop and Wait 10 of 10 at 0.78 and 0.77 s, mm-hmm 0 of 10, and the TV 10 false stops

How it works

Stop time is the interval between two edges on two channels: the moment the caller starts talking on the left, and the start of the silence that ends the agent's speech on the right. Three things add up to it. The first is the server's confirmation, which ends when speech_started arrives. The second is the client dropping the audio it has queued but not yet played, which in this repository happens within one 20 ms tick. The third, on real hardware, is the sound card's output latency, which no recording made inside the client can see.

The first part depends on what the server believes the caller can hear. Our client sends no playback reports, so, per the overview, the server estimates how long each reply stays audible from a typical speaking rate. Every number in this guide was measured that way.

Two recorded calls drawn as 20 ms frame levels on one time axis. Top: the caller starts "Sorry, wait, is there parking?" at 10.84 s while the agent is mid-rundown; speech_started arrives and playback is cut 0.78 s later, after which only the caller's channel has sound, and a dashed segment marks the sound card's output latency the recording cannot see. Bottom: a 0.51 s "mm-hmm" over the same rundown raises no speech_started and the agent talks on

The recorder's design follows from that definition. The channels stay separate because on a mixed recording the agent's last words and the caller's first words are one signal. The recording is written on the playback clock because the server streams the reply in 200 ms chunks, so the client holds 0.08 to 0.5 s more audio than it has played (measured on api.studio on 10 November 2026, with this repository's client), and a recording taken when audio arrives would place the agent's last word up to half a second away from where the caller heard it. The caller's intent comes from labels written by the caller script, because a length cutoff cannot do that job: our "mm-hmm" clip lasts 0.51 s, which is longer than our "Wait." clip.

Prerequisites

To follow along, you will need Python 3.13 and uv, which installs the exact dependency versions from the lockfile. You will also need a Dialog-RSN-1 API key: sign in at studio.poly.ai and create one in the API Keys tab.

You do not need a microphone. Every call in this guide is a scripted caller playing WAV files into a simulated room, which keeps echo out of the numbers and makes each run repeatable. To score your own stack in Step 6, you will need a two-channel recording of one of its calls.

Set up the project

Start by cloning the repository and installing the dependencies:

git clone https://github.com/pmckeetx/barge-in-voice-agent.git cd barge-in-voice-agent cp .env.example .env uv sync 

The repository already holds the agent (agent.py), its audio (audio.py) and the scripted caller (caller.py); this guide adds recorder.py, record.py and bargein.py, and changes caller.py to feed the recorder and to give each caller line an intent. Open .env and add your key:

# Your Dialog-RSN-1 API key: sign in at https://studio.poly.ai and create one in the API Keys tab. POLY_API_KEY= # Optional: the full realtime endpoint. Defaults to wss://api.studio.poly.ai/v1/realtime. # DIALOG_REALTIME_URL= 

POLY_API_KEY is the key you just created. DIALOG_REALTIME_URL stays commented out unless your key belongs to a different PolyAI host. Before you spend any API time, run the offline tests, which need no key and no network:

uv run pytest tests/test_bargein.py tests/test_recorder.py 
tests/test_bargein.py .............. [ 77%] tests/test_recorder.py .... [100%] ============================= 18 passed in 15.86s ============================== 

test_recorder.py takes about 15 seconds because it runs two real-time calls through the simulated room against a fake realtime server.

Step 1: Record both sides of the call on one clock

The scripted caller in caller.py runs a loop that ticks every 20 ms. On each tick it pulls the next 20 ms of the agent's audio from the speaker, which is the audio the caller hears at that moment, and builds the next 20 ms of what the agent's microphone picks up. Both halves of the recording already exist inside that loop; the recorder only has to write them side by side.

Here is the recorder, excerpted from recorder.py:

 def add(self, caller: bytes, agent: bytes): """One tick: what the agent heard from the caller, and what the caller heard from the agent.""" pair = np.column_stack([np.frombuffer(caller, np.int16), np.frombuffer(agent, np.int16)]) if self.step > 1: pair = pair.reshape(-1, self.step, 2).mean(axis=1).astype(np.int16) self.frames.extend(pair.tobytes()) def now(self) -> float: """Seconds of call recorded so far: the recording's clock.""" return len(self.frames) / 4 / self.rate def label(self, line: str, intent: str | None): self.lines.append({"line": line, "onset_s": round(self.now(), 3), "intent": intent}) def mark(self, event: str): self.events.append({"type": event, "t_s": round(self.now(), 3)}) 

add() interleaves one tick from each side into stereo frames with np.column_stack, so a sample on the left and the sample beside it on the right happened at the same instant. now() turns the bytes written into seconds (four bytes per stereo frame), which makes the recording its own clock, and label() and mark() note caller lines and server events on it. save(), not shown, writes the WAV and the JSON.

Watch out: Record in the playback loop
Do not tap the agent's channel in the response.output_audio.delta handler. Audio arrives up to half a second ahead of playback, so a recording taken there places the agent's words up to half a second away from when the caller heard them, and the stop time inherits that error.

Then hook the recorder into the room's tick in caller.py. Excerpt from caller.py:

 async def run(self): start, n = time.monotonic(), 0 while True: played = np.frombuffer(self.speaker.pull(CHUNK), dtype=np.int16) ... mic = np.frombuffer(said, dtype=np.int16).astype(np.float32) + self.air.popleft() * self.echo_gain mic = mic.clip(-32768, 32767).astype(np.int16).tobytes() if self.recorder: self.recorder.add(mic, played.tobytes()) # both sides, from the same tick await self.desk.send_audio(mic) ... def say(self, name: str, intent: str | None = None): self.desk.log("", f"(caller says {name}.wav)") if self.recorder: self.recorder.label(name, intent) self.voice.extend(read(name)) 

played is what the speaker gave the room on this tick. mic is the caller's line as the agent receives it; self.air and echo_gain can mix a delayed copy of the agent's audio back in to imitate laptop speakers, but the gain is zero in every run in this guide. Passing both to add() before send_audio() keeps the two channels in step. say() labels each line at the moment it starts to play, which gives the analyzer the caller's intent and a time to match it against.

Step 2: Find speech on each channel

With a stereo file in hand, the next job is to decide, frame by frame, when each channel is speaking. Here is frames_db() in bargein.py, which reads the WAV and measures every 20 ms frame:

def frames_db(path: Path, caller: str = "left") -> tuple[list[float], list[float]]: """Read a 16-bit stereo WAV and return the level of every 20 ms frame, in dBFS: (caller, agent).""" with wave.open(str(path)) as w: if w.getnchannels() != 2 or w.getsampwidth() != 2: raise SystemExit(f"{path}: needs a 16-bit two-channel WAV, not {w.getnchannels()} channel(s)") rate, pcm = w.getframerate(), array("h", w.readframes(w.getnframes())) if sys.byteorder == "big": pcm.byteswap() n = round(rate * FRAME_S) def levels(channel: array) -> list[float]: out = [] for i in range(0, len(channel) - n + 1, n): frame = channel[i:i + n] rms = math.sqrt(math.sumprod(frame, frame) / n) out.append(20 * math.log10(rms / 32768) if rms else -120.0) return out left, right = levels(pcm[0::2]), levels(pcm[1::2]) return (left, right) if caller == "left" else (right, left) 

The function refuses anything but 16-bit stereo and computes each frame's root-mean-square level with math.sumprod, which is fast enough in pure Python to score all 70 calls in about a second. Levels are in decibels relative to full scale (dBFS), with -120 for digital silence, and the caller argument says which side is the caller.

Levels alone are not enough, because speech is full of short silences between words, and an agent that pauses for 200 ms is not an agent that stopped. segments() turns the levels into runs of speech:

def segments(db: list[float], hangover_s: float) -> list[tuple[float, float]]: """Where a channel is speaking, as (start_s, end_s), with gaps between words held together. Speech starts above the on level and stops only after hangover_s below the off level. The end is the start of that quiet run: the moment the listener last heard it. """ floor = max(sorted(db)[len(db) // 10], FLOOR_DB) if db else FLOOR_DB on, off, hang = floor + ON_DB, floor + OFF_DB, round(hangover_s / FRAME_S) runs, start, quiet = [], None, 0 for i, level in enumerate(db): if start is None: if level >= on: start, quiet = i, 0 elif level < off: quiet += 1 if quiet == hang: runs.append((start * FRAME_S, (i + 1 - hang) * FRAME_S)) start = None else: quiet = 0 if start is not None: runs.append((start * FRAME_S, (len(db) - quiet) * FRAME_S)) return [(round(a, 2), round(b, 2)) for a, b in runs] 

The function first estimates the channel's noise floor as its tenth-percentile level, clamped at -70 dBFS (FLOOR_DB) so that a perfectly clean channel does not turn every stray sample into speech. Speech starts on a frame 16 dB above that floor (ON_DB) and only ends once frames fall below a lower level, 10 dB above it (OFF_DB). The gap between the two levels stops the analyzer flickering on and off as a voice hovers around one threshold. Speech ends only after hangover_s of continuous quiet, and the end it reports is the start of that quiet run, which is the moment the listener last heard the voice.

The first hangover I chose for the agent was wrong. These are the defaults at the top of bargein.py:

# How much quiet ends a channel's speech. The agent's has to outlast its longest pause between sentences and stay # shorter than its quickest answer after a stop. On api.studio.poly.ai's server voice (17 November 2026) the # longest pause inside a reply was 0.36 s, and the shortest quiet after a cut was 0.72 s. HANGOVER_S = {"caller": 0.4, "agent": 0.5} 

A pause longer than the hangover looks like a stop, and an answer that starts sooner than the hangover merges into the reply it interrupted. I started at 0.8 s, from pauses I had measured in narrated voice-overs. In the live calls, the server's voice never paused for more than 0.36 s inside a reply, but after "Stop." the agent began its answer 0.72 to 0.90 s after the cut, so at 0.8 s half of the "Stop." calls were scored as the agent talking on. At 0.5 s, every call scored correctly. If you score a different voice, measure its pauses first and pass --agent-hangover.

Step 3: Score every overlap

An overlap is a caller burst that starts while the agent is speaking. Each one becomes an Overlap, defined in bargein.py:

@dataclass class Overlap: file: str onset_s: float # when the caller started, on the recording's clock line: str | None # the label's name for what the caller said intent: str | None # barge-in, backchannel, background, or None with no label talked_s: float # how long the agent kept talking after the caller started result: str # stopped, talked on, or ended (the reply finished as the caller started) verdict: str | None # stopped, missed, ignored or false stop, once the intent is known 

talked_s is the stop time. result is what the audio shows, and verdict is what the audio means once you know what the caller intended. Here are the four constants that set the rules, and score(), which walks the caller's bursts and classifies each one. Excerpt from bargein.py:

LEAD_S = 0.2 # the agent must have been talking this long when the caller starts, or it's the agent cutting in REACT_S = 0.15 # no detector reacts faster: an agent that goes quiet sooner was finishing anyway MISS_S = 2.0 # still talking this long after the caller finished: it didn't stop STOP = {"barge-in", "turn"} # the intents that should stop the agent; backchannel and background should not ... def score(file: str, caller: list[tuple[float, float]], agent: list[tuple[float, float]], labels: list[dict]) -> list[Overlap]: """One Overlap for every time the caller started while the agent was already talking.""" out, unused = [], sorted(labels, key=lambda label: label["onset_s"]) for onset, end in caller: talking = next(((a, b) for a, b in agent if a + LEAD_S <= onset < b), None) if not talking: continue talked = round(talking[1] - onset, 2) result = "ended" if talked < REACT_S else "stopped" if talking[1] <= end + MISS_S else "talked on" # A label belongs to the first caller burst that starts at or up to a second after it. label = next((x for x in unused if x["onset_s"] - FRAME_S <= onset <= x["onset_s"] + 1.0), None) if label: unused.remove(label) intent = label and label.get("intent") verdict = None if intent and result != "ended": should_stop, stopped = intent in STOP, result == "stopped" verdict = {(True, True): "stopped", (True, False): "missed", (False, False): "ignored", (False, True): "false stop"}[should_stop, stopped] out.append(Overlap(file, onset, label and label.get("line"), intent, talked, result, verdict)) return out 

A burst only counts if the agent had already been talking for LEAD_S (0.2 s) when it started; otherwise the agent started over the caller, which is a different problem. The agent's segment end, minus the caller's onset, is the stop time.

A stop faster than REACT_S (0.15 s) is reported as ended, because no voice activity detector reacts that quickly: the reply was finishing on its own. An agent still talking MISS_S (2 s) after the caller's burst ends has talked on. The miss window counts from the end of the burst because a long sound, such as our 4.6 s TV clip, can stop the agent while it is still playing. Each label is matched to the first burst that starts within a second after it, and the pair of "should it stop?" and "did it stop?" becomes the verdict.

Note: Audio alone cannot tell a stop from an ending
When a caller starts talking just as a reply finishes, the recording looks exactly like a fast stop, which is how a log ends up reporting a barge-in handled in 0 ms. REACT_S skips the obvious cases, and the labels supply the intent, but the analyzer has no way to know what the agent had left to say.

Step 4: Test the analyzer on synthetic calls

Before the analyzer reads a real call, prove it on calls where you already know the answer. The tests build stereo WAVs from noise with a helper in tests/test_bargein.py:

def synth(path, agent, caller, gaps=0.15, agent_left=<...
Source:https://dev.to/pmckeetx/measure-barge-in-stop-time-from-a-stereo-call-recording-in-python-4pk0