I work on YTVT (Your Voice Translated), a Chrome and Edge extension that translates the speech in a video while it plays. You press Start, and a few seconds later you hear an AI voice in your language and see the original and the translation side by side in a subtitle panel.

The translation itself comes from realtime speech-to-speech models. We support several, and they are good at it. What took most of our time was not translation. It was time: keeping the translated voice, the subtitles and the video in agreement.

Below are five timing bugs we found and fixed, with the numbers that told us they were real. The numbers come from our own test runs, most of them 150 seconds long on the same English video. Treat them as examples, not benchmarks.

Three clocks

Almost every bug in this post is one clock being read as another, so it helps to name them first.

  • Video clock: video.currentTime. The viewer can pause, seek or sit through an ad.
  • Uplink clock: how much source audio we have sent. We send 16 kHz mono PCM16 in 200 ms frames, so 32 bytes is one millisecond.
  • Dub clock: how much translated audio has actually played.

Each subtitle row has to land on the video clock, each cue has to appear on the dub clock, and the uplink clock is the bridge between the two.

1. The dub is longer than the speech

Translated speech is often longer than the original. On one engine and language pair, the translated audio ran 1.31× the length of the source (median of 38 runs). Play that at normal speed and the voice falls further behind every minute.

Our first player had a jitter buffer that dropped the oldest audio once it held more than about three seconds. In one 150-second run it dropped 26 times and lost 33 seconds of speech. Users described it as "the dub keeps getting cut off".

We first blamed the silence trimmer, which shortens pauses between sentences. Then we did the arithmetic: the trimmer could recover at most 38 seconds, and the backlog needed 53. The budget did not close, so something else was eating audio. It was the drop. When a budget doesn't close, find the missing sink before designing a fix.

The fix was to speed speech up gently before it enters the queue. We use WSOLA (waveform similarity overlap-add), which changes speed without changing pitch, with 30 ms windows and a ±8 ms search. The rate depends on how far the queue is past its target:

const TARGET_MS = 2100; const MAX_RATE = 1.15; function playbackRate(backlogMs: number): number { const start = TARGET_MS + 500; // start speeding up here const full = TARGET_MS + 2000; // reach MAX_RATE here if (backlogMs <= start) return 1; if (backlogMs >= full) return MAX_RATE; return 1 + ((MAX_RATE - 1) * (backlogMs - start)) / (full - start); }

A hard guard still drops audio past 8 seconds, as a last resort. We chose 1.15× by listening to offline replays of real sessions at several rates. The verification run had zero drops, with the queue at 3.1 s median and 6.3 s at its worst. The cost is speech up to 15% faster, and a dub that can trail by a couple of extra seconds at peaks. We decided that losing no words was worth it.

2. Choppy playback when the model is slow

Sometimes the model supplies audio slower than real time, for example 320 ms of audio every 0.6 to 1 second. When the player runs dry it pauses and waits for some audio to build up before it resumes. Ours resumed at 300 ms, a value tuned for an earlier engine that streamed continuously. With slow supply, every frame played on its own: sound, gap, sound, gap.

We gave it a metric: a chop is a resume followed by another underrun within one second. Normal runs had none. Runs on a slow afternoon had 6 to 9.

The fix had three parts:

  • Resume at 900 ms of buffered audio instead of 300.
  • When the model says a response is finished, play whatever is held, even if it is short.
  • If held audio has waited 2.5 seconds with nothing new, play it anyway.

The third rule had its own bug. We measured the 2.5 seconds from the moment the player paused, not from the last audio that arrived. So the first frame of a new sentence was often released alone, right as it arrived. In one Japanese-to-English run, 11 out of 11 releases were this case. Measure waits from the last event that matters.

One warning if you test this kind of thing: a recording made by stitching together the frames you received will sound perfectly smooth. Count playback events, or record the real output.

3. Subtitles racing ahead of the voice

The server spreads each response's subtitle cues across that response's dub audio. Our wrapper around one engine cut a single model response into many pieces, using a 5-second audio cap and a 1.2-second idle timer. Each piece then re-planned all the text so far onto its own short slice of audio.

Every cue hit our 650 ms minimum, and a response's subtitles ran out a third of the way into its audio. In one run, 4 model responses became 43 playback segments, and in 31 of the 43 plans every single cue sat on the minimum.

We didn't fix it straight away. First we logged a close reason at every place a segment could end. That showed the 5-second cap was doing most of the cutting. It also showed that the obvious one-line guard would never have fired: the idle timer had already destroyed the object that knew where the model's response really ended.

The fix makes the model's own response boundary the authority, and plans cues over the measured dub length. After it, 4 responses made 4 segments, and cue span divided by audio length went from a median of 1.65 to 1.00. The number of distinct subtitle lines that lit up in 150 seconds went from 18 to about 70.

4. Pairing each sentence with its translation

The subtitle panel works best when each source sentence and its translation appear together, at the moment that sentence's dub starts. One of our engines streams both texts as it goes. In its logs we found:

  • Within a turn, completed source sentences and completed translated sentences alternate 93 to 100% of the time.
  • The translated sentence completes about 35 ms after its source (median).
  • A translated sentence never completed before its source.

So we pair sentences simply by arrival order. The interesting case is a speaker who runs on for 18 seconds without a full stop while the translation has already ended three sentences:

const RUN_ON_MIN_MS = 1000; function onTranslatedSentenceEnd(s: PairState, nowMs: number): void { if (s.newSourceEnds > 0) { pairWithLatestSourceSentence(s); } else if (s.sourceTailOpen && nowMs - s.lastSourceEndMs >= RUN_ON_MIN_MS) { pairWithSentenceStillBeingSpoken(s); } else { mergeIntoPreviousPair(s); // the translation split one sentence in two } }

Replayed over real sessions, the run-on rule changed 16 of 279 pairs, and all 16 went from wrong to right. Overall, panel blocks per run went from 13 to 34, and English text per block dropped from about 200 characters to 81. The share of moments where the translation showed ahead of its source went from 16–18% to 0%.

Getting there took three more fixes in the video overlay, and the unit tests passed every time the screen was wrong. Each test handed the final rendering function the full text, so it skipped the step that broke it: the content script replaced the text with only the unfinished sentence before rendering. A test that skips a step proves nothing about that step. We ended up confirming that wiring with a real end-to-end run.

5. Putting each row on the video clock

Each panel row shows the video time when its sentence was spoken, and clicking it seeks there. We map the model's audio position to video time through anchors: pairs of video.currentTime and the uplink position, sampled as the video plays. That mapping had three bugs.

Late acknowledgements. The server acknowledges each frame with how much audio it has received. We compared that number with our local count when the acknowledgement arrived. On a slow downlink, acknowledgements arrived up to 20 seconds late, which looked exactly like the server falling behind, and rows drifted by up to 20 seconds. Now we remember the local count at the moment each frame was sent, and compare against that:

const BYTES_PER_MS = 32; // 16 kHz × 2 bytes const sentMsBySeq = new Map<number, number>(); let localMs = 0; let correctionMs = 0; function onFrameSent(seq: number, bytes: number): void { localMs += bytes / BYTES_PER_MS; sentMsBySeq.set(seq, localMs); } function onAck(seq: number, serverReceivedMs: number): void { const atSend = sentMsBySeq.get(seq); if (atSend !== undefined) correctionMs = serverReceivedMs - atSend; }

With the downlink delayed 0.4 to 4.2 seconds, the correction now stays at zero. Rows land within 0.3 s of a reference run at the median and 0.8 s at p90.

Reconnects. Every reconnect starts a new server session, so the model's audio position starts again at zero. Rows after a reconnect were mapped through the old anchors and ended up 67 seconds off. The client now keeps one timeline for the whole session: on disconnect it adds the old connection's position to an offset.

Sentences that span two turns. One engine sometimes ends a turn mid-sentence, and the panel joins the two halves into one row. That row took its start time from the second turn, 2 to 5 seconds late. Taking the start from the turn where the sentence began brought p90 error from 3.0 s to 0.5 s.

How we measured: the video's own captions

You need a ground truth that is independent of your own pipeline. We used the video's caption track.

The trap: many caption tracks are line-level, with one start time for a line lasting 0.7 to 2.2 seconds. Comparing a word in the middle of a line against the line's start makes every timestamp look late. We nearly "fixed" a lateness of half a second that didn't exist. The fix is to place each word inside its line by character position:

function wordTimeMs(line: CaptionLine, charOffset: number): number { return line.startMs + line.durationMs * (charOffset / line.text.length); }

On the English test video after these fixes, row start minus caption time is 0.2 to 0.4 s at the median and 0.3 to 1.25 s at p90, depending on the engine. Across six source languages on one engine, the median ranged from −0.4 to 0.0 s, and p90 error was about a second or less.

A second trap: we had two logs written by two different capture processes. Compared directly, a change looked like agreement dropping from 68% to 22%. The logs were simply 6 to 8 seconds apart. Before comparing timestamps from two places, line them up on one event both of them recorded.

What I'd tell myself at the start

  • Name your clocks, and put the clock in the variable name.
  • When a budget doesn't close, look for the missing sink.
  • Log why every segment ends before you change how segments end.
  • Check your ground truth's granularity before trusting a measurement.
  • Don't drop zero-length things by reflex. One speech-to-text stream sent the last word before a pause with zero duration, and filtering those cut rows off mid-word ("a lot of inc").
  • Listen. The metrics found the chops; our ears picked 1.15×.

YTVT runs on desktop Chrome and Edge, and you can try it with 15 free minutes after verifying your email: ytvt.app. There is a step-by-step guide if you want to see the subtitle panel and the three playback modes. I'm happy to answer questions in the comments.