We build DubbingMate, a tool that reads a YouTube video's captions out loud in another language.
Getting the words right turned out to be the easy part. Keeping a synthetic voice in step with a video that the viewer pauses, seeks and speeds up was most of the work.
Here is what our sync code does today, with the numbers we ended up using.
The panel on the right: dubbing is on, the original audio is ducked to 20%, the dub plays at 100%.
1. Captions are the timeline
Every caption cue has a start and an end time. That is the only clock we trust. Before playback we clean the cue list: if a cue ends after the next one starts, we cut its end to the next start, so two lines never compete for the same second.
If a video has no captions at all, there is nothing to dub. We do not run our own speech recognition, and the app says so instead of guessing.
2. The voice is often longer than the cue
A sentence that takes 2.1 seconds in English can take 3 seconds in Japanese. We handle the overflow in steps, cheapest first:
| Step | What we change | Limit |
|---|---|---|
| 1 | Speak faster | 1.5x for the local neural voice, 1.9x for the browser's Web Speech voice |
| 2 | Slow the video slightly | down to 0.65x (local voice) |
| 3 | Emergency speed-up | 1.9x |
| 4 | Skip the line | only if we fall more than 3 s behind the cue end |
The order is a judgement call. Our reasoning: a slightly slower video is easier to accept than a squeaky voice, and a missing line loses content, so skipping comes last. There is also a constant-speed mode where we pause the video for a moment instead of changing any speed.
3. Detecting seeks without trusting events
We poll the player every 250 ms. If currentTime jumps by more than 2 seconds between polls, we treat it as a seek: cancel the current utterance, wait 300 ms (people often drag the scrubber in several hops), then restart from the cue at the new position.
Playback-rate changes are picked up by the same poll. Pause and resume come from player state changes, and an ad starting resets the queue.
4. Ducking, and defending it
When dubbing turns on, we save the original volume and drop it to 20%. When it turns off, we restore it. If something else changes the volume while dubbing is on, a volumechange listener puts it back, so the dub stays on top.
The local voice output is peak-normalised to 0.7, so a quiet sentence and a loud one sit at a similar level.
5. Two voices, two budgets
The default is the browser's built-in Web Speech voice: zero download, but its timing is less predictable. The other option is a local ONNX voice (Supertonic) running in an offscreen document. We control its output and can measure its length, which is why its speed cap is lower: we would rather slow the video than distort it.
What we would tell anyone building this
- Treat caption timestamps as the source of truth, not the audio element.
- Poll for seeks. Debounce them.
- Rank your fixes by how much a viewer notices them, and skip content last.
If you want to try it: DubbingMate. All features are free for 72 hours from your first dub, no card. After that it is a one-time purchase (price on the site). Limits: videos without captions cannot be dubbed, and translation quality depends on the captions.
Questions about the sync logic are welcome in the comments.
