When I started building CocoReader: Book to Audiobook, I had a pretty simple idea. I wanted the reader to highlight the sentence being spoken and follow the current word as the voice moves through the page.
The obvious approach was to use the speech synthesizer's callbacks and highlight whatever text range it reports. I tried that, but it wasn't reliable enough.
The problem is that this needs to work across EPUBs, PDFs, web articles and even photos of printed pages. It also needs to handle pauses, seeking, speed changes and audio that hasn't been generated yet. On top of that, the text on the page isn't always the same as the text being sent to the speech engine. Drop caps, footnotes and chapter titles all make things more complicated.
So instead of relying on callbacks from the synthesizer, I built the highlighting around a clock.
One source of truth for words
Everything uses the same word list for the chapter. Audio playback, highlighting and seeking all reference the same word indices.
This is important because if each part has its own idea of where a word starts and ends, things eventually get out of sync.
Building that list isn't as straightforward as it sounds. A drop cap might be split across two elements but still represent one spoken word. Footnote numbers exist in the text but aren't spoken, so the highlight needs to skip them. Chapter headings are spoken once and followed by a pause. Repeated book titles at the beginning of chapters are skipped entirely.
We handle these cases directly in the word list.
We also account for paragraph endings, headings and scene breaks by inserting short pauses in the audio timeline. During these pauses, the highlight stays on the last spoken word instead of immediately jumping ahead.
The goal is to make it feel like someone is actually reading a book rather than playing back a stream of generated words.
We generate the audio and handle playback ourselves
Instead of letting the speech synthesizer control playback, we generate speech in chunks and write the resulting audio to our own player.
Even Apple's system voices go through this same pipeline.
The reason is pretty simple. If the synthesizer controls playback, we're limited to whatever timing information it provides. Some engines don't provide enough information to reliably track individual words.
I also wanted playback to use the same clock everywhere, including the reader, lock screen, scrubber, Control Centre and car controls.
Every speech engine is essentially just an audio source. Playback and timing are controlled by CocoReader.
How the word clock works
A display link drives the playback loop. On every tick, we check the current audio time and ask the timeline which word should be highlighted.
The timeline is built from the audio chunks we've actually generated. Each chunk knows which words it contains and how long the audio lasts.
Within each chunk, we distribute the duration evenly across its words. This isn't perfectly accurate because longer words naturally take more time than shorter ones, but it gives us a predictable timeline that stays in sync with playback.
It also means we never highlight words from audio that hasn't been generated yet.
Before audio is available, we use an estimated reading speed of 175 words per minute so the scrubber has an initial timeline to work with. Once audio chunks arrive, we recalibrate using the actual number of words and seconds generated.
The scrubber is still based on the word list rather than the estimated duration. This prevents it from jumping around whenever the reading speed gets recalculated.
There are a few rules we enforce to keep everything in sync.
- The highlight never moves beyond the last word we've generated.
- When seeking, the highlight stays on the requested word until the new audio starts playing.
- Skipping forward or backwards uses the current voice's reading speed, so a 15 second skip actually represents 15 seconds of playback.
Rendering the highlight
Once we know which word is being spoken, rendering it is relatively straightforward.
The sentence being read gets a continuous highlight, with the current word rendered as a solid chip on top.
In the EPUB reader, these are drawn as decorations over the existing book layout. This lets us preserve the publisher's original styling instead of modifying the underlying text.
Other readers use the same word ranges but render them directly over their text content.
User highlights are also kept separate from playback highlighting. They stay underneath and remain unchanged as the voice moves through the document.
What I'd improve next
The biggest compromise right now is how we distribute audio duration evenly across words within a chunk.
Some speech engines already provide better per-word timing information, and we capture that data. The next step is making better use of those timestamps without allowing inaccurate timing information to move the highlight to the wrong word.
I also want to improve the transition immediately after seeking. Keeping the highlight on the requested word while new audio is being generated works, but the transition back to normal playback should be completely seamless.
Most of this is infrastructure that users shouldn't have to think about.
You should just be able to look at the page and see the word you're hearing. That's really the whole point.
