Aligning an audiobook to an ebook on iPhone with SpeechAnalyzer

Ebook and audiobook sync: the problem

There are three ways a book can come into the app. You bring text, and Kokoro reads it. You bring text and an audiobook, and the narrator reads it. You bring only an audiobook, and the words come from recognition. This post is about the middle one.

Bring            Text from     Audio from   Sync from
Text             the file      Kokoro       Kokoro's timings
Text + audio     the file      the narrator alignment
Audio            recognition   the narrator recognition's timings

In the middle case two files describe the same words and share nothing. The audiobook might be one 13-hour M4B or 59 MP3s. It might have chapter markers or none. The ebook has front matter, an epigraph and a licence page that nobody reads aloud. The narrator says "Chapter one" where the page says "I". Somewhere in there is the thing the reader actually wants: put down the phone while reading, press listen, and hear the next sentence, not the next chapter.

That is the bar we set, and ADR-0014 says it plainly: what sync has to get right is the place, which sentence the voice is in. Word-by-word following is polish, judged by eye, and gates nothing. Keep that in mind for the numbers below. We measure sentences.

Pairing is explicit. You press "Add audiobook" on a book. We considered matching an audiobook to a book by title and rejected it. Files are copied into the book's folder, hashed, and described in a manifest (durations, tags, chapter markers, where each chapter was found). A multi-file set is laid on one timeline, so everything downstream works in seconds from the start of the set. Audiobooks with DRM (Audible, Apple Books) are refused with a message that says why. We don't attempt them.

Forced alignment with SpeechAnalyzer: why not Whisper

The recogniser is SpeechAnalyzer with a SpeechTranscriber, new in iOS 26 and macOS 26, set to en-US and asking for word time ranges:

let transcriber = SpeechTranscriber(locale: locale, transcriptionOptions: [],
                                    reportingOptions: [],
                                    attributeOptions: [.audioTimeRange])

It is not strictly forced alignment. We never give it the text. It transcribes, and we align the transcript to the book afterwards. That turned out to be the right shape for books, because a narrator leaves things out and adds things, and a recogniser that has been told the answer will not admit it.

We picked it in ADR-0014, and the argument was about what we would have to carry. The alternative was to stay on iOS 18 and ship WhisperKit: a second set of weights in the bundle or a download, a second engine in memory beside MLX Kokoro to measure and keep there, and a model to maintain. We also rejected keeping iOS 18 and gating the recogniser to 26, because that is two code paths for the one feature that decides whether an audiobook can come in. SpeechAnalyzer is on both platforms at the new floor, gives word timings, bundles no model, and loads none beside MLX. The speech model is installed by the system through AssetInventory; the first time, the app shows a "Downloading speech recognition" line. We had not shipped 1.0, so moving the deployment target to iOS 26 stranded nobody.

The recogniser sits behind a small protocol, so tests hand the aligner scripted words with known times and need no speech model. Spelling corrections a reader makes (a name fixed once) are also fed to the analyser as contextual strings, so the name is heard right afterwards. How much that helps later chapters is something we have not measured.

Chunking audio for SpeechAnalyzer: stretches, not books

The aligner never recognises "a book". It recognises a stretch of the set's timeline. The stretches it uses are a 45-second window at the top of a file or chapter marker, a chapter's first 180 seconds, and then 600-second strides as the listener gets near the end of what has been heard. A hard ceiling of four hours is a lesson from the story below. A chapter is heard progressively and aligned whole: anything that reasons about a chapter, such as whether it is read at all, only sees complete ones.

The feed is pull-based. About one second of audio is decoded at a time, converted to the analyser's preferred format (16 kHz Int16, measured on the way), and handed over when the analyser asks for it. Recognition is foreground-only, for the same reason Kokoro rendering is: a backgrounded app that keeps a core busy is terminated. Leaving the foreground stops a long listen in 0.4 seconds, and the renderer takes the same chapter again when you come back.

Matching recognised words to the ebook: unique anchors, a rising chain, and LCS

The matcher is one pure function: book sentences and recognised words in, sentences with times out. It is the part we are happiest with, because it is small enough to test exhaustively. The first prototype used Python's difflib matching blocks as anchors. The Swift version replaced it.

Both sides are normalised the same way: lowercase, keep a to z, digits and apostrophes, fold curly apostrophes to straight. Then:

1. Anchors. Take every run of three words. A run is an anchor if it appears exactly once in the book window and exactly once in the heard window. A run that occurs twice proves nothing, because you can't tell which occurrence you heard.
2. A rising chain. Sort the anchors by book position and keep the longest chain that also rises in heard position, using patience sorting in O(n log n). An anchor that appears once in each but in a different order is a coincidence, and the chain drops it.
3. Gap fill. Recurse into the windows between consecutive anchors. A window with no three-word anchors tries two-word ones, then single words. Whatever is left goes to a plain longest-common-subsequence, but only for windows up to 400 by 400 cells. Anything bigger is left unmatched instead of paid for quadratically, since a window that large is almost always something the narrator didn't read.
4. Interpolation. A book word nobody heard takes an even slice of the time between its matched neighbours. Before the first heard word and after the last, it takes none: the preamble and the credits belong to no sentence.

var anchors: [(Int, Int)] = []
for (gram, positions) in inA where positions.count == 1 {
  if let theirs = inB[gram], theirs.count == 1 { anchors.append((positions[0], theirs[0])) }
}
anchors.sort { $0.0 < $1.0 }
// The longest chain that also rises in b: a run that appears once in each
// but in a different order is a coincidence, not an anchor.
let chain = risingChain(anchors)

A sentence's span is its first word's start to its last word's end. With the narrator, each sentence then runs on to the next one's first word, so the sentences tile the chapter with no hole between them.

The 12-word rule: when the narrator skipped it

Narrators skip epigraphs, tables and footnotes. If more than 12 consecutive book words have nothing heard, any sentence wholly inside that run is marked unnarrated. It reads as text, gets a short silence, and the emphasis jumps over it. Twelve or fewer is treated as the recogniser missing a phrase, and the sentence takes its neighbours' time. One matched word anywhere in a sentence clears the flag.

The 30% refusal

A whole chapter the narrator never reads, a glossary or a licence, would otherwise be force-fitted to whatever audio is nearby. So the confidence number is the share of book words matched, and below 30% the chapter is refused with a message saying how few of its words were heard. The verdict is written into the manifest along with a rules version, so a changed rule asks again. The same figure is the first thing in our log for every chapter:

aligner: chapter N in A-B s, P% of words heard, K of M sentences narrated

Finding each chapter in the audio

Matching assumes you already know roughly where the chapter is. Finding it is the harder problem, and it is where most of the bugs lived. The aligner tries, in order:

1. What the manifest already knows. An earlier probe, seek or import wrote it down.
2. Structure. For a multi-file set, recognise the first 45 seconds of each file in order, and only until the chapter we want is bracketed by one located before and one after. For a single file with chapter markers, do the same from each marker. Each top is matched against the openings of every unlocated chapter. On a 59-file LibriVox set, all 59 files were located to their chapter from their first 45 seconds.
3. Proportion and rare vocabulary. A single file with no markers has no structure. Guess where the chapter should be from its share of the book's sentences, hear a 45-second window there, and work out which chapter it came from by its rare words, those appearing in three chapters or fewer. Then correct the guess, at most eight windows, and bisect the chapter's edge in at most twelve more.
4. A walk. The last resort: ten-minute strides from the last known chapter, never from the front unless nothing is known.

Step 3 exists because of a person waiting. We were at chapter 65 of a very long audiobook, attached it, pressed play, and the aligner walked twenty ten-minute strides from the front while we watched a spinner. The identifier is a small vote: each distinct rare word heard scores 1/n for each of the n chapters it appears in, and a result needs two points and a runner-up under half the winner.

for word in words {
  let token = TextAlignment.normalise(word.text)
  guard seen.insert(token).inserted, let chapters = index[token] else { continue }
  for chapter in chapters { score[chapter, default: 0] += 1 / Double(chapters.count) }
}
let ranked = score.sorted { $0.value > $1.value }
guard let best = ranked.first, best.value >= 2 else { return nil }
if let second = ranked.dropFirst().first, second.value * 2 > best.value { return nil }

Chapter headings are left out of the opening used for matching. "Chapter I." alone matches the top of every chapter.

Front matter: chapter index is not chapter number

An EPUB's table of contents begins with a title page, a copyright page, a dedication and sometimes an epigraph. The aligner keys everything by position in that list. In one 13-hour commercial audiobook, "Chapter 1" is index 4. Any code that assumes n + 1 is the next chapter is wrong the first time it meets a book with a dedication, and the fallbacks that say "Chapter n+1" to the reader shipped that wrongness to the screen. The aligner now asks for the next chapter in the book, not the next number.

Front matter has a second cost: the audio. Before Chapter 1 begins, the narrator says a title, a credit, and "Chapter one". We chose to put all of that in no sentence, and the player steps over it. That is a choice a reader may one day want reversed, and the code says so, in the one place to reverse it.

The narrator's audio is played as it is

The first build cut each sentence out of the audiobook and re-encoded it into the same 24 kHz mono AAC we use for Kokoro. It was the cheap option, and nothing downstream had to change. We rejected it the same day it landed. Our own note from that day: "I do not want to reduce quality. If it is a high quality audiobook with stereo, we need to keep that."

So a sentence is not audio. It is a reference: which file, which frame, how many frames, at that file's own sample rate and channel count. Stereo stays stereo. 44.1 kHz stays 44.1 kHz. Nothing is decoded for output while aligning, and the player configures its output to the file's format and reads exactly those frames from the file you brought.

return SentenceStore.Reference(source: source, startFrame: startFrame,
                               frames: span.isNarrated ? endFrame - startFrame : 0,
                               sampleRate: rate, channels: Int(format.channelCount))

Because sentences tile the chapter, the narrator's pauses are the narrator's. The player puts no beat between sentences for a human voice, where Kokoro gets 0.18 seconds. The index "learned a second kind of entry" and the sentence store did not change, and nothing downstream knows whose voice it is. The only audio that belongs to no sentence is whatever comes before a chapter's first heard word.

Numbers: how fast, how accurate, how much memory

All of these are from an iPhone 16 Pro Max (A18, 8 GB) or an M-series Mac. That is the best case. ADR-0014 puts the floor at an iPhone 11. Read them as a ceiling.

Recognition speed
iPhone 16 Pro Max   72.6x realtime   764 s of audio in 10.5 s (a LibriVox chapter)
M-series Mac        ~75x realtime    mean over all 59 chapters of that book

End to end on the Mac (real recogniser, real files)
a 4.3 h fantasy novel, one M4B      249 s   2868 of 2976 sentences narrated (96.4%)
a 20.5 h LibriVox novel, 59 MP3s    925 s   9108 of 9297 sentences narrated (98.0%)

Accuracy
LibriVox novel, 59 chapters (Python prototype)   97.7% of words matched, 93.6% worst chapter
the 4.3 h fantasy novel (Swift)                  95.4% matched, 94.9% worst chapter,
                                                 3230 of 3255 sentences anchored, all monotonic
a short LibriVox chapter (the App Store sample)  66 of 66 sentences aligned, 97% words heard

Memory on the phone, one long listen
3171 s chapter   peaked 173 MB
1528 s chapter   peaked 176 MB
(before the fix below: 2148 MB)

Two caveats. The 97.7% comes from the Python prototype, not the Swift aligner; the Swift figures are the 95.4% and the per-book narrated percentages. And the word starts are only spot-checked: they sit within about a tenth of a second of the audible word on the dozen we checked. That is not a measurement of karaoke accuracy, and we don't claim one. The unmatched words in the fantasy novel are invented names, which is a recogniser problem and not a matcher problem.

Four things that broke

2.1 GB while "aligning": one 18-hour recognition call

The phone showed 2.1 GB used while a long audiobook aligned. The memory trace read like a leak: 170 to 190 MB through the back matter, about 313 MB through chapters 4 to 10, then a jump from 795 to 2148 MB with nothing logged. It wasn't a leak. The manifest had chapters 0 to 11 located and nothing after until 65. Our "is the chapter after this one located?" check really meant "is any later chapter located?", so chapter 11's stretch ran from 8,749 s to 73,659 s. That is 64,910 seconds, 18 hours, in one recognise call.

The arithmetic then closes it. At 16 kHz Int16, 64,910 seconds is 2.08 GB. Observed growth was 1.92 GB. The input was a pushed AsyncStream, decoded at disk speed far ahead of an analyser that runs at about 50 times realtime, so every buffer waited in memory. The fixes were three. The feed became pull-based. A four-hour ceiling went on any stretch. And the next chapter is now sought before a suspicious stretch is heard.

// The analyser asks for its audio; nothing is pushed at it. It takes what
// it is about to hear - a couple of minutes ahead, 13 MB measured - where a
// stream it was fed held every buffer until the last word: an hour in 119
// MB, and eighteen hours in 2 GB on the phone (#90).
let feed = RecognitionFeed(chunks: audio.chunks(from: start, to: end),
                           from: audio.format, to: target)

The lesson is the one from the Kokoro post: a number on the dashboard is not the cause. Do the arithmetic against the log.

SpeechAnalyzer never finishes: the negative-length range

A test of every pair of books hung for hours on a LibriVox novel, at chapter 1, with no thread busy. Two faults had stacked. First, 58 of the 59 MP3s had no track tag and one, chapter 38, had "38". A tagged file sorted before every untagged one, so the timeline opened on chapter 38. The rule is now that a tag orders files only if every file carries it. Second, with the timeline wrong, chapter 1's stretch ran from 2,019 s to 29 s. The end was before the start.

We fed SpeechAnalyzer no audio. It finalises, and then never ends its results sequence. Our await on the collected words never returned, and nothing in the process was busy to point at. The guard is one line:

// Nothing to hear is nothing heard. An analyzer fed no audio finalises
// and then never ends its results, and the await below never returns.
guard end > start else { return [] }

A hang with no busy thread is a result stream that was never closed. Look at what you fed it before you look at what it is doing.

The sentence the player asked for 7,073 times in 55 seconds

"Gibson bowed his head." showed "Preparing" forever. The sentence store matches an entry only under the pronunciation fingerprint the caller expects. The aligner wrote the narrator's sentences under an untouched fingerprint, since no pronunciation override can reach a human recording. The player built its own from the text, and since we added heteronym rules, "bowed" is a heteronym, so its fingerprint differed. The renderer said ready. The player said not found. It asked 7,073 times in 55 seconds at full CPU. Nothing on disk needed repair; the index was right all along. Now the player keys narrator chapters the way the aligner does, and a mismatch rests the player with a message instead of spinning.

Chapter 1 started at sentence 14

After a crash fix on the 13-hour audiobook (a sentence stored as "the rest of the file", 1,073,519,123 frames, which became an 8.6 GB buffer request and an std::overflow_error), "Listen from here" on Chapter 1's first word started at sentence 14. The first thirteen were skipped. The audio of Chapter 1 opens at about 15 seconds. We had located it at 67 to 69.

The aligner reaches back from a chapter's located start to where the previous chapter was last heard, to catch an opening it found late. Two checks stopped it. One looked back only three chapters, and the Copyright, Dedication and Epigraph pages, all unread, are three. The other counted those unread chapters' written-down starts as chapters in between. Neither should count an unread chapter. Now Chapter 1 is heard from 0:08 and all 270 of 270 sentences are narrated, where it was 256. Our first note on this had blamed "credits and the epigraph". It was wrong: the epigraph isn't read at all, and its "23% heard" was a false match inside Chapter 1's audio. We left the correction in the issue.

What's still unsolved

• We have not measured an iPhone 11. Everything above is A18 or a Mac. The nearest measured device to the floor is an iPhone 13 for the Kokoro side. No A13 number exists for recognition or alignment.
• Recognition beside live MLX weights is unmeasured. The two models share a process and we have not logged the peak with both resident.
• Different editions. We built and tested on pairs that match closely. A commercial audiobook of a different edition than the ebook is something we haven't studied systematically.
• English only. The recogniser is created for en-US and we found no other path.
• Names and numbers. Invented names are the main source of unmatched words. The recogniser writes "ten years" as "10 years" and "millions" as a number with a suffix. Both still cost matches.
• Narrator skips we can't explain. One chapter of the 4.3-hour novel matches 67 of 157 sentences in every run. Either the audiobook abridges it or its stretch is misbounded, and we have not diagnosed which.
• The window before the next chapter is prepared. If you reach the end of a chapter before the next one is aligned, you wait.
• Two fixes await the phone. The Chapter 1 fix and the title-page fix were uploaded to TestFlight without our phone checklist, and their issues are open until we confirm them by ear.
• A reported mismatch we couldn't reproduce. A tester reported the voice drifting from the text in one book. We could not reproduce it. Settings now has a "Recalculate alignment" action that discards the aligned recordings and rebuilds them from the original files. It is a recovery tool, not evidence that the mismatch is fixed.
• Sentences that span two files are clamped to one. We think it is rare enough not to justify a second kind of reference.

What we'd tell anyone aligning audio to text

FAQ

Can SpeechAnalyzer do forced alignment of an audiobook to an ebook?

Not directly. SpeechTranscriber transcribes and, with the .audioTimeRange attribute option, reports a time range for every word. You then align that transcript to the book's text yourself. We use unique n-gram anchors, a longest rising chain, a bounded longest-common-subsequence for the gaps, and interpolation for words nobody heard.

Why does SpeechAnalyzer hang and never finish its results?

If it is fed no audio, it finalises and then never ends its results sequence, so an await on the collected words never returns. We hit this with a negative-length range, a stretch whose end was before its start. Return early when end > start is false.

How fast is on-device audiobook recognition on iPhone?

On an iPhone 16 Pro Max, 764 seconds of audio was recognised in 10.5 seconds, about 72.6 times realtime. A Mac averaged about 75 times realtime over a 59-chapter book. We have not measured an older phone.

Does syncing an audiobook with an ebook re-encode the audio?

Not in Unison Reader. Each sentence is stored as a file, a start frame and a frame count at the original sample rate and channel count. The player reads those frames from the file you imported.

Why does the app refuse some chapters?

If fewer than 30% of a chapter's words are heard in the audio near where it should be, the narration probably doesn't read it, as with a glossary or licence. It is marked unread and the player moves on.

Unison Reader is on the App Store. It reads your ebooks aloud with Kokoro, or plays your own audiobook against the same text. More at unisonreader.com, or go straight to the App Store listing.

Kyle Twogood is the founder of Magrathea Software. He’s been building production software since 1997.