Every TTS leaderboard gives you a couple of metrics to look at that are supposed to tell you if the voice vendor is “the best” and therefore, the one you should choose. One of those metrics is speed, and then there’s another about quality and it's usually asking listeners to rate vendors based on the "naturalness" of the voice.
What those benchmarks don’t answer are the questions you're actually wondering about if you're building voice for high-volume enterprise use cases: Will this voice read a tracking number correctly? Would a customer want to hear it explain a refund?
So we set out to measure three things separately to help builders make better decisions and along the way we decided to make it something we built in public, and gave everyone access to the methodology, the clips used, and everything you’d need to run it yourself.
The three new benchmarks we created to help voice builders are:
- AlphaBench checks whether a voice reads exact identifiers correctly, character by character
- SupportBench asks listeners which voice they'd rather hear on a customer support call
- ContentBench asks the same paired question about narration
Five TTS vendors (Rime, ElevenLabs, Deepgram, Cartesia, OpenAI), 1,730 scripts, and 32,450 human judgments from US listeners recruited through Podonos, a listener research platform. We think this nuance allows a builder to understand which voice works best for their specific use case instead of relying on a generic "natural" metric or latency alone.
If you want to skip the details, the outcomes are here.
Below are some data points from the research. We’ll share more about what we learned from building benchmarks in general (spoiler: it’s not easy!) in another piece.
The results
On support calls, listeners picked Rime over all four competitors: +0.90 on a five-point scale against OpenAI, +0.37 against Deepgram, +0.20 against Cartesia, +0.12 against ElevenLabs, every interval clear of zero and every comparison significant after Holm correction.
On narration, we beat Cartesia and OpenAI, and ElevenLabs and Deepgram beat us. Yes, we're publishing a benchmark we partly lose because we don’t build voice for content narration - we build for conversations. It’s a completely different job.
The third graph is an inventory rather than a leaderboard, and its largest entry is Deepgram's 84 clips flagged as misread by a majority of listeners, mostly omissions and early stops, against 6 to 16 for everyone else. Flags mark clips for human review; they aren't error rates.
Why do we win support and lose narration? Rime is built by linguists who study how people actually talk to each other, and Coda (our flagship TTS) is tuned for conversations with stakes: short turns, a refund amount and an apology in the same breath, digits a caller has to catch on the first pass. We built Coda for conversation. Narration gets whatever transfers.
AlphaBench: did it read that account number right?
A listener sees the full script, hears one clip, and flags whether any required content is missing, wrong, added, repeated, out of order, cut off, or unclear. Then they annotate exactly what they heard. The 580 scripts cover the stuff of real support calls: confirmation codes, order and tracking numbers, flight numbers, license plates, policy IDs, long codes, deliberately confusable sequences, and name spellings. For a code like XQ47B, the question is whether every character arrives intact. A pause or an unspoken hyphen isn't an error.
Listeners flagged things like omissions, substitutions, insertions, repetitions, reordering, early stops, and ambiguous readings.
SupportBench: which voice would you rather hear on this call?
Listeners hear two recordings of the same response and answer one question: "Which recording would you prefer to hear during a customer-support call?" One judgment, five-point scale, ties allowed. The 750 scripts mix conversational responses with the information customers need to catch: phone numbers, dates and times, amounts, addresses, and lines like "A supervisor has to sign off on the card replacement, but that usually takes minutes, not days."
Clarity, naturalness, and fit can all move a listener's answer, and so can a mistake they notice. This measures the spoken response.
ContentBench: would you keep listening?
Same paired structure, just different context: 400 passages of literature and authored narration, and the question "Which recording would you prefer for listening to this passage?". These are short excerpts, so the results don't establish comfort over a full audiobook.
The methodology
The run applies the framework to US-English material using one configured voice per provider. Judgments came from US listeners recruited through Podonos. Audio was level-matched to -23 LUFS under a -1 dBTP ceiling at 24 kHz (loudness-normalized, so no system wins on volume), with the platform's own normalization turned off. The design compares Rime with each competitor; it's a set of head-to-heads rather than an all-pairs tournament.
We average the ratings within each script, then weight scripts equally, so an item with seven ratings can't outvote one with three. Confidence intervals come from a family-clustered bootstrap (10,000 replicates), the paired test is a family-level sign-flip test, and Holm correction covers the four comparisons within each benchmark.
ContentBench also gets an author/genre sensitivity analysis, which qualifies the primary results without changing any direction.
The results show that SupportBench favored Rime in all four comparisons, ContentBench favored Rime over Cartesia and OpenAI and favored Deepgram and ElevenLabs over Rime.
The results don't explain why preferences differ across tasks, they don't cover other voices from these providers, and the intervals account for shared source material but not for the same listener rating many scripts. Treat them as a reason to run your own evaluation, and use the repo contents as a template.
The full set of benchmarks are open-source
The corpora, tasks, synthesis code, and analysis are all in the GitHub repo. Let us know if you ran this yourself with other vendors and what you found!
More from the lab
.png)
Rime: The voice behind Dialpad's agentic AI
Dialpad named Rime its premier voice model provider. Our models will speak for every agentic voice product across their global communications platform.
Writing for the ear: Prompting your TTS to sound human
Rime was founded by linguists. We spent years studying how people actually talk before we trained models on it, so our prompting guide teaches something different: how to transcribe speech the way it’s spoken. Below is the short version of that guide and gives some context behind why we do it this way.
.png)