Writing for the ear: Prompting your TTS to sound human

Every text-to-speech (TTS) vendor publishes formatting rules that say the same thing: use normal casing, don’t send markdown, and wrap your confirmation codes. Even if you follow all these guidelines, your agent may pronounce everything correctly but still sound like it’s reading a press release into the phone. 

The hardest part about making TTS sound human, that no one talks about, is that LLMs are trained to write prose for readers, and voice agents need speech for listeners. Of course no one’s system prompt says “sound like a memo” but that’s the default nature of LLMs and no amount of casing rules will fix it.

Rime was founded by linguists. We spent years studying how people actually talk before we trained models on it, so our prompting guide teaches something different: how to transcribe speech the way it’s spoken. Below is the short version of that guide and gives some context behind why we do it this way.

Background

Text is a lossy, low-dimensional representation of spoken language and bears a similar relationship to speech as clothing does to anatomy. Both are artificial tools, constructed so as to have close correspondences to natural phenomena (letters to vowels and consonants, sleeves to arms and shoulders), but both are underspecified with respect to those natural phenomena.

Text was built slowly, over time, for silent reading in your head. Of course it’s possible to read text aloud, like when reading a speech or an announcement, but in these cases the text just serves as a mnemonic device of sorts. And the best speakers deviate drastically from what is written.

Since the advent of synthetic speech, text has been the most sensible basis for training and generation. It was chosen because it’s convenient and human-parsable. But at heart, extrapolating speech from text is as difficult as extrapolating anatomy from clothing. And traditional TTS systems show those shortcomings.

At Rime, we are uniquely aware of the jury-rigged relationship between text and speech and have designed and trained our models to both ease the friction in this relationship and to allow easy, intuitive user control of it.

In this post, I’ll go into some powerful ways of getting the most out of this relationship.

Training

When training a text to speech model, we pair audio files with text, and then coax the models to learn that this squiggle in the wave form corresponds to this word and vice versa. This means that the more the text input to our model resembles text in the training data, the better the output will be.

Oftentimes, normal written language conflates or ignores what spoken language distinguishes. For instance the sentences below can be spoken in multiple ways:

1. She’s mia.
   
   
2. His score was 170.
   
   

The first sentence could be about someone named ‘Mia’ and thus ‘mia’ would be pronounced like a single word. It could also be about someone ‘missing in action’ and thus mia would be pronounced like three letters in a row: m. i. a.

The number in the second sentence can be uttered in a number of ways: one seventy, one hundred seventy, one hundred and seventy, a hundred seventy, a hundred and seventy, one seven zero, one seven oh, etc.

For these and other reasons, we annotate our data in ways that help us disambiguate text and to encode things via text that are normally not encoded.

Of course, we don’t want users to have to learn all the arcane things we do to contort text to be more amenable to speech. And we do a lot of work behind the scenes to ‘clean up’ user text input for a seamless experience.

However, because we train our models on specific guidelines intended to capture things that text doesn’t normally capture, this allows users in principle to have a lot of control over the output by modifying the text. 

In the rest of the post I’ll discuss some simple ways of harnessing this power.

Write the mess in

LLMs are trained on countless billions of text strings, all intended for silent reading. Because of this, their output is profoundly ill-suited for deterministic reading aloud. 

As an experiment, have your favorite LLM respond to you in some casual spoken dialect. Because of what it was trained on, it will sound stilted, corny or insultingly stereotypical. This is one of the main reasons why TTS models often sound bad. They have tin-eared scriptwriters.

At Rime, we train on natural speech paired with nuanced data and this gives you more control.

People don’t naturally speak in clean sentences. They restart, repeat, trail off, and hesitate. A caller hears “I, I just think we should wait” and understands something about the speaker’s state that “I think we should wait” doesn’t carry.

So write the mess in. Instruct your LLM to repeat short, common words for a false start or add a hyphen where a word gets cut off. You can even add fillers like where a person genuinely pause to think like: um, uh, mmhm, uh-huh. These are all vocabulary and sprinkled into a conversation, they make your agent sound present. 

Our guidelines are the best place to start for arriving at a standardized text representation of these non-standard things. Here are some common ones to have your LLM use:

Non-standard words

‍

False starts

False starts are when people begin to utter a word, but stop before they finish the word. We want to indicate them with a hyphen affixed to the end of the word, roughly where the person cuts off:

Bu- but I don’t wanna!

Repeated words

In spoken language we often repeat the smallest, most common words. I, the, we, but, etc. It is very helpful to instruct your LLM to occasionally do this. Those not for longer or less common words.

I I I just don’t know.

Filler words

In English, filler words are not some mistake or lazy speaking, but are use rhetorically subconsciously to help the listener understand. Tell your LLM to sprinkle in some um’s and uh’s before long or infrequent words to sound more human and ease the understanding of your user.

Better yet, the use of a filler word between a repeated frequent short word like below “the um, the” works really well and matches real world speech subtly and effectively

We should modulate the um, the frequency of the filler words

Punctuate how it sounds

Punctuation was created in a vain attempt to encode some prosodic things we do in speech, but our repertoire of ? ! . , can only do so much if we stick to the rules of grammar that were forced upon us in school.

In our models, punctuation is a prosody instruction: commas are short rises, periods are falling pitch, ellipses are hesitation, question marks are rising pitch. Even mid-sentence when nothing is being asked.

Periods

These should be used more or less normally, to indicate the end of a sentence that is neither a question nor an exclamation. This indicates falling pitch.

Commas

We are taught innumerable and contradictory sets of rules regarding commas in school, but for our purposes, we use commas to indicate pauses with a corresponding short rise in pitch. 

Question marks

We want question marks to indicate a rising intonation at the end of an utterance. Usually this will found in traditional questions. But it can also be found in other contexts like surprisal (what?!) or so-called ‘up-talk’ (I actually think I’m pretty cool?)

Exclamation points

Traditional grammars will tell you that exclamation points belong at the end of imperative/command sentences like “tell me a story!” no matter what. But we only want to use exclamation points for a notable difference in the pronunciation of the sentence from that of a casual pronunciation.

Interrobangs

Sometimes people ask questions with an extra amount of enthusiasm or intensity. We try to capture this by using the combo !? at the end of the sentence. The relative order of the ? and ! doesn’t matter. Feel free to use as many as you’d like in any order as sort of a palette.

These are general rules, but it’s good to treat punctuation like a parameter: render two versions, listen, and keep the better one. 

Write for the voice you cast

Here’s the part almost nobody does: register is a per-voice decision.

A reserved voice wants ellipses and room to breathe. An energetic voice can carry repeated words and stacked exclamation marks. Give casual, chatty text to a formal voice and you get the vocal equivalent of a bad actor, technically saying the lines, obviously miscast.

This means your system prompt should be written for the specific voice you've cast, and updated if you recast. When we help teams tune agents, voice selection and text style are one decision. A banking agent and a food-ordering agent shouldn't share a prompt any more than they'd share a voice.

In your prompt, for a casual and un-intimitating experience, perhaps for a younger voice, it’s good to prompt your LLM to output more repeated words, fillers, and false starts.

For a more formal voice experience, tell it to do things more sparingly or not at all. Have it start sentences with ‘well,’ or ‘so,’. Even so, the default output is so unlike natural speech that using the casual markers occasionally still really helps.

Why we don’t use SSML

You may have noticed what's missing from all of this: No break tags, no emotion tags, no speed controls. 

When the writing already encodes delivery through punctuation, spelling, and word choice, and the model reads meaning, a tag layer is redundant at best. We kept the interface small because the text is the interface. Learn to write for the ear and you have more control than any tag set gives you, in a form that survives every stack and every orchestration layer between your LLM and our API.

Some people hear "no SSML" and hear fewer knobs. We'd put it this way: writers have been encoding pacing, pitch, and emphasis in text for a few centuries, and it works.

Start here

The full prompting guide, including a drop-in system prompt you can paste into your agent today, is here. 

‍