Teaching our Text-to-Speech to sound natural, every time

Text-to-speech (TTS) models have become remarkably good. They don't just read and pronounce words correctly. They can laugh, whisper, and sound warm or worried, often to the point where synthetic speech is hard to tell apart from a real person.

Generate the same sentence many times, though, and cracks start to show. Most takes sound great, but the same voice may sound tired in one and upbeat in the next, even though only one of those fits the sentence. One version might take a breath in the middle of a phrase, another could stress the wrong word. Each take sounds human on its own. The problem is that the model doesn't consistently choose the delivery that fits the situation.

Changing the model's settings can help, but often gives limited improvement. Many TTS models let you change the temperature, just like an LLM. It controls how much the model varies when it chooses the next piece of audio. With a low temperature, the speech is safer but also often flatter. With a high temperature, there is more variety but also more mistakes. In both cases you only change how much the model varies, but you can't change which delivery it prefers. And in order to make good delivery the normal behavior, we need to change the model itself.

Illustrative. Lowering the temperature narrows the spread of deliveries and raising it widens the spread, but the center stays where it is. RL moves the whole distribution toward the deliveries people like.

In this post we explain why supervised training does not fix this, and how reinforcement learning (RL) can help. You don't need a machine learning background, and we will explain each term when it first appears.

Why supervised training is not enough

Almost all TTS models today are built with supervised training. This means the model learns by copying examples.

It usually happens in two stages: first pretraining on a very large amount of recorded speech, and then fine-tuning (often called SFT, supervised fine-tuning) on a smaller set of high-quality recordings. In both stages the idea is the same → the model gets a piece of text and a recording of a person saying it, and we train the model by adjusting its parameters so that it learns the mapping between the text and the audio. After thousands of hours of recordings, the model can take new text and produce audio that matches what it has heard before.

This works very well when there is basically one correct answer. But that’s not the case for speech, because one sentence can be said in many good ways:

The same sentence spoken three ways. Calm: a short pause after "Honestly?" and a breath before "and then decide." Hesitant: a long pause and a breath early, and another gap before the last phrase. Brisk: almost no pauses.

All three are fine. A calm speaker might pause briefly after "Honestly?" and take a breath before the last phrase. A hesitant one lingers, breathes early, and leaves a gap before "and then decide." And a brisk one barely stops.

When a model is trained to imitate all of these at once, it doesn't learn to pick one. It tends to average:

Illustrative. Speaker A pauses after "Honestly?". Speaker B pauses after "Monday,". A model imitating both ends up with half a pause in each spot, which neither speaker did.

If half your recordings pause early and half pause late, the safest guess is a bit of pause in both places. (A real model doesn't literally average audio, but the pull toward in-between deliveries is real. In practice you get both: some takes land on one of the good deliveries, others somewhere in between.) The problem is that a model trained to be safe across ten good performances often ends up with none of them: something in between, which might sound like mush.

There's a second problem. Learning by imitation only measures how close the output is to the recording. It has no way to say that one natural sounding take is better than another natural sounding take. And to get past that, we need a different kind of feedback.

Reinforcement Learning: learning from feedback, not examples

Reinforcement learning (RL) works the other way around. Instead of giving the model a recording to copy, we let it say a sentence several times, score each take (the reward), and make the high-scoring takes more likely and the low-scoring ones less likely.

This fixes the averaging problem. A take that pauses after "Honestly?" scores well, a take that pauses after "Monday," scores well too, and both can stay. But the half pause in both places is likely to sound odd, score low, and get pushed out. This is the key difference: in supervised training the model is only shown good recordings and never hears about its own mistakes. In RL it gets feedback on its own takes, including the bad ones. The score can also capture what imitation can't: when two takes both sound fine, which one fits the sentence better.

This is the same recipe used for LLMs today. After supervised training they go through RL, where the score either comes from an automatic check, like a math answer or code passing tests [1], or from a model trained to predict which answer people prefer, which is RLHF [2]. Our reward uses both: a naturalness score from a judge model, because there's no rule for natural delivery, and a word check from speech-to-text, which can be measured directly.

The same recipe in both cases. An LLM produces many answers and learns from which ones people (or a reward model) prefer. Our voice model produces many takes and learns from which ones the judge and the word check prefer.

The loop we built

The training loop: take a sentence, have the voice model produce several takes, score each one for naturalness and for WER, and nudge the model toward the high scorers. Repeat with a new sentence.

We start with a sentence, the kind of sentence that our voices actually say in real conversations. We then ask the voice model to say it several times, and because the model has some natural variety in how it speaks, the takes come out slightly different: some breathe in a sensible spot, some rush, some put a pause somewhere odd.

A separate model, which we call the judge, listens to every take alongside the text and scores how natural it sounds in context. The text matters, because a long pause is perfect before delivering bad news and strange in the middle of reading out a phone number. Each take is also checked to make sure the words themselves are still clear and correct.

Then we adjust the voice model slightly, so that the takes that scored well become a little more likely and the ones that scored badly become a little less likely. Then we move on to the next sentence, and repeat this many thousands of times.

Notice that every take came from the model itself. So RL doesn't really teach the model new skills, at least not here. The model can already breathe in the right place or pick the right tone, it just doesn't do it every time. What RL changes is how often it does the good version. And we deliberately keep it close to how it originally spoke (more on why below). Think of a vocal coach: the singer already has the voice, and the coach helps them use it well more of the time.

The reward: naturalness + word error rate

Each take, together with its text, is scored by the judge for naturalness and checked by a speech-to-text model for word errors. The two are combined into one reward.

Every take gets one number, the reward, and it's built from two parts:

• Naturalness, from the judge. How natural does this delivery sound for this text?
• WER (word error rate), from a speech-to-text model. It writes down what it hears, and we compare that with the original text. WER is the share of words that came out wrong, missing or extra.

Why do we need both? Because with naturalness alone, the model could learn to sound relaxed and human while mumbling half the words. And with WER alone, it would probably learn to read very clearly but flatly and slowly, because clean, slow speech tends to get lower WER. Together they push toward speech that sounds natural and says the right thing.

The judge

Everything depends on the judge agreeing with real listeners, so it's worth explaining what it is.

Our judge is a multimodal LLM, an LLM that can listen to audio as well as read text, and that writes text as output. We give it the audio and the text it was supposed to say, and ask it to rate how natural the delivery is in that context. If you want to try this yourself, there are good open-source models to start from, for example Qwen3-Omni [3], Xiaomi's MiMo-Audio [4], or StepFun's Step-Audio 2 [5].

One thing to keep in mind: these are all general models. They have a good sense of what natural speech sounds like in general, but they don't know what "good" means for your product (a calm support agent and an excited sales voice need very different things). If you want a judge that really matches your use case, you'll need to fine-tune it on your own data, for example pairs of takes where people picked the better one.

Whatever judge you use, measure how often it agrees with people.

What the judge actually hears: real examples

Here are two real examples from training. Same voice, same text, several takes, and the judge's score for each one. The waveforms are drawn from the real audio on the same time scale, so you can literally see the pauses.

The first one is a spelling confirmation, something our voices do all the time on calls:

Real takes from training. The best take pauses after "Briarwood Court" and then groups the letters like a person would (B-R, I-A-R, W-O-O-D). The lower ones get slower and more mechanical, and the worst one takes more than twice as long to say the same thing.

Listen for yourself:

The second one is a long, two-sentence explanation. Here the problem is the opposite: not too slow, but no room to breathe.

Real takes from training. The two top takes pause at the sentence break and before "because". The two bottom ones rush through more than 40 words almost without stopping, and you can see it: the waveform barely has any gaps.

Listen for yourself:

What we like about these examples is that the judge's reasoning matches what a person would say after listening. Nobody told it "pause between sentences" or "group the letters when spelling". It just knows what natural speech sounds like, and that is exactly the feedback the model learns from.

Using GRPO (Group Relative Policy Optimization)

We now have a score for every take, but a score alone doesn't tell the model what to do. Is 70 a good take? If the model usually scores around 50 on this sentence, yes: it's better than usual and should become more likely. If it usually scores around 90, the same 70 is one of its worse takes and should become less likely. So the model needs a reference point → which is the score that normally gets assigned to this sentence? The difference between a take's score and that reference is called the advantage. Better than the reference score means "do this more", worse than the reference score means "do this less". Methods like the two below differ mainly in how they set that reference.

The classic method is PPO [6], the one used in the RLHF work behind InstructGPT [2]. Like any RL method, PPO doesn't produce the scores itself. They come from outside: from a reward model in RLHF, from tests for code, etc. And for the reference score, PPO relies on a critic (or value model) model: a second network, trained alongside the main one, that predicts what score a take is likely to get. That prediction is the reference. The critic is often about as big as the model being trained, so it's expensive. And when its guesses are off, the updates are off too.

GRPO [7] gets its scores the same way, but removes the critic. Since we already generate several takes of each sentence, it uses the group itself as the reference: the reference is simply the group's average, and each take's advantage is how far it lands above or below that average, divided by how spread out the group's scores are. No second network, and no guessing.

Illustrative scores for eight takes of one sentence, shown relative to the group average of 6.6 (out of 10 here, to keep it simple). Takes above the average are made more likely; takes below it are made less likely.

For us, grading within the group has a second benefit. How hard a sentence is changes what a score means: a long sentence full of names and numbers might top out at 6, while "Sure, no problem!" easily gets a 9. With raw scores, the model would mostly learn "easy sentences are good", which doesn't help it say anything better. Comparing takes of the same sentence cancels out the difficulty, and what's left is the delivery itself.

This also explains which behaviors RL can improve. If all takes of a sentence get the same score, there's nothing to compare, and the reward teaches the model nothing from that sentence. RL learns the most from things the model gets right sometimes, which is exactly the kind of problem we described at the start.

For the technically curious: the GRPO objective

For each sentence q, the model πθ produces a group of G takes o1, ..., oG, and each take gets a reward ri. The advantage of a take is its reward compared with the rest of its group:

Every audio token in take i gets the same advantage Âi. The model is then updated to maximize the objective from [7]:

where ρi,t = πθ(oi,t | q, oi,<t) / πθold(oi,t | q, oi,<t) says how much more (or less) likely the current model makes token t of take i, compared with the model that generated it:

• Âi is the group-relative advantage described above. The group mean takes the place of PPO's critic.
• The clip comes from PPO [6]: it stops any single update from changing a token's probability too much, so training moves in small steps.
• The KL term is the leash from the guardrails section below: a penalty for drifting away from the original model πref.

With the spelling example from above, using only the judge scores (91.2, 76.8, 68.8, 56.2), the group mean is 73.2 and the standard deviation is about 12.7. The advantages are roughly +1.41, +0.28, −0.35 and −1.34: the best take gets the biggest push up, the worst gets the biggest push down, and the 76.8 take, just above average, only gets a small push.

What changes after RL

We didn't train the model for one specific thing. The reward only asks two questions: does this take sound natural for this text, and are the words still right? But "natural" covers a lot, so the model can improve on several things at the same time:

• Breathing. Breaths move to natural breaks in the sentence, like after a comma or at the end of a phrase, instead of the middle of one. And long paragraphs get a more human rhythm: not a breath every few words, but not a full minute without air either.
• Pauses. Short pauses where a person would think or change topic, and fewer odd gaps in places like phone numbers or names.
• Energy and tone. Less switching between a tired and a happy voice from one take to the next, and a tone that better matches what the sentence is saying.
• Pacing. Slowing down a bit for important details, instead of reading everything at the same speed.

Illustrative. Before and after for the four behaviors above: a breath moves from the middle of a phrase to the comma, the gaps inside a phone number become short and even, the energy stops swinging from take to take, and the voice slows down for the important detail.

We never told the model "breathe after a comma" or "slow down for numbers". It simply found that takes like that scored better, so it started doing them more often.

What goes wrong

RL has one famous failure mode, and anyone who uses it runs into it sooner or later. The model optimizes the score, not what you meant by the score. If there's a shortcut to a high number, it will find it. Researchers call this reward hacking.

In speech it can look like this. If the judge likes breathing, the model can learn to breathe constantly. Exaggerated ups and downs in pitch can sound "expressive" to a judge and like a bad audiobook to a person. The model can trade clarity for style, swallowing the ends of words because it sounds more relaxed. And any judge has its own small preferences, perhaps for a particular pace or pitch, which the model will drift toward until the voice starts losing its own character.

You can usually see it coming in the gap between the judge and real listeners:

Illustrative. Early in training, the judge's score and human preference rise together. Later the judge's score keeps climbing while human preference falls, a sign the model has found a loophole.

Typically the two rise together at first. Left unchecked, there comes a point where the judge's score keeps going up while listeners start liking the output less. From then on, a higher score no longer means a better voice.

We use three guardrails:

1. A leash. During training the model pays a penalty for drifting too far from how it originally spoke, so it can refine its habits but not reinvent itself to please the judge. We measure this with something called KL divergence. For each piece of audio, the model has a set of probabilities for what to say next. KL divergence is a single number that says how different the new model's probabilities are from the original model's. The bigger that number, the bigger the penalty during training. So the model can shift its habits a little, but every big shift costs it.
2. WER is always part of the reward. Naturalness is never scored alone, so the model can't earn a better naturalness score by mumbling.
3. Regular checkpoints with human evaluation. We save the model regularly during training, and real people listen to them in blind tests. The checkpoint we keep is the one people prefer, not the one with the highest judge score.

The three guardrails. The leash keeps the trained model close to the original voice. WER stays in the reward so mumbling never pays. And the checkpoint we ship is the one listeners pick, not the one with the highest judge score.

How we check it actually worked

Most of our checks don't involve the judge at all.

The main one is blind side-by-side listening: people hear the same text from the original model and the trained one, in random order, and say which they prefer. We keep the question simple: which one sounds more natural?

Blind side-by-side test: same text, same voice, random order.

We also ran an A/B test in production, following the approach we describe in this post. Live calls were randomly split between the original voice and the RL-trained one. With the new voice, fewer people hung up in the first few seconds, caller sentiment was more positive, and more calls ended with the task completed. These are early results from a single experiment, so we don't over-read them, but they point the same way as our listening tests.

What we took away from it

Ask a voice actor why they paused where they did and the honest answer is often "it felt right." That has long made delivery one of the hardest parts of speech synthesis to improve, because "it felt right" isn't something you can write rules for.

It turns out you don't have to. We never wrote a single rule about breathing, pauses or tone. We only built a reward that recognizes natural delivery and keeps the words correct, let the voice model try many variations, and let it learn from what worked. Breathing, pauses, energy and pacing can all improve from that same simple loop. And because the loop doesn't care what "natural" means exactly, we can keep making the judge better at hearing what people like, and the voice will follow.

References

[1] DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", 2025. arXiv:2501.12948

[2] L. Ouyang et al., "Training language models to follow instructions with human feedback", 2022. arXiv:2203.02155

[3] J. Xu et al., "Qwen3-Omni Technical Report", 2025. arXiv:2509.17765

[4] Xiaomi LLM-Core Team, "MiMo-Audio: Audio Language Models are Few-Shot Learners", 2025. arXiv:2512.23808

[5] StepFun Audio Team, "Step-Audio 2 Technical Report", 2025. arXiv:2507.16632

[6] J. Schulman et al., "Proximal Policy Optimization Algorithms", 2017. arXiv:1707.06347

[7] Z. Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", 2024. Introduces GRPO. arXiv:2402.03300