Text to Speech vs Speech to Speech: Which Sounds More Like the Voice?

Most pages on this question define the two terms and stop. We already had numbers: we had to pick an engine for every voice in our AI Voice Generator, so we scored both. This post is what those scores, plus one extra test on the live site, showed.

What is the difference between text to speech and speech to speech?

Text to speech reads words you type and decides the timing itself, while speech to speech takes a recording and changes only the voice. With speech to speech, the words, pauses, emphasis and pronunciation all come from the recording.

One catch none of the explainers mention: typed text doesn't always mean text to speech. Many famous-voice tools, ours included, have a voice model that only converts speech. For those voices, a stock voice reads your text first and the voice model converts that reading. The model is RVC (Retrieval-based Voice Conversion), an open speech-to-speech model trained on one voice. In our AI Voice Generator, 35 of the 82 voices read typed lines with a text-to-speech model, and the other 47 use the read-then-convert path.

Which one sounds more like the famous voice?

Text to speech, in most voices. Across 51 famous and character voices, the best text-to-speech model of each voice scored closer to the real voice than our speech-to-speech version in 35. The average likeness was 0.60 for text to speech against 0.50 for speech to speech, on a 0 to 1 scale from a speaker-recognition model.

Speech to speech won the other two measures:

• Naturalness: 3.67 for speech to speech against 3.22 for text to speech, out of 5 (UTMOS, a model that predicts a listening panel's score).
• Word errors: 2.4% of words misheard by Whisper on speech to speech against 3.9% on text to speech.
• Big misses: the 16 voices where speech to speech won include one where the text-to-speech model scored 0.15, clearly someone else.

Our read of why: a text-to-speech model of a voice learns how that person talks, including rhythm and pitch movement. A speech-to-speech model swaps the tone of the voice but keeps the reader's rhythm, and when the reader is a stock voice, the character's way of speaking is missing. That's why we moved typed lines to text to speech in only 29 of our 82 voices: the ones where it matched or beat the old path on likeness, naturalness and words together.

Which one keeps your timing and emotion?

Speech to speech keeps your timing; text to speech replaces it. Our 26-word test line ("Wait. Did you hear that? Something is moving in the vents...") was a 9.1-second recording whose speech ended at 8.4 s. Converted with Keep my delivery, the speech ended at 8.5 s. Its three longer pauses started within 0.07 s of the recording's, and only the shortest, 0.16 s after "Okay,", got too short for us to measure.

Typed in the same voice, the text-to-speech model took 12.9 s, about 4.5 seconds longer, and paused in its own places, including a dramatic stop after "Wait." That suits an announcer. It won't fit under a video you already cut. Whisper got all 26 words right in all three versions, so neither path lost words on a clean line.

The typed line in the Super Smash Bros. Announcer voice paused in the same places as our recording, to within 0.06 s. Our recording was read by the same stock voice our read-then-convert path uses, so the match shows that path is speech to speech under the hood.

Why does my typed line sound different from my recording in the same voice?

In voices with a text-to-speech model, typing and recording use two different engines, and they make two different versions of the character. In our test, the typed Mortal Kombat Announcer line scored 0.91 against the samples on the voice's tile, about as close as those samples are to each other (0.90). The recorded version scored 0.75 against the same samples but 0.94 against the voice's speech-to-speech preview. The two engines' samples score 0.77 against each other. For scale, every version scored 0.54 to 0.64 against our test voice, which is a different speaker.

So if you want the voice you heard on the tile, type the line. If you recorded and want that tile voice with your words, pick Most realistic voice instead of Keep my delivery: it transcribes your recording and speaks the words with the text-to-speech model. It's paid only, and it gives up your timing.

How long does each one take?

Typed lines came back fastest. On facecam.ai, the typed text-to-speech line took 7.6 s from click to player. The recording took 52 s, because the speech-to-speech model runs on a GPU (graphics card) server that had to start up; the tool warns that the first one can take about a minute. The next job on that server, a typed line in a read-then-convert voice, took 8.4 s.

How do you get the best result from each?

1. 1

   Pick by the job

   Dubbing a clip, matching lip movement or acting out a scene: record it. A one-liner, a meme or a soundboard clip where likeness counts most: type it.

2. 2

   Typed: write the pauses in

   Full stops and commas are where the model breathes. Our announcer paused after "Wait." and "Okay," because the punctuation told it to. Free lines are up to 200 characters; plans allow 500.

3. 3

   Recorded: perform it

   Record in a quiet room, at the character's pace, with the emphasis you want, because speech to speech copies all of it. Free recordings are up to 15 seconds; plans allow 60, at 5 credits a recording.

4. 4

   Download and check the end

   Free downloads end with a short spoken "Made with Facecam.ai" (2.28 seconds). Files made on a plan don't have it.

For longer recordings, the AI Voice Changer does speech to speech on whole files. For how AI voices compare with pitch effects, see do AI voice changers sound real.

What didn't we test?

• A real performance. Our recording was read by a text-to-speech voice. A good human take, with real emotion, is where speech to speech should do best, and we didn't measure that.
• Listeners. Likeness and naturalness come from models, not people. They track listening tests well, but they aren't one.
• A level field. The text-to-speech side is the best of up to three models per voice, against one speech-to-speech model, which favours text to speech.
• More than one run. The timing test is one line, once per path. Text-to-speech output can change from one try to the next.
• Other tools. We only tested Facecam's engines.