Introduction to Text-to-Speech (TTS)

Text-to-speech technology converts written text into spoken words using algorithms that analyze text and output generated audio. One of the primary drivers of early TTS research was its use as an accessibility tool for those with communication impairments, one famous use of TTS software is late theoretical physicist Stephen Hawking's speech synthesizer.

Traditional TTS software was built on rule-based algorithms split into several stages, including parsing text, converting to phonemes (basic sound units), and determining inflection and rhythm patterns. Traditional TTS software, while efficient, often produces the recognizable unnatural speech patterns associated with robotic speech generation (think early Siri). Just as large language models have revolutionized text generation, recent advances in deep learning have revolutionized TTS by training models on a vast amount of speech data, yielding more natural, human-sounding speech patterns without the robotic cadence.

Commercial vs Open Source Options

While commercial (proprietary) TTS software can be quite expensive, ranging from $4-$20 per million characters, there are numerous open source options providing quality, cost-effective TTS capabilities, often with more customizability due to source code access, making open source options appeal to developers. Before we dive into the open source models, let's take a brief look at the best commercial options.

Commercial TTS Options

• ElevenLabs: Market leader for voice cloning and emotions/expression.
• Amazon Polly: Most scalable and cost-effective for commercial production, integration in AWS
• Google Cloud TTS: Most natural sounding, integration in Google Cloud
• Microsoft Azure AI Speech: Praised for customizability, integration in Azure
• IBM Watson TTS: Very reliable for enterprise purposes, integration with IBM

Open Source TTS Options

We will now explore the wide range of different open source text-to-speech options. While commercial products are often more polished and easy to use due to detailed documentation and customer support, many open source projects have large communities passionate about contributing, writing documentation, and supporting other members of the community.

Festival Speech Synthesis

Let's begin with one of the classics! Developed at the University of Edinburgh, Festival Speech Synthesis is one of the earliest TTS systems. Festival Speech Synthesis is not a deep learning system and its speech quality is known to be more robotic than modern deep-learning models, but it is a solid choice for resource-constrained projects or where natural-sounding speech is not a priority.

eSpeak NG

A community fork of the original 2007 eSpeak, eSpeak NG ("Next Generation") also predates modern deep-learning models and has a more robotic speaking style. However, it's extremely lightweight and fast, and is an ideal choice for accessibility and assistive tools, where resources are constrained and speed is a priority.

VITS

VITS is an important foundational model serving as a framework backbone for various other adaptations, including many multilingual models with specialized tokenizers. VITS is known for being natural and high-quality with solid voice-cloning, but requires a strong GPU for training.

Coqui XTTS (Mozilla)

Mozilla originally developed their own TTS technology, but when priorities shifted, many of the Mozilla researchers and engineers continued the work at a 2021 startup Coqui, which unfortunately has since shut down - but not before releasing XTTS v1 and XTTS v2.

XTTS can clone voices with 6 seconds of audio and supports over 15 languages with high quality output. However, XTTS is computationally-intensive and requires a good GPU for decent inference times. XTTS is available under Coqui's public model license, which means it can only be used for non-commercial purposes.

MeloTTS

MeloTTS, developed in part by MyShell.ai, is a lightweight model which is optimized for real-time CPU-based inference. It offers many English accents and a few languages, and is particularly notable for handling mixed Chinese/English. MeloTTS is considered one of the easiest high-quality TTS models to use, which combined with its optimization for CPU, makes it a very solid choice for a large range of projects.

ChatTTS

ChatTTS is an open source project tailored for dialogue tasks, such as AI assistants, and allows users to have "fine-grained control" over prosodic features such as laughter and pauses.

However, this model is also only available for non-commercial use, and the repository owner explains in the GitHub README disclaimer that they intentionally trained it on data with high frequency background noise and compress the audio quality as much as possible to make it more difficult for ChatTTS to be used for unethical or criminal purposes.

Tacotron2

Originally researched and developed by Google Brain, Tacotron2 was built on Google's Tacotron and Google subsidiary DeepMind's WaveNet. While the original 2016 WaveNet model is no longer in use, WaveNet was one of the earliest neural networks for TTS and is still used today for the non-Gemini Google Assistant voice.

Tacotron2 produces high quality, natural-sounding speech, but it is very resource-intensive and a solid option primarily if you have access to a strong GPU.

MaryTTS

Lightweight and well-documented, MaryTTS is another non-deep-learning option well-suited to projects where resources of limited. MaryTTS is great for customizability due to its modular structure, though it is written in Java, which can either be a selling point or a dealbreaker depending on the developer. Its speech quality is not as high as more modern options, but it's a great option for multilingual use.

Fish Speech

Known for its phoneme-free language-agnostic processing, Fish Speech is ideal for projects needing very precise and customizable speech. It supports cloning from 10-30 second samples, however it is a very resource-intensive model and requires significant GPU access.

Open Source TTS Superlatives

We just explored a lot of solid open source text-to-speech options, so how should you pick which will be right for your project? Here are the best models for specific cases:

Best at voice-cloning: XTTS-v2, Fish Speech

Most natural/human-sounding: XTTS-v2, ChatTTS, Tacotron2

Best for efficient CPU use: Festival Speech Synthesis, eSpeak NG

Best deep-learning model for CPU use: MeloTTS

Best if you have GPU access: Tacotron2, VITS

Best for real-time speech: eSpeak NG

Most languages available: eSpeak NG (over 100 languages)

Best for multilingual use: MaryTTS, Fish Speech, VITS adaptations

Best quality/ease of use ratio: MeloTTS - If you're looking to dip your toes into TTS for the first time, MeloTTS is an appealing place to start.

Takeaways

To summarize, deep-learning models are known for quality but are typically more resource-intensive, while pre-deep-learning systems are extremely fast and lightweight but can sound more robotic. Consider the priorities for your project to determine which model is best for you.

Another important consideration is the ease of use. Some models (such as MeloTTS) have user-friendly APIs with nice Docker setups. However, some models are aimed more at researchers and require proper deep-learning environments with a much higher barrier to entry. Which side of the spectrum you prefer will depend on your project's needs. Additionally, licensing terms should be carefully reviewed to ensure compliance, especially when considering models for commercial use.

The landscape of TTS technology has expanded exponentially in the past few years, and with more models, especially high-quality open source models, available than ever before, choosing the right model can be a challenging task. However, that just means there are many more solid options than ever before!