Speech Synthesis
Acoustic models and vocoders, Tacotron, FastSpeech, HiFi-GAN, neural codecs and voice cloning.
20concepts
139flashcards
155minutes of reading
- 01 Attention Failures in TTS Attention-based TTS systems fail in predictable ways - word skipping, repetition, and unstable alignment - and understanding the mechanics behind each failure mode is essential for building reliable speech synthesis pipelines.
- 02 Bark and Fully Generative Audio Bark is a transformer-based model that generates speech, music, and nonverbal audio from text by autoregressively predicting discrete audio codec tokens, without any phoneme pipeline or continuous acoustic model.
- 03 Duration Modelling and Alignment Duration modelling assigns how many audio frames each phoneme occupies; alignment is the mechanism that learns or infers that mapping from text-audio pairs without manual annotation.
- 04 Evaluating TTS with MOS Mean Opinion Score is the field's primary yardstick for TTS naturalness, but its reliability hinges on listener pool design, anchoring, and context choices that most papers under-report.
- 05 FastSpeech and Non-Autoregressive TTS FastSpeech eliminates the sequential mel-frame dependency of autoregressive TTS by predicting all frames in parallel, trading model simplicity for a duration predictor and a length regulator.
- 06 Griffin-Lim and Classical Vocoders Griffin-Lim is an iterative phase-recovery algorithm that converts a magnitude spectrogram back into a waveform without any learned parameters, and understanding it clarifies exactly what neural vocoders had to replace and why.
- 07 HiFi-GAN and Neural Vocoders Neural vocoders convert acoustic feature representations into raw audio waveforms, and HiFi-GAN achieves near-human quality at 167x real-time speed using a multi-scale, multi-period GAN architecture.
- 08 Neural Audio Codecs Neural audio codecs compress waveforms into discrete token sequences using learned vector quantisation, enabling language models to generate speech token-by-token.
- 09 Prosody and Style Control How TTS systems encode and manipulate pitch, duration, energy, and speaking style so that synthesised speech sounds intended rather than merely intelligible.
- 10 Residual Vector Quantisation Residual vector quantisation stacks multiple codebooks to approximate a continuous audio embedding with increasingly fine-grained corrections, making it the compression backbone of modern neural audio codecs.
- 11 Streaming TTS and Latency Streaming TTS pipelines generate and deliver audio incrementally to cut time-to-first-audio from several seconds to under 300 ms, but doing so imposes hard trade-offs on chunk size, model architecture, and prosodic coherence.
- 12 Tacotron 2 Tacotron 2 is a two-stage neural TTS pipeline that converts text to mel spectrograms with a sequence-to-sequence model, then synthesises raw audio with a conditioned WaveNet vocoder, achieving near-human MOS scores.
- 13 Text Normalisation and Phonemisation Text normalisation converts raw written text into a speakable form, and phonemisation maps those words to phoneme sequences; together they determine what a TTS system says before any audio is generated.
- 14 The Acoustic Model and Vocoder Split Modern neural TTS splits the problem into two specialised sub-networks - an acoustic model that maps text to a compact spectral representation, and a vocoder that reconstructs a full audio waveform from that representation.
- 15 The Mel-Spectrogram Interface The mel spectrogram is the agreed-upon intermediate representation that decouples acoustic modelling from waveform generation in modern TTS pipelines.
- 16 WaveNet WaveNet is a fully autoregressive convolutional neural network that models raw audio waveforms one sample at a time, achieving near-human speech quality at the cost of extremely slow sequential generation.