Diffusion Models for Speech

WaveNet synthesised speech so convincingly that listeners scored it above 4.0 MOS in 2016, but generating one second of audio required thousands of sequential autoregressive steps. Every improvement in quality came with a corresponding tax on latency. Diffusion models broke that coupling: DiffWave (ICLR 2021) reached MOS 4.44 on a vocoding task while generating audio in parallel, matching a strong WaveNet baseline at orders-of-magnitude higher throughput.

What diffusion actually does

A diffusion model defines two Markov chains. The forward chain gradually corrupts a clean data sample x_0 by adding Gaussian noise across T steps, arriving at x_T which is approximately standard normal. The reverse chain learns to undo that corruption step by step, recovering structure from noise.

For a fixed noise schedule beta_1, ..., beta_T the forward marginal has a closed form:

q(x_t | x_0) = N(x_t ; sqrt(alpha_bar_t) * x_0, (1 - alpha_bar_t) * I)

where alpha_bar_t = product of (1 - beta_s) for s = 1..t. This lets you sample any noisy version of the data directly without running the chain forward step by step.

The model is trained to predict either the added noise epsilon or the clean sample x_0 from (x_t, t, conditioning). At inference, you start from x_T ~ N(0, I) and run the learned reverse transitions, gradually refining the signal.

For speech there are two places to apply this process: the raw waveform at 16 kHz or 24 kHz, or a compressed intermediate such as a mel-spectrogram.

Waveform-domain models

WaveGrad (Chen et al., 2020) and DiffWave (Kong et al., 2020) both operate directly on 1-D waveforms conditioned on a mel-spectrogram. They differ slightly in architecture (WaveGrad uses a film-conditioned U-Net; DiffWave uses a dilated WaveNet backbone) but share the same core insight: the reverse diffusion chain can be computed in parallel across timesteps of the audio signal, unlike an autoregressive model which must generate each sample after the previous one.

The diffusion timestep T is a hyperparameter with real engineering consequences:

WaveGrad showed that six reverse steps often suffice for production-quality vocoding, provided the noise schedule is fine-tuned. This schedule mismatch (train long, infer short) is central to practical deployment.

Spectrogram-domain models and full TTS pipelines

Grad-TTS (Popov et al., 2021) applies score-based diffusion to mel-spectrograms rather than waveforms. It frames TTS as aligning text to a target mel-spectrogram where the prior is not pure Gaussian noise but a noise-corrupted version of a text-dependent mean computed by a learned duration/alignment model. This gives the reverse chain a much more informative starting point.

The forward process in Grad-TTS is defined as a stochastic differential equation (SDE):

dx = -1/2 * beta(t) * x dt + sqrt(beta(t)) dW

with W a Wiener process. The reverse-time SDE is:

dx = [-1/2 * beta(t) * x - beta(t) * score(x, t)] dt + sqrt(beta(t)) dW_reverse

The score function (gradient of the log probability density) is estimated by a neural network trained with denoising score matching. At inference, numerical SDE solvers (Euler-Maruyama or a predictor-corrector scheme) integrate the reverse SDE from noise to speech.

Because the prior is centred on an encoder output rather than zero, the model needs fewer reverse steps to "find" the right phoneme alignment and prosody. Grad-TTS trades a small amount of architectural complexity for a significant reduction in the number of function evaluations required.

Full pipelines typically combine a diffusion spectrogram model with a separately trained neural vocoder (HiFi-GAN, WaveGrad, or similar) to produce final audio. End-to-end waveform diffusion is possible but increases model size and the cost of each forward pass considerably.

Conditioning and control

Diffusion models condition cleanly through classifier-free guidance: train the model jointly with and without the conditioning signal, then at inference scale the conditional update by a guidance weight w:

epsilon_guided = (1 + w) * epsilon(x_t, t, c) - w * epsilon(x_t, t, null)

For speech this generalises naturally to multiple conditioning axes:

• Speaker identity via a d-vector or x-vector embedding enables voice cloning from a short reference clip.
• Prosody control via reference audio embeddings, style tokens, or predicted pitch/energy sequences.
• Prompt-based editing (e.g., change one word without re-synthesising the whole utterance) by running the forward process only partially and then reversing with a new conditioning signal.

The inpainting analogy is useful here: mask out the region to edit, add noise only to that region, and run the reverse chain conditioned on the surrounding unmasked speech. Several works (e.g., FluentSpeech, EdiTTS) formalise this.

When it falls down

Slow sampling. Even with accelerated schedules, a high-quality waveform diffusion model still requires 6 to 50 neural-network forward passes per audio segment. Compared to a single-pass GAN vocoder or a flow-based model, this is expensive on constrained hardware. Knowledge distillation (consistency models, progressive distillation) can collapse the chain to 1-4 steps but requires additional training effort.

Noise schedule sensitivity. The beta schedule controls the signal-to-noise trajectory. A schedule tuned for 1000 training steps produces poor audio when naively truncated to 6 inference steps. Practitioners typically run a separate optimisation pass (e.g., grid search or Bayesian optimisation) to find a short custom schedule, which adds deployment complexity.

Alignment failures with SDE solvers. In Grad-TTS-style models, the numerical SDE solver can accumulate errors over many steps, occasionally producing spectrogram frames that drift away from the correct phoneme alignment. Phoneme boundaries become smeared, particularly on long utterances or unusual prosodic patterns.

Out-of-distribution conditioning. Voice cloning from a very short (under 3 seconds) or acoustically unusual reference clip degrades speaker similarity, because the conditioning embedding cannot capture all relevant vocal characteristics. Fine-tuning on the target speaker resolves this but removes the zero-shot convenience.

Evaluation gap. MOS remains the standard but is expensive and noisy. Automated proxies (UTMOS, DNSMOS) correlate only moderately with human judgements, making it easy to over-optimise a proxy metric while actual perceptual quality regresses.

Further reading

• DiffWave: A Versatile Diffusion Model for Audio Synthesis - Kong et al., ICLR 2021; waveform diffusion matching WaveNet MOS.
• WaveGrad: Estimating Gradients for Waveform Generation - Chen et al.; score-based waveform synthesis in as few as six steps.
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech - Popov et al.; SDE-based spectrogram diffusion with text-aligned prior.
• Denoising Diffusion Probabilistic Models - Ho et al., NeurIPS 2020; foundational DDPM formulation underpinning all of the above.