Speech Recognition
Spectrograms, CTC, RNN-T, Conformer, Whisper, streaming, diarisation and self-supervised audio.
20concepts
140flashcards
156minutes of reading
- 01 Audio Features and Spectrograms Raw audio waveforms are rarely fed directly to speech models; this concept explains how and why they are first converted into spectrogram-based representations that compress perceptual information into a learnable 2-D grid.
- 02 The ASR Problem and Pipeline Automatic speech recognition converts a raw audio waveform into a word sequence by solving an alignment problem that classical NLP never had to face.
- 03 Beam Search Decoding in ASR Beam search decoding navigates the exponentially large label sequence space in ASR by maintaining a fixed-width frontier of the most probable partial hypotheses at each step, making it the default inference strategy for CTC, RNN-T, and attention-based models.
- 04 CTC: Connectionist Temporal Classification CTC is a training objective that lets a neural network learn to align variable-length audio to text without any hand-labelled frame-level annotations.
- 05 Causal and Chunked Attention for Streaming Streaming ASR requires attention mechanisms that never look at future audio; causal masking and chunked attention are the two principal techniques, each trading latency against accuracy in different ways.
- 06 Endpointing and Voice Activity Detection Endpointing and voice activity detection are the mechanisms that decide when a user has finished speaking, directly controlling the latency and correctness of every streaming ASR system.
- 07 Listen, Attend and Spell LAS is a purely sequence-to-sequence ASR model that replaces HMMs, CTC, and explicit pronunciation dictionaries with a pyramidal RNN encoder and an attention-based character decoder trained end-to-end.
- 08 Robustness to Noise and Accents How modern ASR systems are trained and adapted to handle environmental noise, channel distortions, and speaker accent variability without collapsing to near-zero accuracy.
- 09 Speaker Diarisation Speaker diarisation segments an audio recording into speaker-homogeneous regions and assigns each region a speaker identity, answering the question "who spoke when" without necessarily transcribing what was said.
- 10 Streaming vs Offline ASR Streaming ASR emits partial transcripts incrementally as audio arrives, trading access to future context for low latency, while offline ASR processes the full utterance and consistently achieves lower word error rates.
- 11 The CTC Blank Token and Alignment The CTC blank token is a special output symbol that lets a neural network emit one label per time-step without needing a hand-crafted alignment between audio frames and characters.
- 12 The Conformer Architecture The Conformer interleaves convolution and multi-head self-attention inside each encoder block to capture both fine-grained local acoustic patterns and long-range sequence dependencies, achieving state-of-the-art ASR accuracy on LibriSpeech.
- 13 Whisper's Multitask Decoder Whisper conditions a single sequence-to-sequence decoder on a prefix of special tokens that specify language, task, and timestamp behaviour, allowing one model to handle transcription, translation, and language identification without any task-specific heads.
- 14 Whisper: Weakly-Supervised ASR Whisper trains a sequence-to-sequence Transformer on 680,000 hours of weakly-supervised internet audio to achieve robust multilingual speech recognition without task-specific fine-tuning.
- 15 Why Sequence Length Is Hard in Audio Audio produces roughly 100 frames per second of speech, making sequence lengths 10-50x longer than equivalent text, and this mismatch drives almost every architectural and training decision in modern ASR.
- 16 Word Error Rate and ASR Evaluation Word Error Rate measures ASR accuracy as the minimum edit distance between a hypothesis and a reference transcript, normalised by reference length, but its apparent simplicity hides a nest of normalisation choices that make numbers across papers routinely incomparable.
- 17 wav2vec 2.0 and Self-Supervised Audio wav2vec 2.0 learns speech representations from raw audio without transcripts by masking latent features and solving a contrastive task over learned discrete units, then fine-tunes on as little as ten minutes of labelled speech to reach competitive word error rates.
- 18 HuBERT and Discrete Audio Units HuBERT pre-trains a speech encoder by predicting offline k-means cluster labels for masked audio frames, producing discrete unit sequences that rival phoneme transcriptions without any text supervision.
- 19 Language-Model Fusion Language-model fusion techniques inject text-only knowledge into end-to-end ASR models at inference time or training time, and the correct method depends on how much implicit language bias the acoustic model has already absorbed.
- 20 The RNN-Transducer The RNN-Transducer is a fully neural, streaming-capable sequence transduction model that replaces CTC's conditional independence assumption with a learned label-context network, enabling accurate on-device speech recognition.