The Alignment Problem in Speech Recognition: CTC, RNN-T, and the Cost of Not Waiting

A ten-second utterance sampled at 16 kHz, framed every 10 ms, is 1,000 acoustic frames. Its transcript, "the quick brown fox jumps over the lazy dog", is 43 characters. Training a neural network on that pair requires a target for each frame, and nobody supplied one. No annotator marked the millisecond where the b in brown begins, and no annotator ever will, because transcription costs a few times real time and frame-level phonetic labelling costs hundreds.

For thirty years the field's answer was to manufacture the missing labels: run a forced aligner, assign every frame a state, and train a classifier on those pseudo-labels. It worked and it was ugly, because the alignment came from a model that itself needed an alignment to train, and the whole edifice — pronunciation lexicon, context-dependent phone states, decision-tree tying — existed to prop up that bootstrap. In 2006 a different answer arrived: stop labelling frames, and instead sum over every alignment that could have produced the transcript. That single change is the ancestor of every end-to-end speech system running today, and the differences between those systems are almost entirely differences in how they handle the sum.

Why this matters: The alignment mechanism is not an implementation detail buried under the model. It determines whether the system can emit words while the user is still speaking, whether the transcript can be conditioned on what was already said, and how much memory a training step costs. Choose CTC and you get a fast streaming model that cannot spell; choose attention and you get a fluent model that must hear the whole utterance first. Every production ASR decision traces back to this fork.

TL;DR

• Alignment is the actual problem. Given \(T\) frames and \(U\) output labels with \(T \gg U\), the number of monotonic alignments is combinatorial; CTC and RNN-T both work by marginalising over all of them with dynamic programming rather than committing to one.
• The blank token is the whole trick. Adding one symbol meaning "emit nothing here" turns an unaligned pair into a set of equal-length paths that a forward algorithm can sum in \(O(TU)\) instead of exponential time.
• CTC's speed is bought with an independence assumption. Frames are conditionally independent given the audio, so the model cannot use "what I just wrote" to decide what to write next. This is why CTC systems lean on external language models and why they misspell in characteristic ways.
• RNN-T fixes that and pays in memory. Adding a label-side prediction network makes outputs autoregressive, at the cost of a \(B \times T \times U \times V\) joint tensor: for a realistic batch that is roughly 3.2 GB in fp32 for a single loss computation, which is why pruned and memory-efficient transducer losses exist at all.
• Attention encoder-decoders are the most accurate and the least streamable. Global attention over the encoder means the model can look at the end of the utterance before writing the beginning; Whisper's 680,000 hours of weak supervision buy remarkable zero-shot robustness and inherit exactly this constraint.
• Streaming costs accuracy in a measurable, tunable way. Every architecture has a knob that trades right context for latency, and the sensible question is never "streaming or not" but "how many milliseconds of future audio is a point of word error rate worth".
• Self-supervision changed the labelled-data calculus, not the alignment problem. wav2vec 2.0 reached 4.8/8.2 WER on LibriSpeech using ten minutes of labelled audio; the alignment machinery underneath was unchanged.

At a Glance

flowchart LR
  A[Waveform 16 kHz] --> B[Log-mel frames every 10 ms]
  B --> C["Encoder: Conformer or Transformer"]
  C --> D{"How are T frames mapped to U labels?"}
  D -->|"Blank symbol, frames independent"| E[CTC head]
  D -->|"Blank plus label-side LM"| F[Transducer joint network]
  D -->|"Attention over all frames"| G[Autoregressive decoder]
  E --> H[Transcript]
  F --> H
  G --> H
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class A,B blue
  class C,D purple
  class E,F,G amber
  class H teal

The diagram hides one asymmetry that the rest of this article is about: the two upper branches can emit as audio arrives, and the lower one, in its standard form, cannot.

[IMAGE: Side-by-side spectrogram and transcript for a three-second utterance, with vertical lines showing one plausible frame-to-character alignment and, beneath it, three other plausible alignments of the same pair. Caption: "The supervision says these two sequences correspond. It does not say how, and every one of these alignments is consistent with the label."]

Before End-to-End: Thirty Years of Manufacturing Labels

The classical pipeline treated recognition as decoding a hidden state sequence. A hidden Markov model represented each phone as a small chain of states, a Gaussian mixture modelled the acoustics of each state, a pronunciation dictionary mapped words to phones, and an \(n\)-gram language model scored word sequences. Training required frame-level state labels, which were produced by the Baum-Welch algorithm — itself an expectation-maximisation procedure that marginalises over alignments, so the field already knew the trick; it simply applied it to a generative model with strong independence assumptions rather than to a discriminative neural network.

Deep learning entered this pipeline without disturbing it. The hybrid DNN-HMM systems that produced the first large accuracy jump replaced the Gaussian mixtures with a neural network predicting HMM state posteriors, keeping the lexicon, the state topology, and the forced alignment step (Hinton et al., 2012, Deep Neural Networks for Acoustic Modeling in Speech Recognition, IEEE Signal Processing Magazine 29(6), 82–97). The network was better; the scaffolding was identical.

What made the scaffolding worth removing was not elegance but the cost of building it. A new language needed a phone inventory, a pronunciation lexicon, and linguistic expertise to define context-dependency rules. End-to-end models need audio and text.

timeline
    title Twenty years of answers to the alignment question
    1970s : IBM and CMU cast recognition as HMM decoding
          : Alignment supplied by forced alignment and Baum-Welch
    2006 : Graves et al. introduce CTC
         : Alignment marginalised away; frame labels no longer needed
    2012 : Graves publishes the RNN transducer, adding a label-side predictor
         : Hinton et al. document deep networks replacing Gaussian mixtures in hybrid systems
    2015 : Chan et al. Listen, Attend and Spell removes HMMs, CTC and the lexicon at once
    2020 : Gulati et al. Conformer interleaves convolution with self-attention
         : Baevski et al. wav2vec 2.0 reaches usable accuracy from ten minutes of labels
    2022 : Radford et al. Whisper trades streaming for zero-shot robustness at 680,000 hours

How CTC Actually Works

Connectionist temporal classification asks the network for a distribution over the label vocabulary at every frame, plus one extra symbol: the blank, written \(\varepsilon\), meaning "no output here" (Graves et al., 2006, Connectionist Temporal Classification, ICML). A frame-level path \(\pi\) of length \(T\) is then mapped to a label sequence by a collapse function \(\mathcal{B}\): merge consecutive duplicate symbols, then delete blanks. So A A ε B B and A ε ε B ε both collapse to AB, and the blank is what lets a genuinely repeated letter survive — ε A ε A ε gives AA where A A A A A gives A.

The probability the model assigns to a transcript is the sum over every path that collapses to it:

Written out, the sum has combinatorially many terms. Computed with the forward algorithm on a trellis over the blank-interleaved target, it costs \(O(TU)\), and the loss \(-\log p(\mathbf{y}\mid\mathbf{x})\) is differentiable with respect to every frame's distribution. The network is never told where anything is; it is told only that some alignment must work, and gradient descent discovers which.

The assumption hiding in the product

Look at the product again. Each factor is \(p_t(\pi_t \mid \mathbf{x})\) — conditioned on the audio, and on nothing else. The output at frame 400 does not depend on what the model emitted at frame 399. This conditional independence is what makes the dynamic program simple and the decoder parallel, and it is also a real modelling limitation: the model has no mechanism for "I have just written th, so e is likely". Everything a language model knows has to be squeezed into the acoustic encoder or bolted on at decode time via beam search with an external LM (see beam search decoding in ASR).

The characteristic symptom is spelling. A CTC model that hears an unfamiliar proper noun produces a phonetically defensible, orthographically wrong string, because nothing in its architecture represents the constraint that English words are spelled particular ways.

[IMAGE: The CTC collapse function illustrated as a funnel. Top row: five different length-8 frame paths over the alphabet {C, A, T, blank}. Arrows converge through a "merge repeats, drop blanks" gate to a single output "CAT". Caption: "Many paths, one label: the loss sums over the whole preimage rather than choosing a member of it."]

RNN-T: Giving the Output a Memory

The transducer keeps CTC's blank and marginalisation and repairs the independence assumption by adding a second network (Graves, 2012, Sequence Transduction with Recurrent Neural Networks, arXiv:1211.3711). Three components:

• The encoder maps audio frames to \(f_t\), exactly as in CTC.
• The prediction network is a causal language model over the labels emitted so far, producing \(g_u\) from \(y_{1:u-1}\). It never sees audio.
• The joint network combines them, typically \(z_{t,u} = W \tanh(f_t + g_u)\) followed by a softmax over the vocabulary plus blank.

Emission now happens on a two-dimensional lattice indexed by frame \(t\) and label position \(u\). At each node the model either emits a real label, which advances \(u\), or emits blank, which advances \(t\). A monotonic path from \((1,1)\) to \((T,U)\) is an alignment, and the loss again sums over all of them by dynamic programming.

The consequence is that the distribution over the next label is conditioned on the previous labels, which is exactly what CTC lacked. A transducer carries its own language model, learned jointly, so it does not need an external one to spell — though it usually still benefits from one for rare words.

The memory bill

The joint network is evaluated at every lattice node, so its output tensor has shape \(B \times T \times U \times V\). Put realistic numbers in: batch 32, encoder output subsampled by 4 so \(T = 250\), target length \(U = 100\) word-pieces, vocabulary \(V = 1000\), four bytes per float.

For one tensor, in one loss computation, before the backward pass allocates its own. This is the transducer's defining engineering problem, and it is why the literature is full of function-merging, pruned, and chunked transducer losses whose entire purpose is to avoid materialising that array. It is also why transducer training scripts have batch sizes that look absurdly small next to the encoder's capacity.

[IMAGE: Stacked-bar memory breakdown of one transducer training step at batch 32, T=250, U=100, V=1000, showing the joint-network output tensor at 3.2 GB dwarfing encoder activations, prediction-network state, and parameters. A second bar shows the same configuration under a pruned transducer loss with the lattice restricted to a narrow band. Caption: "The joint tensor is not a component of the memory budget; it is the memory budget."]

flowchart TB
  subgraph CTC["CTC head"]
    C1[Encoder frame f_t] --> C2[Softmax over vocab plus blank]
    C2 --> C3["Independent per frame"]
  end
  subgraph RNNT["Transducer"]
    R1[Encoder frame f_t] --> R3[Joint network]
    R2["Prediction net over y up to u minus 1"] --> R3
    R3 --> R4["Lattice over t and u"]
  end
  subgraph AED["Attention decoder"]
    A1[All encoder frames] --> A3[Cross attention]
    A2["Decoder state after y up to u minus 1"] --> A3
    A3 --> A4["Soft alignment, not monotonic"]
  end
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  class C1,C2,C3 amber
  class R1,R2,R3,R4 purple
  class A1,A2,A3,A4 rose

Attention Encoder-Decoders: Dropping Monotonicity Entirely

Listen, Attend and Spell removed the remaining structure (Chan et al., 2015, arXiv:1508.01211). No blank, no lattice, no monotonicity constraint: a decoder autoregressively produces characters, and at each step attends over the entire encoder output with a soft, learned weighting. Alignment becomes a by-product of attention rather than an object the loss sums over.

This is strictly more expressive, and the expressiveness is the problem. Speech is monotonic — audio and transcript advance together — and an unconstrained attention mechanism is free to violate that, which is the mechanism behind the classic attention-ASR failures of repeated and skipped words on long inputs (see attention failures in TTS for the same pathology in the other direction). More fundamentally, attending over all frames means the model may consult the last frame before committing to the first character, and no amount of engineering makes that streamable.

Whisper is this architecture at scale: an encoder-decoder trained on 680,000 hours of weakly supervised multilingual audio, with the decoder additionally used as a task interface, so special tokens select transcription versus translation, language, and timestamps (Radford et al., 2022, arXiv:2212.04356). Its robustness across accents, domains, and recording conditions is a data result rather than an architectural one, and it inherits the architecture's constraints unchanged: it processes fixed 30-second windows and is not a streaming model.

Seeing It in Motion

A streaming transducer decode, with the endpointing decision that makes it feel responsive:

sequenceDiagram
    participant M as Microphone
    participant E as Streaming encoder
    participant J as Joint plus prediction net
    participant U as User interface
    M->>E: Chunk 1, 320 ms of audio
    E->>J: Encoder states for chunk 1
    J->>U: Partial hypothesis "how"
    M->>E: Chunk 2
    E->>J: Encoder states for chunk 2
    J->>U: Partial hypothesis "how much"
    Note over J: Blank emitted repeatedly; no new label this chunk
    M->>E: Chunk 3
    E->>J: Encoder states for chunk 3
    J->>U: Partial "how much is it"
    Note over M,U: Silence detected; endpointer waits 600 ms before finalising
    J->>U: Final hypothesis with rescoring

The Note blocks carry the two facts that dominate perceived quality. Blank emission means a chunk can arrive and produce no visible output, which users read as lag. And the endpointer's hold-off — the silence duration before the system decides the turn ended — is a direct trade between cutting people off mid-sentence and making them wait (see endpointing and voice activity detection).

[IMAGE: Timeline strip of a spoken sentence with three tracks beneath it: raw audio energy, per-chunk emitted tokens (many chunks emitting only blank), and the user-visible partial transcript updating in bursts. Caption: "Emission is bursty even when speech is continuous, because blank is a legal and frequent output."]

Watch It Run

By the Numbers

Sources: Conformer figures from Gulati et al., 2020; wav2vec 2.0 figures from Baevski et al., 2020; Whisper from Radford et al., 2022. The Whisper number is approximate on purpose: published test-clean figures for the large models fall in roughly the 2.5 to 2.9 range depending on checkpoint version and text-normalisation conventions, and cross-paper comparisons on this benchmark are unreliable at the tenth of a point. Note also that the Whisper row is measured zero-shot, without training on LibriSpeech at all, which is a materially different claim from the rows above it.

The structural comparison matters as much as the accuracy one, and it is not measured but definitional:

A Concrete Example

Take a three-frame utterance and the target AB, with a vocabulary of {A, B, ε}. Suppose the encoder has produced these per-frame distributions:

Step 1: enumerate the valid paths. A length-3 path collapses to AB if merging repeats and deleting blanks yields exactly AB. There are five: A B ε, A ε B, ε A B, A A B, and A B B. Paths like A ε ε collapse to A, and ε A ε also gives A, so both are excluded.

Step 2: score each path as the product of its per-frame probabilities.

Summing: \(p(\texttt{AB} \mid \mathbf{x}) = 0.510\), so the CTC loss is \(-\ln 0.510 = 0.673\).

Step 3: get the same answer without enumerating. Interleave the target with blanks to get \(\ell' = [\varepsilon, A, \varepsilon, B, \varepsilon]\), positions \(s = 1 \ldots 5\), and let \(\alpha_t(s)\) be the total probability of length-\(t\) paths ending at position \(s\).

Initialise at \(t=1\), where only the first blank or the first real label can be entered: \(\alpha_1(1) = 0.2\), \(\alpha_1(2) = 0.7\), and everything else zero.

At \(t=2\), each cell sums the ways to arrive — stay at \(s\), step from \(s-1\), or skip from \(s-2\) when the skip does not merge two identical labels — then multiplies by frame 2's probability for the symbol at \(s\):

• \(\alpha_2(1) = 0.2 \times 0.3 = 0.06\)
• \(\alpha_2(2) = (0.7 + 0.2) \times 0.2 = 0.18\)
• \(\alpha_2(3) = (0 + 0.7) \times 0.3 = 0.21\)
• \(\alpha_2(4) = (0 + 0 + 0.7) \times 0.5 = 0.35\) — the skip from \(s=2\) is legal because \(A \neq B\)
• \(\alpha_2(5) = 0\)

At \(t=3\) only the final two positions can end a valid path:

• \(\alpha_3(4) = (0.35 + 0.21 + 0.18) \times 0.5 = 0.370\)
• \(\alpha_3(5) = (0 + 0.35) \times 0.4 = 0.140\)

\(p(\texttt{AB}) = \alpha_3(4) + \alpha_3(5) = 0.370 + 0.140 = 0.510\), matching the enumeration exactly.

The point of doing it both ways is the cost, not the answer. Enumeration touched five paths for \(T=3\); at \(T=250\) and \(U=100\) the path count is astronomical while the trellis stays at \(2U+1\) rows by \(T\) columns. Every practical alignment-free loss in speech — CTC, transducer, and their pruned descendants — is this same substitution of a dynamic program for a sum.

[IMAGE: The 5-by-3 CTC trellis for this example drawn as a grid, rows labelled with the blank-interleaved target and columns with frames, each reachable cell annotated with its alpha value from the worked example, and the legal transition arrows (stay, step, skip) drawn between cells. Caption: "The same 0.510, computed in fifteen cells instead of five paths."]

Where It Breaks

CTC's independence assumption is not a rounding error

The failure is systematic rather than random. Because no output conditions on any other, CTC assigns probability mass to strings that are phonetically plausible and orthographically impossible, and the errors cluster on exactly the tokens users care about: names, technical terms, alphanumeric strings. Beam search with an external LM recovers much of this and introduces its own problems, chiefly a fusion weight that must be tuned per domain and that trades rare-word accuracy against hallucinated common words.

The transducer's memory wall shapes its training recipe

The \(B \times T \times U \times V\) tensor derived above does not merely cost memory; it constrains batch size, which constrains the learning rate schedule, which changes the encoder you can train. Teams routinely discover that a transducer's accuracy gap to a CTC baseline is smaller than expected because the transducer was trained at a quarter of the batch size. Pruned transducer losses that restrict the lattice to a band around a CTC-derived alignment are the standard mitigation and reintroduce, in a controlled way, exactly the alignment commitment the loss was designed to avoid.

Streaming is a spectrum with a measurable price

"Streaming" resolves into a specific quantity: how many future frames the encoder is allowed to see for a given output. A causal encoder sees zero and is the fastest and the least accurate. Chunked attention sees a fixed lookahead per chunk. Layer-wise lookahead accumulates future context with depth, so a 12-layer encoder with two frames of lookahead per layer has an effective right context of 24 frames — 240 ms — which is easy to miss when reading the per-layer configuration (see causal and chunked attention for streaming). The honest framing is a curve of WER against algorithmic latency, and every point on it is a product decision.

Long-form audio breaks fixed-window models in specific ways

Whisper's 30-second window means long audio must be segmented, and errors at segment boundaries propagate: a segment that starts mid-word loses it, and a decoder conditioned on the previous segment's text can inherit and amplify an error. The widely reported failure of transcribing plausible sentences over silence is the same mechanism seen from another angle — an autoregressive decoder with a strong language prior, given no acoustic evidence, generates fluent text because that is what it was trained to do. Voice activity detection ahead of the model, and repetition detection after it, are standard mitigations and are not part of the model.

Evaluation hides more than it reveals

Word error rate weights every word equally, so a system that transcribes "the" perfectly and mangles every proper noun can beat one with the opposite profile. It is also computed after text normalisation, and normalisation choices — casing, punctuation, number formatting, contraction expansion — move reported WER by amounts comparable to real architectural differences. Cross-paper comparisons on LibriSpeech at the tenth of a point are close to meaningless unless the normalisation pipeline is shared.

Alternative Designs

The hybrid deserves its reputation. Attaching a CTC loss to the encoder of an attention model imposes monotonicity during training, which stabilises the attention that was going to be the failure mode, and the CTC head is available at decode time for joint scoring. It is a case where two mechanisms with opposite weaknesses genuinely cover for each other.

[IMAGE: Two-panel accuracy-versus-latency plot. Left panel: schematic WER curves for causal, chunked, and full-context encoders as algorithmic latency increases, converging toward the offline floor. Right panel: the same three configurations shown as strips of visible right context per output frame. Caption: "Right context is the currency; word error rate is what it buys."]

How It Is Used in Practice

Voice assistants and dictation on devices run transducers, and the reason is that the transducer is the only architecture that is natively streaming and carries its own language model, so it can emit partial results with acceptable spelling and no separate LM in the loop. The encoders are Conformers or their successors, quantised, with the prediction network deliberately kept small — often a stateless one- or two-token context — because that shrinks the joint tensor and, empirically, costs little accuracy.

Contact-centre and media transcription runs offline attention models, because nobody is waiting and the accuracy and punctuation are better. Here the operational problems are segmentation, speaker attribution, and long-form consistency rather than latency (see speaker diarisation).

Nearly everything now starts from a self-supervised encoder. wav2vec 2.0's demonstration that ten minutes of labelled audio plus 53,000 hours of unlabelled audio reaches 4.8/8.2 WER changed which languages are economically viable to serve, without changing the alignment machinery at all. The practical recipe — pretrain on unlabelled audio, fine-tune with CTC or a transducer loss, optionally distil for deployment — is now standard.

[IMAGE: Two deployment diagrams side by side. Left, an on-device stack: quantised Conformer encoder, tiny stateless prediction network, joint network, endpointer, with an arrow showing partial results reaching the screen mid-utterance. Right, a server batch stack: segmentation, attention encoder-decoder, diarisation, punctuation restoration, with a clock icon showing no latency constraint. Caption: "Same alignment problem, opposite answers, because one has a user waiting and the other does not."]

Two operational realities rarely appear in papers. Systems are usually evaluated on the wrong distribution, because production audio is telephone-band, noisy, code-switched, and full of proper nouns, while benchmarks are read speech. And contextual biasing — boosting a user's contact list, a product catalogue, or the current screen's vocabulary at decode time — often moves user-visible quality more than a model upgrade does, which is an argument for architectures with a clean place to inject it.

Insights Worth Remembering

1. The blank symbol is the single most consequential design choice in modern speech recognition. One extra output class converts an unaligned supervision problem into a sum over equal-length paths, which converts an intractable enumeration into a dynamic program. CTC, transducers, and their descendants all rest on it.

2. Conditional independence is a decode-speed optimisation with a spelling cost. CTC's parallel decode and simple loss come from the product form; the missing output-side context is not a bug to be fixed within CTC but the price of that form. Systems that need spelling either add a language model or change architecture.

3. The transducer trades memory for language modelling, and the exchange rate is steep. Roughly 3.2 GB for one joint tensor at a modest batch size is not a footnote; it dictates batch size, and batch size dictates everything downstream. Judge transducer-versus-CTC comparisons by whether both were trained at comparable effective batch size.

4. Monotonicity is free structure, and giving it up should be a decision. Speech is monotonic. Architectures that enforce it get streaming and stability; architectures that do not get expressiveness they mostly do not need and failure modes — repetition, skipping, hallucination over silence — that they must be defended against externally.

5. "Streaming" is not binary. The real quantity is right context per output, and it accumulates through depth in ways that per-layer configuration hides. A system described as streaming may carry hundreds of milliseconds of algorithmic latency before any compute is counted.

6. Self-supervision changed the data economics, not the alignment problem. Every improvement from wav2vec 2.0 onward sits on top of a CTC or transducer loss. Pretraining made encoders cheap to obtain; it did not answer which frame produced which character.

7. Whisper's robustness is a data result. Its architecture is a 2015 design; what is new is 680,000 hours of weak supervision and a decoder repurposed as a task interface. That is a lesson about supervision scale, not about attention.

8. Word error rate is a lossy summary and normalisation is part of the metric. Two systems within a few tenths of a point on LibriSpeech may differ substantially on the words a user actually notices, and reported differences of that size can be produced by text normalisation alone.

Open Questions

Can a single model be streaming and offline-accurate without compromise? Dual-mode and cascaded encoders — a streaming pass followed by a full-context rescoring pass sharing weights — narrow the gap and are widely deployed, but a measurable gap persists at every latency budget. Whether it is fundamental or an artefact of current training recipes is unresolved.

Does the alignment structure still earn its keep once the decoder is a large language model? Speech-LLMs project encoder output into a text model's embedding space and let autoregressive decoding handle the rest, discarding the explicit lattice. Early results are strong on accuracy and weak on streaming and on latency, and it is genuinely open whether monotonic alignment machinery returns as a constraint on these systems or is superseded.

What is the right unit of output? Characters, word pieces, whole words, and discrete acoustic units all work, and the choice interacts with \(U\), with the transducer's memory bill, and with rare-word behaviour. There is no principled account of the trade-off, only empirical per-language tuning.

Can hallucination over silence be eliminated architecturally rather than patched? The current mitigations — voice activity detection before the model, repetition detection after — are external. Whether a decoder trained on weak supervision can be made to abstain when acoustic evidence is absent, rather than generating fluent text, is an instance of the broader calibration problem and is not solved.

How should contextual information enter the model? Biasing toward a user's vocabulary demonstrably improves perceived quality, and the mechanisms in use — shallow fusion, trie-based boosting, attention over a context list — are heuristic and interact badly with the internal language model a transducer already has. A principled account of injecting per-utterance priors is missing.

Sources and Further Reading

Alignment-free losses

1. Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks." Proceedings of the 23^rd International Conference on Machine Learning (ICML), 369–376. ACM DL
2. Graves, A. (2012). "Sequence Transduction with Recurrent Neural Networks." arXiv:1211.3711

Attention-based and end-to-end systems

3. Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2015). "Listen, Attend and Spell." arXiv:1508.01211
4. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). "Robust Speech Recognition via Large-Scale Weak Supervision." arXiv:2212.04356

Encoders and representations

5. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., & Pang, R. (2020). "Conformer: Convolution-augmented Transformer for Speech Recognition." Interspeech 2020. arXiv:2005.08100
6. Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations." NeurIPS 2020. arXiv:2006.11477

The pipeline that came before

7. Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., & Kingsbury, B. (2012). "Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups." IEEE Signal Processing Magazine, 29(6), 82–97. IEEE Xplore

Related concepts on this site

8. CTC: connectionist temporal classification
9. The CTC blank token and alignment
10. Streaming versus offline ASR
11. Causal and chunked attention for streaming
12. The Conformer architecture
13. Beam search decoding in ASR
14. Endpointing and voice activity detection

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.