Language-Model Fusion

End-to-end ASR models - RNN-T, attention encoder-decoder, CTC - are trained on paired audio-text data. There is rarely enough of it. A well-trained language model, by contrast, has seen orders of magnitude more text. The question is not whether to use that LM; the question is how to combine two probability distributions that were never optimised against each other without letting one drown out the other.

The Shallow Fusion Baseline

Shallow fusion is the simplest possible answer: at beam search time, add the log-probability of an external LM to the log-probability of the acoustic model, weighted by a tunable scalar.

score(y | x) = log P_AM(y | x) + λ · log P_LM(y)

P_AM is the acoustic model score (attention, RNN-T, or CTC output); P_LM is the LM score; λ is found by grid search on a held-out set, typically in the range 0.1 to 0.4.

This works surprisingly well. The 2018 Toshniwal et al. comparison study found shallow fusion to be the strongest single-pass method across both Switchboard and large-scale Google voice-search data. No retraining required. You can swap LMs without touching the AM. The weakness is that it treats acoustic and language scores as independent, which they are not - the AM already learned something about word sequences from its training transcripts.

The Internal Language Model Problem

Every seq2seq or transducer model trained on paired data develops an implicit language model inside itself. The decoder has seen the training transcripts and has absorbed their n-gram statistics. When you add an external LM on top, you are effectively double-counting that prior: once from the AM decoder, once from the LM.

For in-domain data the double-counting is harmless or even helpful. For out-of-domain data (a medical ASR system being fine-tuned on legal text, for instance) the internal prior actively fights the new LM. The net effect can be worse than no LM at all.

Internal Language Model Estimation (ILME), introduced by Meng et al. (2021), fixes this by subtracting the internal LM score:

score(y | x) = log P_AM(y | x)
             + λ · log P_LM(y)
             - μ · log P_ILM(y)

P_ILM(y) is estimated by running the AM decoder with a zeroed or masked acoustic input - the model then conditions only on its own previous predictions, revealing the language prior it has learned. Subtracting this (with weight μ) cancels the double-counting before the external LM is added. In Meng et al.'s experiments on models trained on 30,000 hours of Microsoft speech, ILME reduced WER by up to 15.5% relative versus standard shallow fusion on out-of-domain LibriSpeech data, with no additional training required.

The follow-up work (Meng et al., ICASSP 2021) went further: during training, a separate ILM loss is minimised to make the internal prior more explicit and easier to subtract, yielding up to 31.5% relative WER reduction. The idea is to make the implicit thing explicit so you can cancel it cleanly.

Cold Fusion: Baking the LM into Training

Shallow fusion and ILME both operate at inference time; the acoustic model is frozen. Cold fusion, originally proposed for neural machine translation and adapted to ASR, takes the opposite route: the LM is fixed at the start, and the acoustic model is trained with live LM hidden states concatenated to its decoder states.

At each decoder step, the LM hidden state h_LM is concatenated or gated with the AM decoder hidden state h_AM:

h_combined = gating_network(h_AM, h_LM)
output_logits = projection(h_combined)

The hypothesis is that the AM learns to off-load lexical/syntactic prediction to the LM during training, freeing its own capacity for acoustic modelling. In practice cold fusion is harder to reproduce than shallow fusion: it requires the LM to be frozen throughout AM training, any change to the LM invalidates the trained AM, and the gains are inconsistent across domains. The Toshniwal et al. comparison found cold fusion competitive only after second-pass rescoring, not in single-pass decoding.

A related variant, deep fusion, adds the LM states after AM training via a learned gating network but without the AM being retrained from scratch with the LM present. Deep fusion avoids the retraining dependency but still couples the LM and AM at inference.

Density Ratio and Hybrid Methods

A principled alternative frames the problem as a Bayesian correction. If the AM was trained on a source-domain corpus with LM P_src, and you want to adapt to a target LM P_tgt, the corrected score is:

log P_AM(y | x) + log P_tgt(y) - log P_src(y)

This is the density ratio approach: subtract the source LM and add the target LM. The challenge is that P_src (the distribution implicit in the AM's transcripts) is not available as a standalone model. ILME can be seen as an approximation to this: the internal LM estimate serves as a proxy for P_src.

For CTC-based models, estimating the internal LM is non-trivial because CTC log-posteriors are computed non-autoregressively. Das et al. (2023) extended ILME to CTC by iteratively masking audio timesteps to estimate a pseudo log-likelihood, achieving up to 9.8% relative WER reduction on out-of-domain benchmarks without fine-tuning.

When it Falls Down

Domain gap is too large. If the test domain shares almost no vocabulary with the AM's training data, no inference-time fusion rescues recognition of rare tokens that the AM has never seen. The AM's acoustic-to-subword mapping is the binding constraint; LM fusion can only rerank hypotheses the beam actually generates.

Beam width acts as a bottleneck. Shallow fusion and ILME improve the scoring of hypotheses; they cannot introduce hypotheses the beam discarded. Narrow beams (width 4-8) used in production streaming systems may never generate the correct sequence, so LM fusion yields no gain. Lattice rescoring after a first-pass decode partially addresses this by expanding the hypothesis space, at the cost of two-pass latency.

Lambda and mu are fragile. The fusion weights found on one validation set transfer poorly to another domain. In production deployments with many target domains, per-domain grid search is expensive, and learned weight-prediction networks add engineering complexity.

Cold fusion degrades on new LMs. Because cold fusion trains the AM to expect a specific LM's hidden states, swapping to a better LM at deployment breaks the coupling. This limits cold fusion's practical utility in systems where the LM is regularly updated.

Streaming constraints. Full-sentence LMs are not causal; using them in a real-time streaming context requires either right-context approximations or chunk-based decoding that re-scores completed chunks. The latency budget often permits only a lightweight n-gram LM in the first pass, with neural LM rescoring applied off the critical path.

Further Reading

• Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition (Meng et al., 2021)
• Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition (Meng et al., ICASSP 2021)
• A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition (Toshniwal et al., 2018)
• Mask The Bias: Improving Domain-Adaptive Generalisation of CTC-based ASR with Internal Language Model Estimation (Das et al., 2023)