HuBERT and Discrete Audio Units

Before HuBERT, the core problem with applying BERT-style masked prediction to speech was this: what is the "word" you are predicting? Text has a fixed vocabulary; raw audio is a continuous 80-dimensional mel-spectrogram where adjacent frames are nearly identical, and no natural token boundary exists. wav2vec 2.0 (Baevski et al., 2020) solved this by jointly learning a quantiser and a contrastive objective, but the interaction between quantisation and the masked loss made training fragile. HuBERT takes a different route: decouple the two problems entirely. Cluster first, predict later.

The Offline Clustering Trick

The central idea is refreshingly simple. Before pre-training begins, run k-means on MFCC features (39-dimensional, standard acoustic features) over the training corpus to produce a fixed label sequence for every audio file. Each 20 ms frame gets one of K cluster IDs, say K = 100. These are the "hidden units" in the name.

During pre-training, the model masks a random 40% of the input frames (in contiguous spans, following wav2vec 2.0 conventions) and tries to predict the cluster label of each masked frame using a cross-entropy loss. The loss is applied only to masked positions, exactly like BERT. Unmasked positions provide context but receive no gradient signal.

The encoder is a standard convolutional feature extractor followed by a 12-layer (base) or 24-layer (large) Transformer. The projection head maps Transformer output to a K-class softmax.

raw audio (16 kHz)
     │
[conv feature extractor]   ← 512-dim, 20ms stride
     │
[random masking of ~40% spans]
     │
[12-layer Transformer]
     │
[linear projection → K logits]
     │
cross-entropy vs offline k-means label   (masked frames only)

The elegance is that the cluster labels need not be linguistically meaningful at first. They just need to be consistent enough that predicting them forces the Transformer to learn useful context. Because the loss is masked, the model cannot cheat by memorising local spectral statistics; it must integrate context across the utterance to correctly predict a hidden frame.

Iterative Refinement of Labels

The offline clustering is iterated. After the first round of pre-training, the Transformer's internal representations (say, layer 6 of the base model) are themselves used as features for a fresh k-means run. This produces higher-quality cluster labels, which are then used to train a second round of the model from scratch (or from the first checkpoint).

In practice, two iterations are sufficient. The first-iteration labels resemble crude phone-like units. After the second iteration, the correspondence to phonemes sharpens dramatically. On phone purity evaluations (where each cluster is mapped to its most common ground-truth phoneme), second-iteration HuBERT large achieves higher purity than first-iteration HuBERT base, and both exceed wav2vec 2.0's quantiser at matched model sizes.

Why does this work? Intuitively, the Transformer has learned to smooth over within-phone variation in its representations, so k-means on those representations finds clusters that respect phone boundaries more reliably than k-means on raw MFCCs.

From Units to Downstream Tasks

The pre-trained encoder is fine-tuned for ASR by adding a CTC or attention decoder and training on labelled data in the usual way. On LibriSpeech, HuBERT large reaches around 1.9% WER on test-clean and 3.7% on test-other with standard 960-hour fine-tuning, competitive with the best supervised systems at the time.

But the discrete units themselves have a life beyond ASR. Because the unit sequence is a symbolic, low-bit-rate summary of speech content, it forms the basis of "textless NLP", research that processes spoken language without ever converting it to text. The GSLM paper (Lakhotia et al., 2021) demonstrated that a language model trained on HuBERT discrete units generates coherent spoken continuations of audio prompts, measured by acoustic and linguistic metrics. The pipeline looks like this:

Run-length compression is important: consecutive identical units (which occur because a single phoneme spans many 20 ms frames) are collapsed to a single token with an optional count. This reduces sequence length by roughly 3-4x and makes LM training tractable.

A related application is voice conversion: encode source speech to units, re-synthesise with a vocoder conditioned on a target speaker's timbre. Because the unit sequence is largely speaker-independent (content is preserved, speaker identity is largely discarded), the synthesised output carries the source's words in the target's voice. This property is not engineered; it emerges from the fact that k-means on encoder representations clusters by acoustic content rather than by speaker.

Why Masked Prediction on Discrete Targets Works Better than Contrastive Loss

wav2vec 2.0's contrastive objective requires sampling negative examples from the same batch, which introduces noise when the negatives happen to be acoustically similar to the positive. HuBERT's cross-entropy over cluster IDs avoids this: the target is fixed (offline), the loss landscape is smooth, and training is stable across a wider range of learning-rate schedules. The tradeoff is that bootstrap quality matters: poor initial clustering produces noisy labels that slow convergence. In practice, even mediocre first-iteration MFCCs produce targets good enough to bootstrap something useful, so this has not been a practical obstacle.

When It Falls Down

Label noise is irreducible. K-means on MFCCs is a lossy phoneme approximation. Coarticulation, speaker-specific allophone variation, and prosodic stress all introduce examples where the "correct" cluster is ambiguous. The model is trained to predict this noise; it cannot be better than its teacher. A key symptom is that rare phonemes (e.g. the phoneme /zh/ in English) cluster poorly and are under-represented in the label set.

The benefit of iteration saturates quickly. Beyond two iterations, improvements are marginal and the cost (train a new k-means, retrain the model) is substantial. You cannot iterate your way to phoneme-perfect targets using this approach alone.

Long-form audio fails without resampling. HuBERT's Transformer context window (the number of frames it can attend to) is bounded by GPU memory. An utterance longer than roughly 30 seconds requires chunking, and the boundary between chunks drops contextual information. This is less critical for ASR (where a CTC/attention decoder handles long sequences) but is a real issue when you use the unit sequence directly as a generative model's vocabulary.

Cross-lingual transfer is inconsistent. HuBERT pre-trained on English LibriSpeech produces units that are phonetically meaningful for English but the same cluster assignments do not cleanly map to phonemes in typologically distant languages. The units capture what English-style k-means finds salient; that is not necessarily what matters in a tonal or agglutinative language. mHuBERT (multilingual HuBERT) addresses this by pre-training on mixed-language data, but the unit inventory becomes language-mixed rather than language-universal.

Discrete units lose prosody and speaker identity by design. For ASR this is fine. For synthesis, voice conversion, or any application where emotional tone or speaking rate matters, the deduplicated unit sequence has already discarded most of that signal. Supplementary conditioning (speaker embeddings, pitch tracks) must be reintroduced at the synthesiser stage.

Further Reading

• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units - Hsu et al., 2021; the primary paper with full ablations.
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations - Baevski et al., 2020; the contrastive predecessor whose instabilities motivated HuBERT's design.
• Generative Spoken Language Modeling from Raw Audio - Lakhotia et al., 2021; introduces the GSLM textless NLP pipeline built on top of discrete units from HuBERT and related encoders.