Self-Supervised Speech Models and Synthesis

This builds on Speech Recognition: Front-Ends and Alignment, which framed ASR as mapping a long acoustic sequence $x_{1:T}$ to a short label sequence $y_{1:U}$ and gave three ways to train through the missing alignment: CTC, attention seq2seq, and the transducer. Every one of those methods needs transcribed audio. This lesson asks what to do when transcripts are scarce or noisy — learn the representation without them, or learn from a mountain of imperfect labels — and then runs the whole pipeline in reverse to synthesize speech from text.

wav2vec 2.0

The pipelines above are supervised: they need transcribed audio, which is scarce. wav2vec 2.0 learns speech representations from unlabeled audio by a masked, contrastive self-supervised objective, then fine-tunes on a small labeled set.¹ Three components compose it.

• Feature encoder. A stack of seven strided 1-D convolutions maps the raw waveform to latent vectors $z_{1:T}$ at about $50\text{Hz}$, learning the front-end instead of using a fixed spectrogram. The strides multiply to $320$, so $16,000$ samples per second become $50$ latents per second, each a $512$-dimensional vector summarizing $25\text{ms}$ of audio.
• Context network. A masked Transformer reads the (partially masked) latents and produces contextual representations $c_{1:T}$, so $c_t$ can see the whole utterance while $z_t$ sees only its local receptive field.
• Quantization. Each latent $z_t$ is mapped to a discrete code $q_t$ from a learned, product-quantized codebook; these codes are the prediction targets. Product quantization splits the vector into $G=2$ groups and picks one of $V=320$ entries per group, giving $V^G\approx 10^5$ composite codes from two small tables, a discrete target vocabulary the contrastive loss can point at.

Spans of latents are masked, and at each masked step $t$ the model must pick the true quantized target $q_t$ out of a set of distractors $\tilde{q}$ sampled from other masked steps, using a cosine-similarity score $\mathrm{sim}(\cdot ,\cdot )$ with temperature $\kappa$. This is an InfoNCE contrastive loss:

where $Q_t$ contains the true target $q_t$ and the distractors (typically $100$ sampled from other masked positions in the same utterance). The numerator rewards a high similarity between the context vector $c_t$ and the true code $q_t$; the denominator, a log-sum-exp over all candidates, is a soft max that the loss drives toward the true target. Minimizing ${\mathcal{L}}_m$ is a $|Q_t|$-way classification: which of these codes is the one that was masked here? The temperature $\kappa$ (around $0.1$) sharpens the softmax, and drawing distractors from the same utterance forces the model to use phonetic content rather than speaker or channel cues that are constant across the clip. A diversity penalty on codebook usage, the entropy of the averaged code-selection distribution, keeps the quantizer from collapsing onto a few codes, which would make the contrastive task trivial and the targets uninformative.

Fine-tuning then adds a small projection on top of $c_{1:T}$ and trains the whole stack with a CTC loss on labeled audio. With pretraining on thousands of hours of unlabeled speech, wav2vec 2.0 matched prior fully-supervised systems using on the order of ten minutes of labels.

Whisper

Whisper takes the opposite data strategy: instead of self-supervision plus a small clean label set, it trains a single encoder-decoder Transformer end-to-end on $680,000$ hours of weakly supervised audio-transcript pairs scraped from the web.² The input is an $80$-channel log-mel spectrogram; the encoder is a Transformer, and the decoder is an autoregressive Transformer language model over text tokens.

The design move that makes one model do many jobs is a multitask token interface: every task is posed as next-token prediction over a sequence that begins with control tokens specifying the language, the task, and whether timestamps are wanted.

Swapping the transcribe token for translate makes the same weights emit an English translation; a notimestamps token toggles whether the decoder also emits time markers. The weak supervision means the training transcripts are imperfect, but the scale and diversity make the model robust: Whisper transcribes accented, noisy, and multilingual audio zero-shot, without any dataset-specific fine-tuning.

Scale replaces curation. wav2vec 2.0 pretrains without labels and fine-tunes on a clean set; Whisper skips both, treating the noisy web pairs as if they were clean and letting the loss average out the label noise. The trade shows in the failure modes. Whisper hallucinates on silence or music, emitting a plausible-looking transcript for audio that has no speech, because its language-model decoder will always produce fluent text and nothing in the objective punishes confident fabrication on out-of-distribution input. It also inherits long-form drift: audio is processed in $30\text{second}$ windows, and timestamp errors at a window boundary can desynchronize the next window, so long files need a sliding-window stitch with the predicted timestamps as anchors. Both flaws are consequences of trading a curated label set for scale.

HuBERT

HuBERT (Hidden-unit BERT) is a third self-supervised recipe, closer to masked language modeling than to contrastive learning.³ An offline clustering step (initially $k$-means on MFCCs, later on the model's own features) assigns each frame a discrete pseudo-label; the model then masks spans and predicts the cluster id of the masked frames with a cross-entropy loss, like BERT predicting masked tokens. Iterating, re-cluster the better features the model has learned, then retrain, steadily sharpens the targets.

The Conformer encoder, which interleaves convolution with self-attention so a layer captures both local spectral detail and global context, is the common backbone under several of these heads and is now the default acoustic encoder in production-grade ASR.⁴

Text to speech

Synthesis runs the pipeline backwards: from a short text sequence $y_{1:U}$ to a long waveform $x_{1:T}$, the same $T\gg U$ mismatch with the arrow reversed. The modern decomposition splits it into two learned stages.

• Acoustic model. Map text (characters or phonemes) to a mel spectrogram, $y_{1:U}\mapsto M\in {\mathbb{R}}^{T^{\prime }\times 80}$. This is the length-expanding step, and the alignment problem returns: how many frames does each phoneme occupy? Autoregressive models (Tacotron) let attention discover the alignment one frame at a time; non-autoregressive models (FastSpeech) predict an explicit per-phoneme duration, then expand each phoneme's encoding to that many frames before decoding, which removes the attention failures and lets all frames be generated in parallel.
• Vocoder. Map the mel spectrogram to a waveform, $M\mapsto x_{1:T}$. The mel discarded the phase, so the vocoder must synthesize a coherent signal from magnitude alone. WaveNet did this with an autoregressive stack of dilated causal convolutions, one sample at a time, which is faithful but slow; GAN vocoders (HiFi-GAN) and flow or diffusion vocoders generate the whole waveform in parallel and now dominate for their speed.

Speech as a token stream

Goodfellow's speech chapter stops where hand-built pipelines end. The developments since converge on one idea: if speech can be turned into a stream of discrete tokens, the entire language-model machinery (next-token prediction, decoder-only Transformers, scaling laws) applies to audio unchanged. Three developments follow.

Neural audio codecs give speech a vocabulary. A codec like SoundStream or EnCodec trains a convolutional autoencoder with a residual vector quantizer: the encoder maps a waveform to a low-rate latent, and a stack of $Q$ codebooks quantizes it, each codebook correcting the residual the previous one left.⁵ The output is a small grid of integer codes — a few thousand tokens per second of audio instead of $16,000$ float samples — that reconstructs to near-transparent audio. This is the audio analogue of subword tokenization: a fixed discrete vocabulary that spells any waveform, and the same object HuBERT's clustered units approximate without the reconstruction guarantee.

Generative TTS becomes autoregressive token prediction. Once audio is a token stream, a text-to-speech model is a language model over codec tokens conditioned on text. VALL-E casts TTS this way: it treats the codec codes as the target vocabulary and predicts them autoregressively from a phoneme prompt plus a few seconds of a speaker's audio, so voice cloning reduces to conditioning the prompt.⁶ This is the same shift the recognition side saw — replace a bespoke acoustic model and vocoder with next-token prediction over a learned discrete code — and it inherits the language model's strengths (in-context adaptation from a short prompt) and its failure modes (occasional repetition and drift, exactly the exposure-bias symptoms of any autoregressive decoder).

Self-supervised encoders scaled into speech foundation models. wav2vec 2.0 and HuBERT were the first instances; later models scaled the recipe. XLS-R and wav2vec 2.0 XLSR pretrain one masked encoder on tens of thousands of hours across a hundred-plus languages, and fine-tuning on a few hours of a low-resource language then reaches usable accuracy — the multilingual payoff of learning a shared acoustic representation before any transcript is seen.⁷ The pattern mirrors text exactly: pretrain one large encoder on unlabeled data at scale, adapt cheaply per task, and let the shared representation carry transfer to languages the labeled data never covered.

In short, the modality gap between speech and text has narrowed to a tokenizer. A codec turns audio into tokens; a clustering or quantization step turns those tokens into a fixed vocabulary; and from there the Transformer stack, the pretraining objectives, and the scaling behavior are the same ones the language-model lessons built. Speech stopped being a special case and became another sequence of tokens.

Takeaways

• wav2vec 2.0 pretrains a CNN-plus-Transformer encoder with a masked InfoNCE contrastive loss against product-quantized targets, then fine-tunes with CTC on a small labeled set; thousands of unlabeled hours plus ten minutes of labels matched prior fully-supervised systems.
• HuBERT swaps the contrast for masked prediction of offline-clustered units with cross-entropy, iterating the clustering to sharpen targets; it sidesteps the distractor-sampling and codebook-collapse issues of the contrastive loss.
• Whisper trades curation for scale: a log-mel encoder-decoder trained on $680,000$ hours of weakly-supervised web audio with a multitask token interface for transcription, translation, language id, and timestamps. Scale yields robustness but leaves hallucination on silence and long-form drift.
• Text-to-speech runs the same length mismatch in reverse: an acoustic model expands text to a mel spectrogram (choosing an alignment by attention, as in Tacotron, or by a predicted duration, as in FastSpeech) and a vocoder turns the mel into a waveform. The mel spectrogram is the shared interface between recognition and synthesis.
• Since then: neural audio codecs (SoundStream, EnCodec) give speech a discrete vocabulary via residual vector quantization, generative TTS (VALL-E) becomes autoregressive token prediction, and self-supervised encoders scaled into multilingual speech foundation models (XLS-R). Speech becomes another token stream.