---
title: "Self-Supervised Speech Models and Synthesis"
module: Large Models & Agents
moduleNumber: 10
lessonNumber: 6
order: 1006
summary: >
  The recognition front-ends and alignment losses of part one all need transcribed
  audio, which is scarce. This second part removes that dependence: wav2vec 2.0 learns
  speech representations from unlabeled audio by a masked contrastive objective,
  HuBERT swaps the contrast for masked prediction of clustered units, and Whisper
  trades curation for scale with weakly-supervised web audio and a multitask token
  interface. We close with text-to-speech (the same length mismatch run backwards)
  and a tour of speech foundation models, discrete audio codecs,
  and neural TTS.
topics: [Large Models & Agents]
sources:
  - book: Goodfellow
    ref: "§12.3 — Speech Recognition"
  - book: Chollet
    ref: "Ch. 11 — sequence models for non-text modalities"
---

This builds on [Speech Recognition: Front-Ends and Alignment](/deep-learning/large-models-and-agents/speech-and-audio-models),
which framed ASR as mapping a long acoustic sequence $x_{1:T}$ to a short label
sequence $y_{1:U}$ and gave three ways to train through the missing alignment: CTC,
attention seq2seq, and the transducer. Every one of those methods needs transcribed
audio. This lesson asks what to do when transcripts are scarce or noisy — learn the
representation without them, or learn from a mountain of imperfect labels — and then
runs the whole pipeline in reverse to synthesize speech from text.

## wav2vec 2.0

The pipelines above are supervised: they need transcribed audio, which is scarce.
**wav2vec 2.0** learns speech representations from _unlabeled_ audio by a masked,
contrastive [self-supervised objective](/deep-learning/practical/transfer-learning),
then fine-tunes on a small labeled set.[^baevski-w2v2] Three components compose it.

- **Feature encoder.** A stack of seven strided 1-D convolutions maps the raw
  waveform to latent vectors $z_{1:T}$ at about $50\,\text{Hz}$, learning the
  front-end instead of using a fixed spectrogram. The strides multiply to $320$, so
  $16{,}000$ samples per second become $50$ latents per second, each a
  $512$-dimensional vector summarizing $25\,\text{ms}$ of audio.
- **Context network.** A masked [Transformer](/deep-learning/architectures/the-transformer-architecture)
  reads the (partially masked) latents and produces contextual representations
  $c_{1:T}$, so $c_t$ can see the whole utterance while $z_t$ sees only its local
  receptive field.
- **Quantization.** Each latent $z_t$ is mapped to a discrete code $q_t$ from a
  learned, product-quantized codebook; these codes are the prediction targets. Product
  quantization splits the vector into $G = 2$ groups and picks one of $V = 320$ entries
  per group, giving $V^G \approx 10^5$ composite codes from two small tables, a discrete
  target vocabulary the contrastive loss can point at.

Spans of latents are masked, and at each masked step $t$ the model must pick the
true quantized target $q_t$ out of a set of distractors $\tilde{q}$ sampled from
other masked steps, using a cosine-similarity score $\operatorname{sim}(\cdot,\cdot)$
with temperature $\kappa$. This is an InfoNCE contrastive loss:

$$
\mathcal{L}_m = -\,\log
\frac{\exp\!\parens{\operatorname{sim}(c_t,\, q_t)\,/\,\kappa}}
{\displaystyle\sum_{\tilde{q}\,\in\, Q_t} \exp\!\parens{\operatorname{sim}(c_t,\, \tilde{q})\,/\,\kappa}} ,
$$

where $Q_t$ contains the true target $q_t$ and the distractors (typically $100$
sampled from other masked positions in the same utterance). The numerator rewards a
high similarity between the context vector $c_t$ and the true code $q_t$; the
denominator, a log-sum-exp over all candidates, is a soft max that the loss drives
toward the true target. Minimizing $\mathcal{L}_m$ is a $|Q_t|$-way classification:
which of these codes is the one that was masked here? The temperature $\kappa$
(around $0.1$) sharpens the softmax, and drawing distractors from the same utterance
forces the model to use phonetic content rather than speaker or channel cues that
are constant across the clip. A diversity penalty on codebook usage, the entropy of
the averaged code-selection distribution, keeps the quantizer from collapsing onto a
few codes, which would make the contrastive task trivial and the targets
uninformative.

$$
% caption: wav2vec 2.0: convolutions encode the waveform to latents, a span is masked,
% and a Transformer context predicts the masked step's quantized code contrastively.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  lat/.style={draw, thick, minimum width=7mm, minimum height=7mm},
  ctx/.style={draw=acc, text=acc, thick, minimum width=7mm, minimum height=7mm}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % waveform
  \draw[acc, very thick] plot[domain=0:6, samples=140]
    (\x*0.85, {-3.7 + 0.28*sin(7*\x r) + 0.12*sin(29*\x r)});
  \node[black, anchor=east, font=\footnotesize] at (-0.2,-3.7) {\texttt{wave}};
  % CNN encoder band
  \draw[black, thick] (-0.1,-2.95) rectangle (5.3,-2.35);
  \node[black, font=\footnotesize] at (2.6,-2.65) {\texttt{CNN}\quad\texttt{feature}\quad\texttt{encoder}};
  % latents z
  \foreach \i in {0,1,2,3,4,5}
    \node[lat] (z\i) at (\i*0.95+0.35,-1.6) {};
  \node[black, anchor=east, font=\scriptsize] at (-0.2,-1.6) {$z_t$};
  \foreach \i in {0,...,5} \draw[->, black, thick] (\i*0.95+0.35,-2.3) -- (z\i);
  % mask the middle two latents
  \node[lat, draw=red, fill=red!15] at (2.25,-1.6) {};
  \node[lat, draw=red, fill=red!15] at (3.20,-1.6) {};
  \node[red, anchor=west, font=\footnotesize] at (5.4,-1.6) {\texttt{masked span}};
  % quantized targets q below-right
  \node[lat, draw=red, fill=red!15] (q) at (2.25,-0.4) {};
  \node[red, anchor=east, font=\scriptsize] at (1.65,-0.4) {$q_t$};
  \draw[->, red, thick] (2.25,-1.25) -- (q);
  % transformer context band
  \draw[acc, thick] (-0.1,0.55) rectangle (5.3,1.15);
  \node[acc, font=\footnotesize] at (2.6,0.85) {\texttt{Transformer context}};
  \foreach \i in {0,...,5} \draw[->, black, thick] (z\i) -- (\i*0.95+0.35,0.55);
  % contextual c
  \foreach \i in {0,...,5} \node[ctx] (c\i) at (\i*0.95+0.35,1.9) {};
  \node[acc, anchor=east, font=\scriptsize] at (-0.2,1.9) {$c_t$};
  \foreach \i in {0,...,5} \draw[->, acc, thick] (\i*0.95+0.35,1.15) -- (c\i);
  % contrastive arrow c_t <-> q_t
  \draw[<->, red, thick] (c1) to[bend right=25] (q);
  \node[red, font=\footnotesize, anchor=west] at (5.4,-0.4) {\texttt{contrast}};
\end{tikzpicture}
$$

Fine-tuning then adds a small projection on top of $c_{1:T}$ and trains the whole
stack with a CTC loss on labeled audio. With pretraining on thousands of hours of
unlabeled speech, wav2vec 2.0 matched prior fully-supervised systems using on the
order of ten minutes of labels.

## Whisper

**Whisper** takes the opposite data strategy: instead of self-supervision plus a
small clean label set, it trains a single [encoder-decoder Transformer](/deep-learning/architectures/the-transformer-architecture)
end-to-end on $680{,}000$ hours of **weakly supervised** audio-transcript pairs
scraped from the web.[^radford-whisper] The input is an $80$-channel log-mel
spectrogram; the encoder is a Transformer, and the decoder is an autoregressive
Transformer language model over text tokens.

> **Definition (Whisper).** A log-mel encoder-decoder Transformer trained on
> large-scale, noisily-labeled multilingual audio. A single model performs
> transcription, translation, language identification, and timestamp prediction,
> selected by **special tokens** prepended to the decoder's output sequence.

The design move that makes one model do many jobs is a **multitask token interface**:
every task is posed as next-token prediction over a sequence that begins with
control tokens specifying the language, the task, and whether timestamps are wanted.

$$
% caption: Whisper's decoder sequence. Leading special tokens (blue) select language and
% task before the text tokens (black); the whole thing is plain next-token prediction.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tok/.style={draw, black, thick, minimum width=15mm, minimum height=8mm, align=center},
  sp/.style={draw=acc, text=acc, thick, minimum width=15mm, minimum height=8mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[sp]  (a) at (0,0)    {\texttt{SOT}};
  \node[sp]  (b) at (1.85,0) {\texttt{EN}};
  \node[sp]  (c) at (3.7,0)  {\texttt{transcribe}};
  \node[tok] (d) at (5.75,0) {\texttt{the}};
  \node[tok] (e) at (7.6,0)  {\texttt{cat}};
  \node[tok] (f) at (9.35,0) {...};
  \node[sp]  (g) at (11.1,0) {\texttt{EOT}};
  \draw[->, black, thick] (a) -- (b);
  \draw[->, black, thick] (b) -- (c);
  \draw[->, black, thick] (c) -- (d);
  \draw[->, black, thick] (d) -- (e);
  \draw[->, black, thick] (e) -- (f);
  \draw[->, black, thick] (f) -- (g);
  \node[acc, anchor=north, font=\footnotesize, align=center] at (1.85,-0.75) {\texttt{task / language}\\\texttt{control tokens}};
  \node[black, anchor=north, font=\footnotesize] at (7.6,-0.75) {\texttt{transcript tokens}};
\end{tikzpicture}
$$

Swapping the `transcribe` token for `translate` makes the same weights emit an
English translation; a `notimestamps` token toggles whether the decoder also emits
time markers. The weak supervision means the training transcripts are imperfect,
but the scale and diversity make the model robust: Whisper transcribes accented,
noisy, and multilingual audio zero-shot, without any dataset-specific fine-tuning.

Scale replaces curation. wav2vec 2.0 pretrains without labels and fine-tunes on a
clean set; Whisper skips both, treating the noisy web pairs as if they were clean
and letting the loss average out the label noise. The trade shows in the failure
modes. Whisper hallucinates on silence or music,
emitting a plausible-looking transcript for audio that has no speech, because its
language-model decoder will always produce fluent text and nothing in the objective
punishes confident fabrication on out-of-distribution input. It also inherits
long-form drift: audio is processed in $30\,\text{second}$ windows, and timestamp
errors at a window boundary can desynchronize the next window, so long files need a
sliding-window stitch with the predicted timestamps as anchors. Both flaws are
consequences of trading a curated label set for scale.

## HuBERT

**HuBERT** (Hidden-unit BERT) is a third self-supervised recipe, closer to masked
language modeling than to contrastive learning.[^hsu-hubert] An offline clustering
step (initially $k$-means on MFCCs, later on the model's own features) assigns each
frame a discrete pseudo-label; the model then masks spans and predicts the cluster
id of the masked frames with a cross-entropy loss, like BERT predicting masked
tokens. Iterating, re-cluster the better features the model has learned, then
retrain, steadily sharpens the targets.

> **Remark (Contrastive vs. masked-prediction SSL).** wav2vec 2.0 and HuBERT both
> mask spans and learn from unlabeled audio, but differ in the target: wav2vec 2.0
> uses an InfoNCE contrast against in-batch distractors, while HuBERT predicts a
> fixed, offline-clustered class id with cross-entropy. The masked-prediction target
> sidesteps the distractor-sampling and codebook-collapse issues of the contrastive
> loss, at the cost of an external clustering pass.

| System | Supervision | Front-end | Loss | Decoder | Streaming |
| --- | --- | --- | --- | --- | --- |
| Deep Speech 2 | supervised | spectrogram | CTC | none (greedy + LM) | partial |
| LAS | supervised | learned (pyramid BiLSTM) | cross-entropy | attention | no |
| RNN-T | supervised | learned | transducer | predictor net | yes |
| wav2vec 2.0 | self-sup. + small labeled | learned CNN | InfoNCE, then CTC | none (CTC head) | partial |
| HuBERT | self-sup. + small labeled | learned CNN | masked cross-entropy | task head | partial |
| Whisper | weakly-sup. (web scale) | log-mel | cross-entropy | autoregressive | no |

The Conformer encoder, which interleaves convolution with self-attention so a layer
captures both local spectral detail and global context, is the common backbone
under several of these heads and is now the default acoustic encoder in
production-grade ASR.[^gulati-conformer]

## Text to speech

Synthesis runs the pipeline backwards: from a short text sequence $y_{1:U}$ to a
long waveform $x_{1:T}$, the same $T \gg U$ mismatch with the arrow reversed. The
modern decomposition splits it into two learned stages.

- **Acoustic model.** Map text (characters or phonemes) to a mel spectrogram,
  $y_{1:U} \mapsto M \in \mathbb{R}^{T' \times 80}$. This is the length-expanding
  step, and the alignment problem returns: how many frames does each phoneme occupy?
  Autoregressive models (Tacotron) let attention discover the alignment one frame at
  a time; non-autoregressive models (FastSpeech) predict an explicit per-phoneme
  duration, then expand each phoneme's encoding to that many frames before decoding,
  which removes the attention failures and lets all frames be generated in parallel.
- **Vocoder.** Map the mel spectrogram to a waveform, $M \mapsto x_{1:T}$. The mel
  discarded the phase, so the vocoder must synthesize a coherent signal from
  magnitude alone. WaveNet did this with an autoregressive stack of dilated causal
  convolutions, one sample at a time, which is faithful but slow; GAN vocoders
  (HiFi-GAN) and flow or diffusion vocoders generate the whole waveform in parallel
  and now dominate for their speed.

> **Takeaway (synthesis mirrors recognition).** ASR compresses a long waveform to a
> short label sequence and marginalizes over the unknown alignment; TTS expands a
> short label sequence to a long waveform and must _choose_ an alignment, either
> implicitly through attention or explicitly through a predicted duration. The mel
> spectrogram is the shared interface: ASR's front-end output and TTS's acoustic-model
> output are the same object.


## Speech as a token stream

Goodfellow's speech chapter stops where hand-built pipelines end. The developments since converge on one idea: if speech can be turned into a stream of _discrete tokens_, the entire language-model machinery (next-token prediction, decoder-only Transformers, scaling laws) applies to audio unchanged. Three developments follow.

**Neural audio codecs give speech a vocabulary.** A codec like **SoundStream** or **EnCodec** trains a convolutional autoencoder with a **residual vector quantizer**: the encoder maps a waveform to a low-rate latent, and a stack of $Q$ codebooks quantizes it, each codebook correcting the residual the previous one left.[^defossez-encodec] The output is a small grid of integer codes — a few thousand tokens per second of audio instead of $16{,}000$ float samples — that reconstructs to near-transparent audio. This is the audio analogue of subword tokenization: a fixed discrete vocabulary that spells any waveform, and the same object HuBERT's clustered units approximate without the reconstruction guarantee.

$$
% caption: A neural audio codec. A conv encoder compresses the waveform to a latent; a
% residual vector quantizer maps it to a small stack of integer code layers, and a conv
% decoder reconstructs the waveform. The codes are a discrete vocabulary for audio.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, thick, minimum width=18mm, minimum height=10mm, align=center},
  q/.style={draw=acc, text=acc, thick, minimum width=7mm, minimum height=6mm, inner sep=1pt, fill=acc!14, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[bx] (w) at (0,0) {\texttt{waveform}};
  \node[bx] (e) at (2.6,0) {\texttt{encoder}};
  \node[q] (q1) at (5.0,0.7) {\texttt{c1}};
  \node[q] (q2) at (5.0,0.0) {\texttt{c2}};
  \node[q] (q3) at (5.0,-0.7) {\texttt{c3}};
  \node[bx] (d) at (7.4,0) {\texttt{decoder}};
  \node[bx] (o) at (10.0,0) {\texttt{waveform}};
  \draw[->, black, thick] (w) -- (e);
  \draw[->, acc, thick] (e) -- (q2);
  \draw[->, acc, thick] (q2) -- (d);
  \draw[->, black, thick] (d) -- (o);
  \node[acc, anchor=south, font=\footnotesize] at (5.0,1.15) {RVQ codes};
  \node[black, anchor=north, font=\footnotesize] at (5.0,-1.15) {discrete tokens};
\end{tikzpicture}
$$

**Generative TTS becomes autoregressive token prediction.** Once audio is a token stream, a text-to-speech model is a language model over codec tokens conditioned on text. **VALL-E** casts TTS this way: it treats the codec codes as the target vocabulary and predicts them autoregressively from a phoneme prompt plus a few seconds of a speaker's audio, so voice cloning reduces to conditioning the prompt.[^wang-valle] This is the same shift the recognition side saw — replace a bespoke acoustic model and vocoder with next-token prediction over a learned discrete code — and it inherits the language model's strengths (in-context adaptation from a short prompt) and its failure modes (occasional repetition and drift, exactly the exposure-bias symptoms of any autoregressive decoder).

**Self-supervised encoders scaled into speech foundation models.** wav2vec 2.0 and HuBERT were the first instances; later models scaled the recipe. **XLS-R** and **wav2vec 2.0 XLSR** pretrain one masked encoder on tens of thousands of hours across a hundred-plus languages, and fine-tuning on a few hours of a low-resource language then reaches usable accuracy — the multilingual payoff of learning a shared acoustic representation before any transcript is seen.[^babu-xlsr] The pattern mirrors text exactly: pretrain one large encoder on unlabeled data at scale, adapt cheaply per task, and let the shared representation carry transfer to languages the labeled data never covered.

In short, the modality gap between speech and text has narrowed to a tokenizer. A codec turns audio into tokens; a clustering or quantization step turns those tokens into a fixed vocabulary; and from there the Transformer stack, the pretraining objectives, and the scaling behavior are the same ones the language-model lessons built. Speech stopped being a special case and became another sequence of tokens.

## Takeaways

- **wav2vec 2.0** pretrains a CNN-plus-Transformer encoder with a masked **InfoNCE**
  contrastive loss against product-quantized targets, then fine-tunes with CTC on a
  small labeled set; thousands of unlabeled hours plus ten minutes of labels matched
  prior fully-supervised systems.
- **HuBERT** swaps the contrast for masked prediction of offline-clustered units with
  cross-entropy, iterating the clustering to sharpen targets; it sidesteps the
  distractor-sampling and codebook-collapse issues of the contrastive loss.
- **Whisper** trades curation for scale: a log-mel encoder-decoder trained on
  $680{,}000$ hours of weakly-supervised web audio with a **multitask token interface**
  for transcription, translation, language id, and timestamps. Scale yields robustness
  but leaves hallucination on silence and long-form drift.
- **Text-to-speech** runs the same length mismatch in reverse: an acoustic model
  expands text to a mel spectrogram (choosing an alignment by attention, as in
  Tacotron, or by a predicted duration, as in FastSpeech) and a vocoder turns the mel
  into a waveform. The mel spectrogram is the shared interface between recognition and
  synthesis.
- **Since then:** neural audio codecs (SoundStream, EnCodec) give speech a
  discrete vocabulary via residual vector quantization, generative TTS (VALL-E)
  becomes autoregressive token prediction, and self-supervised encoders scaled into
  multilingual speech foundation models (XLS-R). Speech becomes another token stream.

[^baevski-w2v2]: **Baevski et al.**, _wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations_, NeurIPS 2020 — a CNN encoder, masked Transformer context, product-quantized targets, and an InfoNCE contrastive objective.
[^hsu-hubert]: **Hsu et al.**, _HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units_, IEEE/ACM TASLP 2021 — masked prediction of offline-clustered frame labels, refined by iterative re-clustering.
[^radford-whisper]: **Radford et al.**, _Robust Speech Recognition via Large-Scale Weak Supervision_, 2022 — a log-mel encoder-decoder trained on 680k hours of weakly-labeled web audio with a multitask special-token interface.
[^gulati-conformer]: **Gulati et al.**, _Conformer: Convolution-augmented Transformer for Speech Recognition_, Interspeech 2020 — interleaving convolution and self-attention so each block models both local and global acoustic structure.
[^defossez-encodec]: **Défossez et al.**, _High Fidelity Neural Audio Compression_ (EnCodec), 2022, and **Zeghidour et al.**, _SoundStream_, IEEE/ACM TASLP 2021 — convolutional autoencoders with residual vector quantization that turn a waveform into a small stack of discrete code layers.
[^wang-valle]: **Wang et al.**, _Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers_ (VALL-E), 2023 — casts TTS as autoregressive prediction of neural-codec tokens conditioned on text and a short speaker prompt, enabling in-context voice cloning.
[^babu-xlsr]: **Babu et al.**, _XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale_, 2021 — a wav2vec 2.0 encoder pretrained on tens of thousands of hours across 128 languages, transferring to low-resource languages with little labeled data.