---
title: ASR Evaluation and Speech Applications
module: Speech
moduleNumber: 8
lessonNumber: 4
order: 804
summary: >
  A recognizer turns a waveform into text; this part scores that text and puts the
  same machinery to other uses. It opens with the self-supervised and weakly-
  supervised systems (wav2vec 2.0, HuBERT, Whisper) that made ASR error rates fall.
  Word error rate reuses the edit distance from the first module, run over words.
  Text-to-speech runs the whole pipeline in reverse — text to mel spectrogram to
  waveform. And a family of smaller tasks — wake-word detection, speaker
  recognition and diarization, language identification — reuse the same log-mel
  front end without the decoder.
topics: [Speech]
sources:
  - book: Jurafsky
    ref: "§26.5 ASR Evaluation: Word Error Rate; §26.6 TTS; §26.7 Other Speech Tasks"
---

This builds on [Automatic Speech Recognition](/natural-language-processing/speech/automatic-speech-recognition),
which built the recognizer end to end: the log-mel front end that turns a waveform into
spectrogram frames, and the encoder-decoder and CTC architectures that turn those frames into
text. Those systems all trained on paired audio and transcripts. This part opens with the two
lines of work that loosened that requirement and drove error rates down, then asks how good a
transcript is, how to run the pipeline in reverse to synthesize speech, and what else the same
front end can do.

## Self-supervised and weakly-supervised ASR

Every architecture so far is trained on _paired_ data — audio with its gold transcript — and
transcribed speech is scarce and expensive. Two lines of work past Jurafsky & Martin's treatment
attacked the data problem from opposite ends, and together they are why ASR error rates fell
sharply after 2020.

**Self-supervised pretraining.** Borrowing BERT's masked objective, the goal is to learn speech
representations from **unlabeled audio** before any transcript. **wav2vec 2.0** (Baevski, Zhou,
Mohamed, and Auli, NeurIPS 2020) is the canonical system. A convolutional front end maps the
waveform to latent frames; a transformer reads them with a fraction of frames
**masked**; and a **contrastive** loss requires the model, at each masked position $t$, to pick the true
quantized latent $q_t$ from a set $Q_t$ of distractors:

$$
\mathcal{L}_t = -\log
\frac{\exp\!\big(\mathrm{sim}(c_t, q_t)/\kappa\big)}
{\sum_{\tilde q \in Q_t} \exp\!\big(\mathrm{sim}(c_t, \tilde q)/\kappa\big)},
$$

with context vector $c_t$, cosine similarity $\mathrm{sim}$, and temperature $\kappa$. No
transcript is used; only afterward is a small labeled set used to fine-tune a CTC head. The
payoff is data efficiency: fine-tuning on all of LibriSpeech reaches roughly $1.8/3.3$ WER
(clean/other), and competitive recognition is possible from only minutes to tens of hours of
labeled speech, where a from-scratch model would need thousands.[^wav2vec] HuBERT (Hsu et al.
2021) replaced the contrastive target with prediction of cluster assignments of masked frames,
and the pretrain-then-fine-tune pattern became standard.

$$
% caption: Self-supervised speech pretraining (wav2vec 2.0 style). A convolutional front end
% makes latent frames; a transformer reads them with some frames masked; a contrastive loss
% picks the true latent for each masked frame from distractors -- all on unlabeled audio.
% Only afterward is a small labeled set used to fine-tune a CTC head.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  blk/.style={draw, minimum width=22mm, minimum height=7mm, align=center, font=\scriptsize},
  frm/.style={draw, minimum width=8mm, minimum height=6mm, font=\scriptsize},
  msk/.style={draw, minimum width=8mm, minimum height=6mm, font=\scriptsize, fill=red!12}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[font=\scriptsize] (wav) at (0,0) {unlabeled waveform};
  \node[blk] (cnn) at (0,1.1) {conv front end};
  % latent frames: plain frames at 0,1,3,5; masked frames at 2,4
  \foreach \x in {0,1,3,5} { \node[frm] at (\x*0.9,2.5) {}; }
  \node[msk] at (2*0.9,2.5) {?};
  \node[msk] at (4*0.9,2.5) {?};
  \node[anchor=west, font=\scriptsize, text=red] at (5.0,2.5) {masked frames};
  \node[blk, draw=acc, text=acc, minimum width=52mm] (tf) at (2.25,3.8) {transformer};
  \node[blk] (loss) at (2.25,5.0) {contrastive loss: true latent vs. distractors};
  \draw[->, acc, thick] (wav) -- (cnn);
  \draw[->, acc, thick] (cnn) -- (0,2.1);
  \draw[->, acc, thick] (2.25,2.85) -- (tf.south);
  \draw[->, acc, thick] (tf) -- (loss);
  \node[anchor=west, font=\scriptsize, text=black] at (5.4,3.8) {then f\/ine-tune a CTC head on a little labeled data};
\end{tikzpicture}
$$

**Large-scale weak supervision.** The opposite bet is to skip pretraining tricks and simply
collect an enormous amount of imperfectly-labeled data. **Whisper** (Radford, Kim, Xu, and
colleagues at OpenAI, 2022) trained an ordinary encoder-decoder transformer — the same AED this
lesson built — on **680,000 hours** of audio paired with transcripts scraped from the web,
across English and 96 other languages. The transcripts are noisy and were never hand-verified,
which is the _weak_ in weak supervision; the scale is what compensates. Trained this way,
Whisper recognizes speech **zero-shot** — it transcribes datasets it never trained on, and does
multilingual transcription and translation, without per-dataset fine-tuning, approaching the
robustness of a human listener across accents, noise, and domains.[^whisper] This is the lesson's
"Whisper-style recipe": an off-the-shelf encoder-decoder over log-mel frames,
made robust not by architecture but by the breadth of its training data.

The two lines are complementary: self-supervision extracts the most from _little_ labeled data,
weak supervision extracts robustness from _much_ noisy data. Both moved ASR from a hand-tuned
pipeline toward a general model trained at web scale — the trajectory the rest of the course
traced for text.

## Evaluation: word error rate

How wrong is a transcript? The standard metric, **word error rate** (WER), is built directly
on the [minimum edit distance](/natural-language-processing/foundations/regex-and-text-normalization)
from the first module — the same dynamic program, run over words instead of characters.[^jm-wer]
Align the recognizer's **hypothesis** to the gold **reference** by the alignment that minimizes
edits, count the word **substitutions** $S$, **insertions** $I$, and **deletions** $D$ needed,
and normalize by the number of words $N$ in the reference:

$$
\text{WER} = 100 \times \frac{S + D + I}{N}.
$$

Because insertions are counted, the rate can exceed $100\%$. A worked example, with the
edit-distance alignment shown per column (a dash marks an unaligned slot):

$$
% caption: Word error rate is the edit distance between hypothesis and reference,
% counted in words. This seven-word reference has 2 substitutions, 1 insertion, and 1
% deletion, for a WER of 100 x (2+1+1)/7 = 57.1%.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  ref/.style={font=\scriptsize},
  hyp/.style={font=\scriptsize},
  ev/.style={font=\scriptsize, text=black}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[anchor=east, font=\scriptsize] at (-0.15,2.0) {REF:};
  \node[anchor=east, font=\scriptsize] at (-0.15,1.2) {HYP:};
  \node[anchor=east, font=\scriptsize] at (-0.15,0.4) {eval:};
  \foreach \r [count=\i from 0] in {she,is,-,leaving,on,the,morning,train} {
    \node[ref] at (\i*1.35,2.0) {\r};
  }
  \foreach \h [count=\i from 0] in {she,was,really,leaving,in,the,morning,-} {
    \node[hyp] at (\i*1.35,1.2) {\h};
  }
  \foreach \e [count=\i from 0] in {ok,S,I,ok,S,ok,ok,D} {
    \node[ev] at (\i*1.35,0.4) {\e};
  }
\end{tikzpicture}
$$

Reading the columns left to right: two substitutions (_is_/_was_, _on_/_in_), one insertion
(_really_), and one deletion (_train_), with the rest correct. Over the seven-word reference
that is $100 \times \tfrac{2+1+1}{7} = 57.1\%$. The three error types partition the alignment:

| Error | Symbol | Reference | Hypothesis |
| --- | --- | --- | --- |
| Substitution | $S$ | one word | a different word |
| Insertion | $I$ | — (gap) | an extra word |
| Deletion | $D$ | a word | — (gap) |

The alignment minimizing $S + I + D$ is the [minimum edit
distance](/natural-language-processing/foundations/regex-and-text-normalization) with unit
costs, computed over the two word sequences $h$ (hypothesis) and $g$ (reference):

```algorithm
caption: $\textsc{WER-Align}(h, g)$ — Levenshtein DP over word sequences, unit costs
input: hypothesis words $h_{1..m}$, reference words $g_{1..n}$
$D[0,0] \gets 0$
for $i = 1$ to $m$ do $D[i,0] \gets i$   // deletions
for $j = 1$ to $n$ do $D[0,j] \gets j$   // insertions
for $i = 1$ to $m$ do
  for $j = 1$ to $n$ do
    $\text{sub} \gets D[i-1,j-1] + [\,h_i \ne g_j\,]$
    $D[i,j] \gets \min(\text{sub},\ D[i-1,j] + 1,\ D[i,j-1] + 1)$
return $D[m,n]$   // = S + I + D; backtrace recovers the per-word edits
```

The same script reports the **sentence error rate** — the fraction of utterances with at least
one error — and confusion statistics. When two systems' rates differ, the matched-pair MAPSSWE
significance test decides whether the gap is real, since word errors are correlated within an
utterance. WER weights every word equally, content and function alike — missing _Tuesday_ costs
the same as missing _of_ — an imperfection with no widely-agreed replacement.

> **Definition (Word error rate).** $\text{WER} = 100 \times (S+D+I)/N$, where $S$, $D$, $I$
> are the word substitutions, deletions, and insertions on the minimum-edit-distance
> alignment between the hypothesis and an $N$-word reference. It is the edit distance of the
> [foundations module](/natural-language-processing/foundations/regex-and-text-normalization),
> applied to words.

## The reverse problem: text-to-speech

**Text-to-speech** (TTS) runs the pipeline backwards: given a string of letters, produce a
waveform. It is built from the same encoder-decoder, in two stages.[^jm-tts]

$$
% caption: Text-to-speech reverses the arrows: an encoder-decoder predicts a mel
% spectrogram from text, and a vocoder converts the spectrogram back into a
% time-domain waveform.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  blk/.style={draw, minimum width=26mm, minimum height=10mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[blk] (txt) at (0,0) {text\\"it's time"};
  \node[blk, draw=acc, text=acc] (sp) at (3.6,0) {encoder-decoder\\(spectrogram)};
  \node[blk] (mel) at (7.4,0) {mel\\spectrogram};
  \node[blk, draw=acc, text=acc] (voc) at (11.1,0) {vocoder};
  \node[font=\scriptsize] at (11.1,-1.1) {waveform};
  \draw[->, acc, thick] (txt) -- (sp);
  \draw[->, acc, thick] (sp) -- (mel);
  \draw[->, acc, thick] (mel) -- (voc);
  \draw[->, acc, thick] (voc) -- (11.1,-0.85);
\end{tikzpicture}
$$

First, **text normalization** — the same
[non-standard-word problem](/natural-language-processing/foundations/regex-and-text-normalization)
from the first module — rewrites numbers, dates, and abbreviations into how they are spoken:
_151_ becomes _one hundred fifty one_ or _one fifty one_ depending on context. Then an
encoder-decoder for **spectrogram prediction** maps the normalized letters to a mel
spectrogram, autoregressively emitting one spectral frame at a time (the Tacotron 2 recipe).
Finally a **vocoder** inverts the spectrogram back to a waveform — the neural WaveNet vocoder
predicts audio samples autoregressively from the spectrogram. TTS is evaluated not by an
automatic metric but by human listeners, who rate synthesized utterances on a **mean opinion
score** or choose between two systems in an AB test.

## Other speech tasks

Recognition and synthesis are the two large problems, but the same front end — sample,
frame, log-mel — feeds a family of smaller tasks that ask something other than _what
words_ of the waveform.[^jm-other]

| Task | Question | Output | Model on the front end |
| --- | --- | --- | --- |
| Wake-word detection | is the trigger phrase present? | binary, streaming | small whole-word classifier |
| Speaker verification | is this speaker $X$? | binary | embedding + threshold |
| Speaker identification | which of $N$ enrolled speakers? | 1-of-$N$ | embedding + nearest match |
| Language identification | which language? | 1-of-$L$ | classifier |
| Diarization | who spoke when? | per-segment speaker label | VAD + embed + cluster |

**Wake-word detection** listens for one short phrase ("Alexa", "Hey Siri") whose sole
job is to _turn the recognizer on_. It runs continuously, so it must be small enough
to run on-device at the edge — also a privacy gain, since nothing reaches a server
until the wake word fires. It uses the ASR front end plus a small whole-word
classifier, not a full sequence model.

**Speaker recognition** asks _who_ is speaking rather than _what_. **Speaker
verification** is a binary decision (is this speaker $X$?), as a voice-authentication
system asks before releasing account information. **Speaker identification** is a
1-of-$N$ match against a database of enrolled speakers. **Language identification**
names the language spoken — useful, for instance, for routing a caller to an operator
who speaks it.

**Speaker diarization** answers _who spoke when_ in a long multi-speaker recording,
marking the start and end of each person's turns. It is what makes an automatic
transcript of a meeting, a classroom, or a doctor-patient visit readable. The classic
pipeline runs **voice-activity detection** to find stretches of speech, extracts a
**speaker embedding** from each stretch, and **clusters** the embeddings so that
segments from the same voice land together; newer end-to-end models map straight from
audio to a per-frame speaker label.

$$
% caption: A diarization timeline. Voice-activity detection finds speech segments,
% each is turned into a speaker embedding, and clustering assigns every segment to a
% speaker (A or B), recovering who spoke when across the recording.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % time axis
  \draw[->, black, thick] (-0.3,0) -- (10.6,0);
  \node[anchor=west, black, font=\scriptsize] at (10.4,-0.3) {time};
  % speaker A segments (blue), speaker B segments (red), on the timeline
  \fill[acc!25, draw=acc] (0.2,0.15) rectangle (2.4,0.75);
  \node[font=\scriptsize, text=acc] at (1.3,0.45) {A};
  \fill[red!20, draw=red] (2.8,0.15) rectangle (4.6,0.75);
  \node[font=\scriptsize, text=red] at (3.7,0.45) {B};
  \fill[acc!25, draw=acc] (5.0,0.15) rectangle (6.2,0.75);
  \node[font=\scriptsize, text=acc] at (5.6,0.45) {A};
  \fill[red!20, draw=red] (6.8,0.15) rectangle (9.6,0.75);
  \node[font=\scriptsize, text=red] at (8.2,0.45) {B};
  % gaps = silence (VAD)
  \node[anchor=north, font=\scriptsize, text=black] at (2.6,-0.05) {silence};
  \node[anchor=north, font=\scriptsize, text=black] at (6.5,-0.05) {silence};
  % legend
  \node[anchor=west, font=\scriptsize, text=acc] at (0.2,1.25) {speaker A};
  \node[anchor=west, font=\scriptsize, text=red] at (2.4,1.25) {speaker B};
  \node[anchor=west, font=\scriptsize, text=black] at (5.2,1.25) {gaps = voice-activity boundaries};
\end{tikzpicture}
$$

None of these needs the decoder or the alphabet of a recognizer; each is a classifier
or a clustering laid on top of the same spectral features, which is why the front end
of this lesson is the foundation for far more than transcription.

## Speech and the whole arc

Speech closes the course by reusing almost everything in it. The waveform
front end is genuinely new — sampling, framing, the DFT, the mel filterbank — but once the
signal is a sequence of spectrogram frames, ASR is the
[encoder-decoder with attention](/natural-language-processing/transformers/transformers-and-attention)
built for translation, trained with the cross-entropy loss of a
[language model](/natural-language-processing/foundations/n-gram-language-models), rescored by
a larger one. CTC is a leaner variant of the same seq2seq approach. Evaluation is the
[edit distance](/natural-language-processing/foundations/regex-and-text-normalization) from
the very first module. TTS is the whole thing inverted, its front end the same text
normalization the course opened with.

From
[what NLP is](/natural-language-processing/foundations/what-is-nlp) through
[classification](/natural-language-processing/classification/naive-bayes-and-sentiment),
[vector semantics](/natural-language-processing/semantics/vector-semantics-and-embeddings),
[sequence models](/natural-language-processing/sequences/rnns-and-lstms), and the
[transformer](/natural-language-processing/transformers/transformers-and-attention), one idea
kept recurring: represent language as sequences of vectors, and learn a function that
transduces one sequence into another. Speech extends that reach from text down to sound and
back — and it does so with the machinery already in hand.

[^wav2vec]: **Baevski, Zhou, Mohamed, and Auli**, "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations," NeurIPS 2020. A convolutional front end plus a transformer is pretrained on unlabeled audio with a contrastive task over masked, quantized latent frames, then fine-tuned with a CTC head on labeled speech. Fine-tuning on all of LibriSpeech reaches ~1.8/3.3 WER (clean/other); competitive recognition is possible from only minutes to tens of hours of labeled data. HuBERT (Hsu et al. 2021) is a related system that predicts cluster assignments of masked frames instead of a contrastive target.
[^whisper]: **Radford, Kim, Xu, et al. (OpenAI)**, "Robust Speech Recognition via Large-Scale Weak Supervision" (Whisper), 2022. An encoder-decoder transformer trained on 680,000 hours of audio paired with noisy, un-verified web transcripts (English plus 96 other languages). The scale of weakly-labeled data, rather than any new architecture, yields robust zero-shot multilingual transcription and translation without per-dataset fine-tuning. The attention-based encoder-decoder itself dates to Chan, Jaitly, Le, and Vinyals, "Listen, Attend and Spell," ICASSP 2016 (pyramidal encoder + attention decoder emitting characters with no independence assumption), and CTC to Graves, Fernández, Gomez, and Schmidhuber, "Connectionist Temporal Classification," ICML 2006.
[^jm-wer]: **Jurafsky & Martin**, §26.5 — ASR Evaluation: Word Error Rate: WER $= 100 \times (I+S+D)/N$ from the minimum edit distance in words between hypothesis and reference; the CALLHOME worked example (6 substitutions, 3 insertions, 1 deletion over 13 words = 76.9%); sentence error rate and the MAPSSWE significance test.
[^jm-tts]: **Jurafsky & Martin**, §26.6 — TTS: the encoder-decoder for spectrogram prediction (Tacotron 2) followed by a neural vocoder (WaveNet) that inverts the mel spectrogram to a waveform (Eq. 26.23); text normalization of non-standard words (§26.6.1); and mean-opinion-score and AB-test evaluation (§26.6.4).
[^jm-other]: **Jurafsky & Martin**, §26.7 — Other Speech Tasks: wake-word detection (small-footprint, edge-based, front end plus whole-word classifier); speaker recognition split into verification (binary) and identification (1-of-N); language identification; and speaker diarization ("who spoke when") by voice-activity detection, speaker embeddings, and clustering, or end-to-end per-frame labeling.