ASR Evaluation and Speech Applications

This builds on Automatic Speech Recognition, which built the recognizer end to end: the log-mel front end that turns a waveform into spectrogram frames, and the encoder-decoder and CTC architectures that turn those frames into text. Those systems all trained on paired audio and transcripts. This part opens with the two lines of work that loosened that requirement and drove error rates down, then asks how good a transcript is, how to run the pipeline in reverse to synthesize speech, and what else the same front end can do.

Self-supervised and weakly-supervised ASR

Every architecture so far is trained on paired data — audio with its gold transcript — and transcribed speech is scarce and expensive. Two lines of work past Jurafsky & Martin's treatment attacked the data problem from opposite ends, and together they are why ASR error rates fell sharply after 2020.

Self-supervised pretraining. Borrowing BERT's masked objective, the goal is to learn speech representations from unlabeled audio before any transcript. wav2vec 2.0 (Baevski, Zhou, Mohamed, and Auli, NeurIPS 2020) is the canonical system. A convolutional front end maps the waveform to latent frames; a transformer reads them with a fraction of frames masked; and a contrastive loss requires the model, at each masked position $t$, to pick the true quantized latent $q_t$ from a set $Q_t$ of distractors:

with context vector $c_t$, cosine similarity $\mathrm{sim}$, and temperature $\kappa$. No transcript is used; only afterward is a small labeled set used to fine-tune a CTC head. The payoff is data efficiency: fine-tuning on all of LibriSpeech reaches roughly $1.8/3.3$ WER (clean/other), and competitive recognition is possible from only minutes to tens of hours of labeled speech, where a from-scratch model would need thousands.¹ HuBERT (Hsu et al. 2021) replaced the contrastive target with prediction of cluster assignments of masked frames, and the pretrain-then-fine-tune pattern became standard.

Large-scale weak supervision. The opposite bet is to skip pretraining tricks and simply collect an enormous amount of imperfectly-labeled data. Whisper (Radford, Kim, Xu, and colleagues at OpenAI, 2022) trained an ordinary encoder-decoder transformer — the same AED this lesson built — on 680,000 hours of audio paired with transcripts scraped from the web, across English and 96 other languages. The transcripts are noisy and were never hand-verified, which is the weak in weak supervision; the scale is what compensates. Trained this way, Whisper recognizes speech zero-shot — it transcribes datasets it never trained on, and does multilingual transcription and translation, without per-dataset fine-tuning, approaching the robustness of a human listener across accents, noise, and domains.² This is the lesson's Whisper-style recipe: an off-the-shelf encoder-decoder over log-mel frames, made robust not by architecture but by the breadth of its training data.

The two lines are complementary: self-supervision extracts the most from little labeled data, weak supervision extracts robustness from much noisy data. Both moved ASR from a hand-tuned pipeline toward a general model trained at web scale — the trajectory the rest of the course traced for text.

Evaluation: word error rate

How wrong is a transcript? The standard metric, word error rate (WER), is built directly on the minimum edit distance from the first module — the same dynamic program, run over words instead of characters.³ Align the recognizer's hypothesis to the gold reference by the alignment that minimizes edits, count the word substitutions $S$, insertions $I$, and deletions $D$ needed, and normalize by the number of words $N$ in the reference:

Because insertions are counted, the rate can exceed $100\%$. A worked example, with the edit-distance alignment shown per column (a dash marks an unaligned slot):

Reading the columns left to right: two substitutions (is/was, on/in), one insertion (really), and one deletion (train), with the rest correct. Over the seven-word reference that is $100\times \frac{2+1+1}{7}=57.1\%$. The three error types partition the alignment:

The alignment minimizing $S+I+D$ is the minimum edit distance with unit costs, computed over the two word sequences $h$ (hypothesis) and $g$ (reference):

The same script reports the sentence error rate — the fraction of utterances with at least one error — and confusion statistics. When two systems' rates differ, the matched-pair MAPSSWE significance test decides whether the gap is real, since word errors are correlated within an utterance. WER weights every word equally, content and function alike — missing Tuesday costs the same as missing of — an imperfection with no widely-agreed replacement.

The reverse problem: text-to-speech

Text-to-speech (TTS) runs the pipeline backwards: given a string of letters, produce a waveform. It is built from the same encoder-decoder, in two stages.⁴

First, text normalization — the same non-standard-word problem from the first module — rewrites numbers, dates, and abbreviations into how they are spoken: 151 becomes one hundred fifty one or one fifty one depending on context. Then an encoder-decoder for spectrogram prediction maps the normalized letters to a mel spectrogram, autoregressively emitting one spectral frame at a time (the Tacotron 2 recipe). Finally a vocoder inverts the spectrogram back to a waveform — the neural WaveNet vocoder predicts audio samples autoregressively from the spectrogram. TTS is evaluated not by an automatic metric but by human listeners, who rate synthesized utterances on a mean opinion score or choose between two systems in an AB test.

Other speech tasks

Recognition and synthesis are the two large problems, but the same front end — sample, frame, log-mel — feeds a family of smaller tasks that ask something other than what words of the waveform.⁵

Wake-word detection listens for one short phrase (Alexa, Hey Siri) whose sole job is to turn the recognizer on. It runs continuously, so it must be small enough to run on-device at the edge — also a privacy gain, since nothing reaches a server until the wake word fires. It uses the ASR front end plus a small whole-word classifier, not a full sequence model.

Speaker recognition asks who is speaking rather than what. Speaker verification is a binary decision (is this speaker $X$?), as a voice-authentication system asks before releasing account information. Speaker identification is a 1-of-$N$ match against a database of enrolled speakers. Language identification names the language spoken — useful, for instance, for routing a caller to an operator who speaks it.

Speaker diarization answers who spoke when in a long multi-speaker recording, marking the start and end of each person's turns. It is what makes an automatic transcript of a meeting, a classroom, or a doctor-patient visit readable. The classic pipeline runs voice-activity detection to find stretches of speech, extracts a speaker embedding from each stretch, and clusters the embeddings so that segments from the same voice land together; newer end-to-end models map straight from audio to a per-frame speaker label.

None of these needs the decoder or the alphabet of a recognizer; each is a classifier or a clustering laid on top of the same spectral features, which is why the front end of this lesson is the foundation for far more than transcription.

Speech and the whole arc

Speech closes the course by reusing almost everything in it. The waveform front end is genuinely new — sampling, framing, the DFT, the mel filterbank — but once the signal is a sequence of spectrogram frames, ASR is the encoder-decoder with attention built for translation, trained with the cross-entropy loss of a language model, rescored by a larger one. CTC is a leaner variant of the same seq2seq approach. Evaluation is the edit distance from the very first module. TTS is the whole thing inverted, its front end the same text normalization the course opened with.

From what NLP is through classification, vector semantics, sequence models, and the transformer, one idea kept recurring: represent language as sequences of vectors, and learn a function that transduces one sequence into another. Speech extends that reach from text down to sound and back — and it does so with the machinery already in hand.