Automatic Speech Recognition

A microphone records air pressure over time: a waveform, a single scalar sampled tens of thousands of times a second. The task of automatic speech recognition (ASR) is to map that waveform to the words that produced it — to turn

with nothing in between but a learned function.¹ Stated that way it is the same shape as machine translation: a long input sequence in, a much shorter symbol sequence out. That is not a loose analogy. Modern ASR is built from the exact encoder-decoder with attention this course already assembled for translation; the only genuinely new work is at the two ends — turning sound into a sequence a network can read, and scoring the words that come out. This lesson does that work, and in doing so closes the loop back to the very first module: the metric for ASR is the edit distance from text normalization.

Why the task is hard

The mapping is many-to-one in a way that defeats any simple template match. The same word never arrives twice as the same waveform.

• Speaker variation. Vocal-tract length, pitch, accent, and speaking rate all reshape the signal. A recognizer trained on standard-dialect adult speakers degrades sharply on regional or ethnic varieties and on children's speech.
• Coarticulation. Phones overlap: the articulators are still moving toward the next sound while producing the current one, so the acoustic realization of a phone bleeds into its neighbors — the /t/ in time is not the /t/ in stop.
• Channel and noise. A close-talking headset in a quiet room is easy; a far-field microphone on a noisy street, or a telephone that band-limits everything above 4 kHz, is hard. Reverberation and background speakers make it harder still.
• Segmentation. There are no spaces in speech. Word and phone boundaries are not marked in the signal; the recognizer has to find them.

These pressures show up directly in the error rates. Read speech from clean audio books sits near a $2\%$ word error rate — effectively solved. Conversational telephone speech between strangers runs $5$–$6\%$; between family, $11\%$. Sociolinguistic-interview and multi-speaker dinner-party recordings run from $27\%$ up past $80\%$.² The gap between the easy and hard ends of that range is the whole research problem.

Phones and the speech signal

The atoms underneath the waveform are phones, the individual speech sounds — the vowel iy in beat, the stop t, the fricative s.³ A phone is not visible in the raw pressure trace, but it is visible in how the signal's energy is distributed across frequency. Vowels have sharp energy bands (formants) at characteristic frequencies; fricatives spread energy into a noisy high-frequency band; stops show a silence followed by a burst. Plotting energy against both time and frequency gives a spectrogram, and the whole front end of an ASR system exists to compute a good one.

The linguistic structure is visible in the spectrogram, which is why every recognizer begins by computing one — and why the very first analog step is to sample and digitize the pressure wave at all.

Feature extraction: the log mel spectrogram

The front end converts the raw waveform into a sequence of acoustic feature vectors, one per short time slice, each summarizing the signal's spectral energy in that slice. The standard feature is the log mel spectrum. Four stages produce it.⁴

Sampling and quantization. A microphone converts pressure to an analog voltage; an analog-to-digital converter samples that voltage at a fixed sampling rate and rounds each measurement to an integer (quantization). By the Nyquist argument you need at least two samples per cycle to capture a frequency, so the highest recoverable frequency — the Nyquist frequency — is half the sampling rate. Telephone speech is band-limited below 4 kHz, so 8 kHz sampling suffices; wideband microphone speech uses 16 kHz. The result is the digitized, quantized waveform $x[n]$, indexed by discrete time $n$.

Framing and windowing. Speech is non-stationary — its spectral character changes continuously — but within a short enough slice it is roughly stationary. So we cut the signal into overlapping frames, typically a 25 ms window taken every 10 ms (a 25 ms window with a 10 ms stride, so successive frames overlap by 15 ms). Cutting is multiplication by a window function $w[n]$:

A rectangular window ($w[n]=1$ inside, $0$ outside) chops the signal off abruptly at the edges, and those discontinuities create spurious high frequencies in the analysis. The Hamming window tapers the ends smoothly toward zero instead:

for a window $L$ samples long. Tapering the edges is worth the mild loss of signal at the frame boundaries.

The discrete Fourier transform. Each windowed frame is still amplitude-over-time; we need energy-over-frequency. The discrete Fourier transform (DFT) extracts, for each of $N$ frequency bands $k$, a complex number $X[k]$ whose magnitude is the energy the frame carries at that frequency:

with $j$ the imaginary unit (Euler's formula $e^{j\theta }=\cos \theta +j\sin \theta$ is what makes this a rotation through frequency). In practice the DFT is computed by the fast Fourier transform (FFT), an $O(N\log N)$ algorithm that requires $N$ to be a power of two. The magnitudes $|X[k]|$ are the frame's spectrum.

The mel filterbank and the log. Human hearing is not linear in frequency: it resolves low frequencies finely and high frequencies coarsely. The mel scale is an auditory frequency axis on which sounds equidistant in perceived pitch are equidistant in mels, and it is a log warp of raw frequency $f$:

We collect the DFT energies not band-by-band but through a bank of triangular filters spaced evenly on the mel scale — closely packed at low frequencies, widely spaced at high ones — so the feature has fine resolution exactly where hearing does.

Finally we take the log of each mel-band energy. Loudness perception is logarithmic too, and the log compresses the dynamic range so the feature is less sensitive to nuisance gain changes — the speaker leaning toward or away from the microphone. The output of the four stages, for one frame, is the log mel spectrum vector; stacked over time it is the log mel spectrogram, and that is what the network reads.

The recognition architecture

With the waveform reduced to a sequence of spectrogram frames $X=x_1,\dots ,x_t$, ASR is a sequence-to-sequence problem: map those frames to an output string $Y$ of characters (or subword pieces). The standard architecture is the attention-based encoder-decoder (AED), also called listen, attend, and spell — the same encoder-decoder used for translation, with the source-language embeddings replaced by acoustic frames.⁵

Two properties of speech shape the design. First, the input is enormously longer than the output: a single 2-second word is $200$ frames of $10$ ms each but only a handful of letters. So encoder-decoders for speech insert a subsampling stage before the encoder that shortens the frame sequence — the simplest scheme stacks each frame with its two predecessors and keeps every third, cutting the sequence length by three. Second, the alignment between frames and letters is unknown and non-monotonic in duration: we do not know in advance which frames spell which letter.

Given the short sequence $X$, the decoder is an ordinary conditional language model over the output alphabet — for English, the letters plus space, punctuation, and an unknown token — factoring the output probability autoregressively:

At each step it emits the most probable next character given the frames and the characters so far (${\hat{y}}_i=\arg {\max }_{c}p(c\mid y_1\dots y_{i-1},X)$, or a beam search when a language model is in play), and it is trained with the same cross-entropy loss and teacher forcing as any conditional language model:

Because the encoder-decoder is itself a conditional language model, it implicitly learns one over its output text — but speech transcripts are a thin source of language statistics compared to the raw text on the web. So ASR systems usually rescore the decoder's $n$-best hypotheses with a large external language model, interpolating the two scores and adding a length term to offset the model's bias toward short strings.⁶ This is the Whisper-style recipe: an off-the-shelf transformer encoder-decoder, fed log-mel frames, trained end-to-end to output text.

CTC: aligning frames to letters

The encoder-decoder handles the unknown alignment by letting attention sort it out. CTC — Connectionist Temporal Classification — handles it differently, removing the decoder entirely.⁷ The model emits one letter per input frame, so the output is exactly as long as the input; a collapse then reduces that per-frame letter sequence to the real transcript.

The naive collapse — merge runs of identical consecutive letters — almost works but breaks on two cases. Consider inference on the word dinner, one letter chosen per frame:

The double n collapses to a single n, and there is no way to mark a frame that spells no letter (silence, or a pause). CTC fixes both with one addition: a special blank token, written $\epsilon$. The model may emit blank at any frame that commits to no letter. The collapse function $B$ then runs in a fixed order — first merge repeated letters, then delete the blanks — so a blank placed between two identical letters survives long enough to keep them apart.

The collapse is many-to-one: many frame-level alignments $A$ map to the same transcript $Y$, since blanks and repeats can be arranged in many ways. Write $B^{-1}(Y)$ for the set of all alignments that collapse to $Y$. CTC makes a strong conditional-independence assumption — the per-frame outputs are independent given the input — so an alignment's probability is just the product of its per-frame probabilities:

Now the architecture is simpler than the encoder-decoder: an encoder produces a hidden state $h_t$ per frame, a softmax over the alphabet-plus-blank reads a letter off each $h_t$, and there is no decoder because each output depends only on its own frame.

There is one flaw in reading off the single most probable letter per frame: the most probable alignment need not collapse to the most probable transcript, because many alignments vote for the same $Y$. The principled score sums over all of them,

and CTC training minimizes $-\log P_{\text{CTC}}(Y\mid X)$ for the gold $Y$. Naively that sum ranges over exponentially many alignments, so it is computed by dynamic programming — a forward-backward recursion, the same one used to train HMMs — that merges alignments sharing a prefix.⁸ The independence assumption that makes CTC cheap also means it learns no language model of its own, so like the encoder-decoder it is combined with an external one; unlike attention, though, CTC can emit letters left to right as frames arrive, which makes it the natural choice for streaming recognition. The two approaches can also be trained together, weighting a CTC loss and an encoder-decoder loss on the same encoder.

A three-frame example shows how far best-path and summed decoding can diverge. Suppose the alphabet is $\{a,b,\epsilon \}$ and a three-frame input has the following per-frame letter probabilities (rows sum to $1$):

The single most probable alignment is the per-frame argmax, $[a,\epsilon ,b]$ with probability $0.5\times 0.4\times 0.4=0.080$, which collapses to the transcript ab. But the transcript b can be spelled many ways, and its total probability sums several alignments. Just three of them:

Those six already sum to $0.200$, more than double the best single alignment's $0.080$. The best alignment ($[a,\epsilon ,b]\to ab$) and the best transcript ($b$, once its alignments are pooled) disagree, and that disagreement is why CTC scores a transcript by the sum, not the max.

RNN-T: putting the language model back

CTC's independence assumption is what makes it streamable and also what caps its accuracy: each frame's output ignores the letters already emitted, so the model has no way to prefer the over teh from its own history. The RNN-Transducer (RNN-T) keeps CTC's frame-synchronous, left-to-right decoding but repairs the independence gap. It adds a second network — a prediction network, an autoregressive model over the output letters emitted so far, exactly a small language model — and a joint network that combines the encoder's acoustic state at the current frame with the prediction network's state to score the next output.⁹ The result conditions each output on both the acoustics and the letters before it, unlike CTC, while still consuming frames one at a time, unlike attention.

RNN-T is the recognizer behind much on-device dictation, where streaming is required and a server round-trip is not acceptable. It occupies the middle of the design space: more accurate than CTC because it models output history, and streamable unlike the attention decoder, which must see the whole utterance before it starts.

From transcribing to scoring

At this point the recognizer is complete: a waveform goes in, the front end turns it into log-mel frames, and an encoder-decoder or a CTC model turns those frames into text. Two questions remain. How wrong is the output — how do we measure a transcript against a reference? And what happens if we run the whole pipeline backwards to turn text into speech? The next part answers both, and surveys the family of smaller speech tasks that reuse the same front end. This continues in ASR Evaluation and Speech Applications.