--- title: Automatic Speech Recognition module: Speech moduleNumber: 8 lessonNumber: 3 order: 803 summary: > Speech recognition maps an acoustic waveform to a string of words, and once the waveform is turned into a sequence of log-mel spectrogram frames the problem is the same sequence-to-sequence transduction the rest of the course already solved. This first part builds the feature front end (framing, the DFT, the mel filterbank, the log), then the modern architectures: the attention-based encoder-decoder, the CTC alignment trick that collapses repeated and blank frames, and RNN-T for streaming. Training-data advances, evaluation, TTS, and the other speech tasks come next. topics: [Speech] sources: - book: Jurafsky ref: "Ch. 26 — §26.1 The Automatic Speech Recognition Task" - book: Jurafsky ref: "§26.2 Feature Extraction for ASR: Log Mel Spectrum; §26.3 Speech Recognition Architecture" - book: Jurafsky ref: "§26.4 CTC and Streaming Models (RNN-T)" --- A microphone records air pressure over time: a **waveform**, a single scalar sampled tens of thousands of times a second. The task of **automatic speech recognition** (ASR) is to map that waveform to the words that produced it — to turn $$ x[1], x[2], \ldots, x[T] \quad\longmapsto\quad \texttt{it's time for lunch} $$ with nothing in between but a learned function.[^jm-task] Stated that way it is the same shape as machine translation: a long input sequence in, a much shorter symbol sequence out. That is not a loose analogy. Modern ASR is built from the exact [encoder-decoder with attention](/natural-language-processing/transformers/transformers-and-attention) this course already assembled for translation; the only genuinely new work is at the two ends — turning sound into a sequence a network can read, and scoring the words that come out. This lesson does that work, and in doing so closes the loop back to the very first module: the metric for ASR is the edit distance from [text normalization](/natural-language-processing/foundations/regex-and-text-normalization). ## Why the task is hard The mapping is many-to-one in a way that defeats any simple template match. The same word never arrives twice as the same waveform. - **Speaker variation.** Vocal-tract length, pitch, accent, and speaking rate all reshape the signal. A recognizer trained on standard-dialect adult speakers degrades sharply on regional or ethnic varieties and on children's speech. - **Coarticulation.** Phones overlap: the articulators are still moving toward the next sound while producing the current one, so the acoustic realization of a phone bleeds into its neighbors — the /t/ in _time_ is not the /t/ in _stop_. - **Channel and noise.** A close-talking headset in a quiet room is easy; a far-field microphone on a noisy street, or a telephone that band-limits everything above 4 kHz, is hard. Reverberation and background speakers make it harder still. - **Segmentation.** There are no spaces in speech. Word and phone boundaries are not marked in the signal; the recognizer has to find them. These pressures show up directly in the error rates. Read speech from clean audio books sits near a $2\%$ word error rate — effectively solved. Conversational telephone speech between strangers runs $5$–$6\%$; between family, $11\%$. Sociolinguistic-interview and multi-speaker dinner-party recordings run from $27\%$ up past $80\%$.[^jm-hard] The gap between the easy and hard ends of that range is the whole research problem. ### Phones and the speech signal The atoms underneath the waveform are **phones**, the individual speech sounds — the vowel [iy] in _beat_, the stop [t], the fricative [s].[^jm-phon] A phone is not visible in the raw pressure trace, but it is visible in how the signal's _energy is distributed across frequency_. Vowels have sharp energy bands (formants) at characteristic frequencies; fricatives spread energy into a noisy high-frequency band; stops show a silence followed by a burst. Plotting energy against both time and frequency gives a **spectrogram**, and the whole front end of an ASR system exists to compute a good one. $$ % caption: A waveform (top) plots amplitude against time; a spectrogram (bottom) % plots energy against time and frequency, and the phones become visible as % distinct patterns of spectral energy. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} % --- waveform panel --- \draw[black] (0,3.0) rectangle (10,4.6); \node[anchor=east, font=\scriptsize] at (-0.15,3.8) {amplitude}; \draw[black] (0,3.8) -- (10,3.8); \draw[acc, thick] (0,3.8) \foreach \i in {1,...,60} { -- ++(0.1667, {0.55*sin(\i*47)*(0.4+0.6*abs(sin(\i*11)))}) }; \node[anchor=south west, font=\scriptsize] at (0.05,4.62) {waveform}; % --- spectrogram panel --- \draw[black] (0,0) rectangle (10,2.4); \node[anchor=east, font=\scriptsize] at (-0.15,1.2) {frequency}; % faux spectral bands per phone region \foreach \x/\lo/\hi in {0.2/0.2/0.7, 1.6/0.9/1.5, 3.4/0.3/0.6, 5.2/1.2/2.0, 7.0/0.4/1.9, 8.6/0.5/1.0} { \foreach \b in {0,1,2,3,4,5} { \pgfmathsetmacro\yy{\lo + \b*(\hi-\lo)/5} \draw[black, line width=0.9pt] (\x,\yy) -- (\x+1.2,\yy); } } \draw[black, dashed] (1.4,0) -- (1.4,2.4); \draw[black, dashed] (3.2,0) -- (3.2,2.4); \draw[black, dashed] (5.0,0) -- (5.0,2.4); \draw[black, dashed] (6.8,0) -- (6.8,2.4); \draw[black, dashed] (8.4,0) -- (8.4,2.4); \node[anchor=south west, font=\scriptsize] at (0.05,2.42) {spectrogram}; \node[font=\scriptsize] at (0.8,-0.35) {iy}; \node[font=\scriptsize] at (2.3,-0.35) {t}; \node[font=\scriptsize] at (4.1,-0.35) {s}; \node[font=\scriptsize] at (5.9,-0.35) {ay}; \node[font=\scriptsize] at (7.7,-0.35) {m}; \draw[<->, black] (0,-0.7) -- (10,-0.7) node[midway, below, font=\scriptsize] {time}; \end{tikzpicture} $$ The linguistic structure is visible in the spectrogram, which is why every recognizer begins by computing one — and why the very first analog step is to sample and digitize the pressure wave at all. ## Feature extraction: the log mel spectrogram The front end converts the raw waveform into a sequence of **acoustic feature vectors**, one per short time slice, each summarizing the signal's spectral energy in that slice. The standard feature is the **log mel spectrum**. Four stages produce it.[^jm-feat] $$ % caption: The feature pipeline: sample and quantize the waveform, cut it into % overlapping frames, take the DFT of each, warp onto the mel filterbank, and take % the log to get one feature vector per frame. \begin{tikzpicture}[>=stealth, font=\footnotesize, stage/.style={draw, minimum width=17mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[stage] (wav) at (0,0) {waveform\\$x[n]$}; \node[stage] (frm) at (2.5,0) {frame\\+ taper}; \node[stage] (dft) at (5.0,0) {DFT\\(FFT)}; \node[stage] (mel) at (7.5,0) {mel\\f\/ilterbank}; \node[stage, draw=acc, text=acc] (log) at (10.0,0) {log}; \draw[->, acc, thick] (wav) -- (frm); \draw[->, acc, thick] (frm) -- (dft); \draw[->, acc, thick] (dft) -- (mel); \draw[->, acc, thick] (mel) -- (log); \node[font=\scriptsize, anchor=north] at (2.5,-0.85) {25 ms / 10 ms}; \node[font=\scriptsize, anchor=north] at (5.0,-0.85) {energy per band}; \node[font=\scriptsize, anchor=north] at (10.0,-0.85) {feature vector}; \end{tikzpicture} $$ **Sampling and quantization.** A microphone converts pressure to an analog voltage; an analog-to-digital converter samples that voltage at a fixed **sampling rate** and rounds each measurement to an integer (**quantization**). By the Nyquist argument you need at least two samples per cycle to capture a frequency, so the highest recoverable frequency — the **Nyquist frequency** — is half the sampling rate. Telephone speech is band-limited below 4 kHz, so 8 kHz sampling suffices; wideband microphone speech uses 16 kHz. The result is the digitized, quantized waveform $x[n]$, indexed by discrete time $n$. **Framing and windowing.** Speech is **non-stationary** — its spectral character changes continuously — but within a short enough slice it is roughly **stationary**. So we cut the signal into overlapping **frames**, typically a 25 ms window taken every 10 ms (a 25 ms window with a 10 ms **stride**, so successive frames overlap by 15 ms). Cutting is multiplication by a window function $w[n]$: $$ y[n] = w[n]\,x[n]. $$ A **rectangular** window ($w[n]=1$ inside, $0$ outside) chops the signal off abruptly at the edges, and those discontinuities create spurious high frequencies in the analysis. The **Hamming** window tapers the ends smoothly toward zero instead: $$ w[n] = 0.54 - 0.46\cos\!\left(\frac{2\pi n}{L}\right), \qquad 0 \le n \le L-1, $$ for a window $L$ samples long. Tapering the edges is worth the mild loss of signal at the frame boundaries. **The discrete Fourier transform.** Each windowed frame is still amplitude-over-time; we need energy-over-frequency. The **discrete Fourier transform** (DFT) extracts, for each of $N$ frequency bands $k$, a complex number $X[k]$ whose magnitude is the energy the frame carries at that frequency: $$ X[k] = \sum_{n=0}^{N-1} x[n]\, e^{-j\frac{2\pi}{N}kn}, $$ with $j$ the imaginary unit (Euler's formula $e^{j\theta} = \cos\theta + j\sin\theta$ is what makes this a rotation through frequency). In practice the DFT is computed by the **fast Fourier transform** (FFT), an $O(N\log N)$ algorithm that requires $N$ to be a power of two. The magnitudes $|X[k]|$ are the frame's spectrum. **The mel filterbank and the log.** Human hearing is not linear in frequency: it resolves low frequencies finely and high frequencies coarsely. The **mel scale** is an auditory frequency axis on which sounds equidistant in perceived pitch are equidistant in mels, and it is a log warp of raw frequency $f$: $$ \text{mel}(f) = 1127 \ln\!\left(1 + \frac{f}{700}\right). $$ We collect the DFT energies not band-by-band but through a bank of **triangular filters** spaced evenly on the mel scale — closely packed at low frequencies, widely spaced at high ones — so the feature has fine resolution exactly where hearing does. $$ % caption: The mel filterbank is a row of triangular filters, densely packed at % low frequency and sparse at high frequency; multiplying the spectrum by each % filter and summing gives one mel-band energy $m_1, m_2, \ldots, m_M$. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \draw[->, black] (0,0) -- (10.4,0) node[right, font=\scriptsize] {frequency}; \draw[->, black] (0,0) -- (0,2.4) node[above, font=\scriptsize] {gain}; % triangular filters: narrow at left, widening to the right (log spacing) \def\centers{0.5,1.0,1.6,2.35,3.3,4.5,6.0,7.9} \foreach \c [count=\i from 1] in {0.5,1.0,1.6,2.35,3.3,4.5,6.0,7.9} { \pgfmathsetmacro\wprev{ifthenelse(\i==1, 0.4, 0)} \pgfmathsetmacro\wl{max(0.28, 0.36*\c)} \draw[acc, thick] (\c-\wl,0) -- (\c,1.9) -- (\c+\wl,0); } \node[anchor=north, font=\scriptsize] at (0.5,-0.12) {$m_1$}; \node[anchor=north, font=\scriptsize] at (1.6,-0.12) {$m_2$}; \node[anchor=north, font=\scriptsize] at (4.5,-0.12) {...}; \node[anchor=north, font=\scriptsize] at (7.9,-0.12) {$m_M$}; \end{tikzpicture} $$ Finally we take the **log** of each mel-band energy. Loudness perception is logarithmic too, and the log compresses the dynamic range so the feature is less sensitive to nuisance gain changes — the speaker leaning toward or away from the microphone. The output of the four stages, for one frame, is the **log mel spectrum** vector; stacked over time it is the log mel spectrogram, and that is what the network reads. > **Definition (Log mel spectrogram).** The sequence of feature vectors produced by > framing the waveform, taking the DFT of each Hamming-windowed frame, projecting the > magnitude spectrum onto a bank of mel-spaced triangular filters, and taking the log of > each filter's energy. Typically an $80$-dimensional vector every $10$ ms. ## The recognition architecture With the waveform reduced to a sequence of spectrogram frames $X = x_1, \ldots, x_t$, ASR is a sequence-to-sequence problem: map those frames to an output string $Y$ of characters (or subword pieces). The standard architecture is the **attention-based encoder-decoder** (AED), also called **listen, attend, and spell** — the same encoder-decoder used for translation, with the source-language embeddings replaced by acoustic frames.[^jm-arch] $$ % caption: The attention-based encoder-decoder for ASR. Feature computation and % subsampling turn the waveform into a short frame sequence X; the encoder maps it % to hidden states H; the decoder attends over H and emits characters one at a % time, each conditioned on the ones before. \begin{tikzpicture}[>=stealth, font=\footnotesize, blk/.style={draw, minimum width=22mm, minimum height=8mm, align=center, font=\scriptsize}, trap/.style={draw, trapezium, trapezium left angle=70, trapezium right angle=70, minimum width=20mm, minimum height=7mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % bottom: waveform to features \node[font=\scriptsize] (wav) at (0,0) {waveform}; \node[blk] (feat) at (0,1.1) {feature computation}; \node[trap] (sub) at (0,2.3) {subsampling}; \node[font=\scriptsize] at (2.9,2.3) {short seq. X}; \node[blk, draw=acc, text=acc] (enc) at (0,3.6) {ENCODER}; \node[blk, draw=acc, text=acc, minimum width=44mm] (dec) at (3.2,5.0) {DECODER}; \draw[->, acc, thick] (wav) -- (feat); \draw[->, acc, thick] (feat) -- (sub); \draw[->, acc, thick] (sub) -- (enc); \draw[->, acc, thick] (enc.east) to[bend left=12] node[left, font=\scriptsize, pos=0.4] {H (atten\/tion)} (dec.west); % decoder output tokens \foreach \c/\x in {i/1.3, t/2.0, '/2.55, t/3.2, i/3.9, m/4.6} { \node[font=\footnotesize\ttfamily] at (\x,6.0) {\c}; \draw[->, black] (\x,5.45) -- (\x,5.85); } \node[font=\scriptsize] at (5.3,6.0) {...}; % decoder input (teacher forcing / prior token) \foreach \c/\x in {$<$s$>$/1.3, i/2.0, t/2.55, '/3.2, t/3.9, i/4.6} { \node[font=\footnotesize\ttfamily] at (\x,4.0) {\c}; \draw[->, black] (\x,4.15) -- (\x,4.55); } \end{tikzpicture} $$ Two properties of speech shape the design. First, the input is enormously longer than the output: a single 2-second word is $200$ frames of $10$ ms each but only a handful of letters. So encoder-decoders for speech insert a **subsampling** stage before the encoder that shortens the frame sequence — the simplest scheme stacks each frame with its two predecessors and keeps every third, cutting the sequence length by three. Second, the alignment between frames and letters is unknown and non-monotonic in duration: we do not know in advance which frames spell which letter. Given the short sequence $X$, the decoder is an ordinary conditional language model over the output alphabet — for English, the letters plus space, punctuation, and an unknown token — factoring the output probability autoregressively: $$ p(y_1, \ldots, y_n \mid X) = \prod_{i=1}^{n} p(y_i \mid y_1, \ldots, y_{i-1}, X). $$ At each step it emits the most probable next character given the frames and the characters so far ($\hat{y}_i = \arg\max_{c} p(c \mid y_1 \ldots y_{i-1}, X)$, or a beam search when a language model is in play), and it is trained with the same cross-entropy loss and teacher forcing as any conditional language model: $$ L_{\text{CE}} = -\sum_{i=1}^{m} \log p(y_i \mid y_1, \ldots, y_{i-1}, X). $$ Because the encoder-decoder is itself a conditional language model, it implicitly learns one over its output text — but speech transcripts are a thin source of language statistics compared to the raw text on the web. So ASR systems usually **rescore** the decoder's $n$-best hypotheses with a large external language model, interpolating the two scores and adding a length term to offset the model's bias toward short strings.[^jm-lm] This is the Whisper-style recipe: an off-the-shelf transformer encoder-decoder, fed log-mel frames, trained end-to-end to output text. ### CTC: aligning frames to letters The encoder-decoder handles the unknown alignment by letting attention sort it out. **CTC** — **Connectionist Temporal Classification** — handles it differently, removing the decoder entirely.[^jm-ctc] The model emits **one letter per input frame**, so the output is exactly as long as the input; a **collapse** then reduces that per-frame letter sequence to the real transcript. The naive collapse — merge runs of identical consecutive letters — almost works but breaks on two cases. Consider inference on the word _dinner_, one letter chosen per frame: $$ % caption: A naive per-frame alignment for "dinner". Collapsing runs of identical % letters merges the true double "n" and gives "diner", and there is no symbol for % frames that correspond to no letter. \begin{tikzpicture}[>=stealth, font=\footnotesize, cell/.style={draw, minimum width=6.2mm, minimum height=6mm, font=\scriptsize}, cellr/.style={draw, minimum width=6.2mm, minimum height=6mm, font=\scriptsize, fill=red!8}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[anchor=east, font=\scriptsize] at (-0.15,1.4) {Y (output)}; \node[anchor=east, font=\scriptsize] at (-0.15,0.4) {A (alignment)}; % alignment cells: d i i n n n n e r r r r r r -> collapse -> d i n e r \foreach \l [count=\i from 0] in {d,i,i,n,n,n,n,e,r,r,r,r,r,r} { \node[cellr] at (\i*0.66,0.4) {\l}; } % collapsed output \foreach \l [count=\i from 0] in {d,i,n,e,r} { \node[cell, fill=acc!8] at (\i*0.66,1.4) {\l}; } \node[red, anchor=west, font=\scriptsize] at (9.7,0.9) {"diner", not "dinner"}; \end{tikzpicture} $$ The double _n_ collapses to a single _n_, and there is no way to mark a frame that spells no letter (silence, or a pause). CTC fixes both with one addition: a special **blank** token, written $\varepsilon$. The model may emit blank at any frame that commits to no letter. The collapse function $B$ then runs in a fixed order — **first merge repeated letters, then delete the blanks** — so a blank placed _between_ two identical letters survives long enough to keep them apart. $$ % caption: The CTC collapse $B$ for "dinner". A blank (drawn as a dash) sits % between the two n's; merging repeats first and deleting blanks second preserves % the double letter that the naive scheme destroyed. \begin{tikzpicture}[>=stealth, font=\footnotesize, cell/.style={draw, minimum width=6.4mm, minimum height=6mm, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[anchor=east, font=\scriptsize] at (-0.2,2.7) {Y (output)}; \node[anchor=east, font=\scriptsize] at (-0.2,1.8) {remove blanks}; \node[anchor=east, font=\scriptsize] at (-0.2,0.9) {merge repeats}; \node[anchor=east, font=\scriptsize] at (-0.2,0.0) {A (alignment)}; % alignment row: d i - n n - n e r r r r - - (dash = blank) \foreach \l [count=\i from 0] in {d,i,-,n,n,-,n,e,r,r,r,r,-,-} { \node[cell, fill=red!7] at (\i*0.68,0.0) {\l}; } % merged repeats: d i - n - n e r - \foreach \l [count=\i from 0] in {d,i,-,n,-,n,e,r,-} { \node[cell, fill=black!5] at (\i*0.68,0.9) {\l}; } % blanks removed: d i n n e r \foreach \l [count=\i from 0] in {d,i,n,n,e,r} { \node[cell, fill=acc!8] at (\i*0.68,1.8) {\l}; } % final output \foreach \l [count=\i from 0] in {d,i,n,n,e,r} { \node[cell, draw=acc, text=acc] at (\i*0.68,2.7) {\l}; } \end{tikzpicture} $$ The collapse is many-to-one: many frame-level alignments $A$ map to the same transcript $Y$, since blanks and repeats can be arranged in many ways. Write $B^{-1}(Y)$ for the set of all alignments that collapse to $Y$. CTC makes a strong conditional-independence assumption — the per-frame outputs are independent given the input — so an alignment's probability is just the product of its per-frame probabilities: $$ P_{\text{CTC}}(A \mid X) = \prod_{t=1}^{T} p(a_t \mid X). $$ Now the architecture is simpler than the encoder-decoder: an encoder produces a hidden state $h_t$ per frame, a softmax over the alphabet-plus-blank reads a letter off each $h_t$, and there is **no decoder** because each output depends only on its own frame. $$ % caption: CTC inference. An encoder emits a hidden state per frame; a per-frame % softmax picks a letter or blank; the collapse B removes blanks and merges % repeats. No decoder is needed because each output reads one frame. \begin{tikzpicture}[>=stealth, font=\footnotesize, blk/.style={draw, minimum width=20mm, minimum height=7mm, align=center, font=\scriptsize}, sm/.style={draw, circle, minimum size=4.5mm, inner sep=0pt, font=\tiny}] \definecolor{acc}{HTML}{2348F2} \node[font=\scriptsize] (wav) at (0,0) {waveform}; \node[blk] (feat) at (0,1.0) {feature comp.}; \node[blk, draw=acc, text=acc, minimum width=42mm] (enc) at (2.2,2.3) {ENCODER}; % per-frame softmaxes \foreach \x in {0.6,1.4,2.2,3.0,3.8} { \node[sm] at (\x,3.3) {}; \draw[->,black] (\x,2.7) -- (\x,3.1); } % letters out \foreach \c/\x in {d/0.6, i/1.4, n/2.2, n/3.0, e/3.8} { \node[font=\footnotesize\ttfamily] at (\x,3.9) {\c}; \draw[->,black] (\x,3.5) -- (\x,3.75); } \node[font=\scriptsize] at (4.5,3.9) {...}; \draw[->, acc, thick] (wav) -- (feat); \draw[->, acc, thick] (feat) -- (enc); \end{tikzpicture} $$ There is one flaw in reading off the single most probable letter per frame: the most probable _alignment_ need not collapse to the most probable _transcript_, because many alignments vote for the same $Y$. The principled score sums over all of them, $$ P_{\text{CTC}}(Y \mid X) = \sum_{A \in B^{-1}(Y)} \prod_{t=1}^{T} p(a_t \mid X), $$ and CTC training minimizes $-\log P_{\text{CTC}}(Y \mid X)$ for the gold $Y$. Naively that sum ranges over exponentially many alignments, so it is computed by dynamic programming — a **forward-backward** recursion, the same one used to train HMMs — that merges alignments sharing a prefix.[^jm-ctc-train] The independence assumption that makes CTC cheap also means it learns no language model of its own, so like the encoder-decoder it is combined with an external one; unlike attention, though, CTC can emit letters left to right as frames arrive, which makes it the natural choice for **streaming** recognition. The two approaches can also be trained together, weighting a CTC loss and an encoder-decoder loss on the same encoder. A three-frame example shows how far best-path and summed decoding can diverge. Suppose the alphabet is $\{a, b, \varepsilon\}$ and a three-frame input has the following per-frame letter probabilities (rows sum to $1$): $$ \begin{array}{c|ccc} & t{=}1 & t{=}2 & t{=}3 \\ \hline a & 0.5 & 0.3 & 0.2 \\ b & 0.3 & 0.3 & 0.4 \\ \varepsilon & 0.2 & 0.4 & 0.4 \end{array} $$ The single most probable **alignment** is the per-frame argmax, $[a, \varepsilon, b]$ with probability $0.5 \times 0.4 \times 0.4 = 0.080$, which collapses to the transcript _ab_. But the transcript _b_ can be spelled many ways, and its total probability sums several alignments. Just three of them: $$ \begin{aligned} [\varepsilon, b, \varepsilon] &: 0.2 \times 0.3 \times 0.4 = 0.024 \\ [b, \varepsilon, \varepsilon] &: 0.3 \times 0.4 \times 0.4 = 0.048 \\ [\varepsilon, \varepsilon, b] &: 0.2 \times 0.4 \times 0.4 = 0.032 \\ [b, b, b] &: 0.3 \times 0.3 \times 0.4 = 0.036 \\ [b, b, \varepsilon] &: 0.3 \times 0.3 \times 0.4 = 0.036 \\ [\varepsilon, b, b] &: 0.2 \times 0.3 \times 0.4 = 0.024 \end{aligned} $$ Those six already sum to $0.200$, more than double the best single alignment's $0.080$. The best _alignment_ ($[a, \varepsilon, b] \to ab$) and the best _transcript_ ($b$, once its alignments are pooled) disagree, and that disagreement is why CTC scores a transcript by the sum, not the max. $$ % caption: Best-path versus summed CTC decoding on the three-frame toy. The single most % probable alignment collapses to "ab" (probability 0.080), but pooling the alignments % that collapse to "b" totals 0.200 -- so the best alignment and the best transcript % disagree. \begin{tikzpicture}[>=stealth, font=\footnotesize, cell/.style={draw, minimum width=8mm, minimum height=6mm, font=\scriptsize}, box/.style={draw, minimum width=34mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % best alignment path \node[anchor=east, font=\scriptsize] at (-0.2,2.2) {best alignment:}; \foreach \l [count=\i from 0] in {a,-,b} { \node[cell, fill=black!5] at (\i*0.9,2.2) {\l}; } \node[box, draw=acc, text=acc] (ab) at (5.0,2.2) {collapses to "ab" = 0.080}; \draw[->, acc] (2.1,2.2) -- (ab.west); % summed transcript \node[anchor=east, font=\scriptsize] at (-0.2,0.6) {sum over "b":}; \node[box] (paths) at (1.6,0.6) {many alignments}; \node[box, draw=red, text=red] (b) at (5.0,0.6) {transcript "b" = 0.200}; \draw[->, red, thick] (paths.east) -- (b.west); \node[red, anchor=west, font=\scriptsize] at (7.0,0.6) {wins}; \end{tikzpicture} $$ > **Definition (CTC blank and collapse).** CTC augments the output alphabet with a **blank** > token $\varepsilon$ and defines a collapse $B$ that first merges runs of identical letters, > then deletes all blanks. The blank lets a frame commit to no letter and, placed between > repeated letters, preserves genuine doublings. Training maximizes the total probability of > all frame-level alignments $A \in B^{-1}(Y)$ that collapse to the gold transcript $Y$. ### RNN-T: putting the language model back CTC's independence assumption is what makes it streamable and also what caps its accuracy: each frame's output ignores the letters already emitted, so the model has no way to prefer _the_ over _teh_ from its own history. The **RNN-Transducer** (RNN-T) keeps CTC's frame-synchronous, left-to-right decoding but repairs the independence gap. It adds a second network — a **prediction network**, an autoregressive model over the output letters emitted so far, exactly a small language model — and a **joint network** that combines the encoder's acoustic state at the current frame with the prediction network's state to score the next output.[^jm-rnnt] The result conditions each output on both the acoustics _and_ the letters before it, unlike CTC, while still consuming frames one at a time, unlike attention. $$ % caption: The RNN-T. An encoder reads frames left to right; a prediction network is a % small language model over the letters emitted so far; a joint network combines the two % to score the next output. This restores output-history conditioning that CTC drops, while % keeping CTC's streaming, frame-by-frame decoding. \begin{tikzpicture}[>=stealth, font=\footnotesize, blk/.style={draw, minimum width=26mm, minimum height=8mm, align=center, font=\scriptsize}, jn/.style={draw=acc, text=acc, thick, minimum width=24mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % encoder (acoustic) \node[blk] (enc) at (0,0) {encoder\\(acoustic, frame t)}; \node[anchor=north, font=\scriptsize, text=black] at (0,-0.75) {reads frames left to right}; % prediction network (language) \node[blk] (pred) at (0,3.0) {prediction network\\(letters so far)}; \node[anchor=south, font=\scriptsize, text=black] at (0,3.75) {small language model}; % joint \node[jn] (joint) at (5.2,1.5) {joint network}; \node[blk] (out) at (9.2,1.5) {next letter\\or blank}; \draw[->, acc, thick] (enc.east) to[out=0,in=200] (joint.west); \draw[->, acc, thick] (pred.east) to[out=0,in=160] (joint.west); \draw[->, acc, thick] (joint) -- (out); % feedback: emitted letter to prediction net \draw[->, black] (out.north) to[out=90,in=0] node[above, font=\scriptsize, pos=0.4] {feed back} (pred.east); \end{tikzpicture} $$ RNN-T is the recognizer behind much on-device dictation, where streaming is required and a server round-trip is not acceptable. It occupies the middle of the design space: more accurate than CTC because it models output history, and streamable unlike the attention decoder, which must see the whole utterance before it starts. ## From transcribing to scoring At this point the recognizer is complete: a waveform goes in, the front end turns it into log-mel frames, and an encoder-decoder or a CTC model turns those frames into text. Two questions remain. How wrong is the output — how do we _measure_ a transcript against a reference? And what happens if we run the whole pipeline backwards to turn text into speech? The next part answers both, and surveys the family of smaller speech tasks that reuse the same front end. This continues in [ASR Evaluation and Speech Applications](/natural-language-processing/speech/asr-evaluation-and-applications). [^jm-task]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), §26.1 — The Automatic Speech Recognition Task: ASR maps an acoustic waveform to a string of words; the task varies along vocabulary size, read versus conversational speech, channel and noise, and speaker or dialect. [^jm-hard]: **Jurafsky & Martin**, §26.1 (Fig. 26.1) — reported word error rates around 2020: ~1.4–2.6% on LibriSpeech read audio books, 5.8% on Switchboard stranger conversations, 11% on CALLHOME, 27% on CORAAL sociolinguistic interviews, and up to 81.3% on distant-microphone dinner-party speech. [^jm-phon]: **Jurafsky & Martin**, Ch. 25 — Phonetics: phones are the individual speech sounds; their acoustic realization appears in the spectrogram as characteristic distributions of energy across frequency (formants for vowels, high-frequency noise for fricatives, a silence-then-burst for stops), and coarticulation blends adjacent phones. [^jm-feat]: **Jurafsky & Martin**, §26.2 — Feature Extraction for ASR: Log Mel Spectrum: sampling and quantization (§26.2.1), windowing into 25 ms frames at a 10 ms stride with a Hamming window (§26.2.2, Eqs. 26.1–26.3), the discrete Fourier transform (§26.2.3, Eqs. 26.4–26.5), and the mel filterbank with a log (§26.2.4, Eq. 26.6, $\text{mel}(f)=1127\ln(1+f/700)$). [^jm-arch]: **Jurafsky & Martin**, §26.3 — Speech Recognition Architecture (Fig. 26.6): the attention-based encoder-decoder (AED / listen, attend, and spell), the subsampling stage that shortens the long acoustic sequence, autoregressive decoding (Eqs. 26.7–26.8), and cross-entropy training with teacher forcing (§26.3.1, Eqs. 26.10–26.12). [^jm-lm]: **Jurafsky & Martin**, §26.3 — Adding a language model: because encoder-decoders trained on speech transcripts learn a weak language model, an $n$-best list is rescored by a large external language model, interpolated with the encoder-decoder score and a length normalization (Eq. 26.9). [^jm-ctc]: **Jurafsky & Martin**, §26.4 — CTC (Connectionist Temporal Classification): emit one label per frame and collapse; the blank token and the collapse function $B$ that merges repeated letters then removes blanks (Figs. 26.7–26.9); the per-frame conditional-independence assumption and best-path inference (Eqs. 26.13–26.14). [^jm-ctc-train]: **Jurafsky & Martin**, §26.4.1–26.4.2 — CTC inference and training: the most probable output sums over all alignments in $B^{-1}(Y)$ (Eq. 26.15), the CTC negative-log-likelihood loss (Eqs. 26.17–26.18) computed by a forward-backward dynamic program, the need for an external language model given CTC's independence assumption, and CTC's suitability for streaming (§26.4.4). [^jm-rnnt]: **Jurafsky & Martin**, §26.4.4 — Streaming Models: the RNN-Transducer (Graves 2012; Graves et al. 2013) improves CTC by adding a prediction network (an autoregressive language model over emitted labels) and a joint network combining it with the encoder's acoustic state, so each output is conditioned on the output history CTC drops, while retaining CTC's streaming, frame-by-frame decoding.