--- title: "Speech Recognition: Front-Ends and Alignment" module: Large Models & Agents moduleNumber: 10 lessonNumber: 5 order: 1005 summary: > Speech is a long, high-rate sequence whose label is short and unaligned, so the whole subject turns on bridging that mismatch. This first part builds the spectral front-ends that compress a waveform into frames (STFT, mel spectrogram, MFCC, with a worked frame-count), derives CTC's marginalization over alignments and its forward-backward recursion with a two-frame numeric example, and contrasts it with attention-based seq2seq (LAS) and the RNN transducer. Self-supervised and weakly-supervised models, and text-to-speech, continue in part two. topics: [Large Models & Agents] sources: - book: Goodfellow ref: "§12.3 — Speech Recognition" - book: Chollet ref: "Ch. 11 — sequence models for non-text modalities" --- Text arrives pre-tokenized; speech does not. A spoken utterance is a pressure wave sampled tens of thousands of times a second, and the transcript it corresponds to is a handful of characters whose boundaries are nowhere marked in the signal. Automatic speech recognition (ASR) is the problem of mapping the long acoustic sequence $x_{1:T}$ to the short label sequence $y_{1:U}$ with $T \gg U$ and no given alignment between them.[^gf-speech] Everything specific to the field, the spectral front-ends, the CTC loss, the transducer, follows from that one structural fact. Three numbers define the problem. The input rate is high (tens of thousands of samples a second), the output rate is low (a few characters a second), and the map between them is unknown. A ten-second clip at $16\,\text{kHz}$ is $160{,}000$ samples; its transcript is maybe $120$ characters. No dataset tells you that samples $41{,}200$ through $43{,}800$ spell the word _cat_. A supervised frame-classifier would need exactly that annotation, and it does not exist at scale. The three model families in this lesson (CTC, attention seq2seq, and the transducer) are three ways to train through the missing alignment, and the representation learning at the end (wav2vec 2.0, HuBERT, Whisper) is three ways to train when even the transcript is scarce or noisy. ## Audio as a sequence A microphone produces a continuous pressure signal; sampling it at rate $f_s$ yields a discrete waveform $x[n] = x_a(n / f_s)$. The Nyquist theorem fixes the floor on $f_s$: to represent content up to frequency $f_{\max}$ without aliasing, $f_s > 2 f_{\max}$. Human speech carries information up to roughly $8\,\text{kHz}$, so telephone audio samples at $8\,\text{kHz}$ and wideband speech at $16\,\text{kHz}$. > **Definition (Waveform).** A waveform is the sequence $x[n]$, $n = 0, \dots, N-1$, > of amplitude samples taken at a fixed rate $f_s$. One second of $16\,\text{kHz}$ > mono audio is $16{,}000$ samples, each a small integer or float. The consequence is severe length. A ten-second utterance at $16\,\text{kHz}$ is $160{,}000$ samples, against a transcript of perhaps a hundred characters: a ratio of more than $1000{:}1$. A single sample carries almost no linguistic content; what matters is the _local frequency content_ of the signal, how energy is distributed across pitches in a short window, and that is invisible in the raw amplitude. Two utterances of the same word, one high-pitched and one low, share almost no samples in common yet must map to the same characters. Feeding $160{,}000$ raw scalars to a network is both wasteful (the sequence is too long for attention, which costs $O(T^2)$) and statistically hard (the network must rediscover the Fourier basis from scratch), which is why classical and modern systems alike first transform the waveform into a time-frequency representation. The transform trades one long axis for two shorter ones. Instead of $160{,}000$ samples on a single time axis, a spectrogram is roughly $1000$ frames on a time axis by a few hundred bins on a frequency axis. The frame rate ($\approx 100$ frames/s) is now only about $10\times$ the character rate rather than $1000\times$, and each frame is a rich vector describing the energy at every pitch, exactly the cue a phoneme classifier needs. | Representation | Rate / size (10 s, 16 kHz) | Captures | Used by | | --- | --- | --- | --- | | Raw waveform | $160{,}000$ samples | amplitude vs. time | wav2vec 2.0 encoder | | STFT magnitude | $\sim 1000$ frames $\times$ 257 bins | linear-frequency energy | spectrogram models | | Mel spectrogram | $\sim 1000$ frames $\times$ 80 bins | perceptual-frequency energy | Whisper, most ASR | | MFCC | $\sim 1000$ frames $\times$ 13 coeffs | decorrelated spectral envelope | classical HMM-GMM | Reading the table top to bottom is the front-end pipeline: each row is the previous row after one more transform, and each transform trades length or channel count for a more compact, more linguistically salient description. The figure below tracks the tensor shape through every stage for the running ten-second, $16\,\text{kHz}$ clip. $$ % caption: Tensor shapes through the front-end for a 10 s, 16 kHz clip. The waveform's % single long time axis (160,000) becomes a short time axis of frames times a % frequency axis, which the mel filterbank then narrows from 257 bins to 80. \begin{tikzpicture}[font=\footnotesize,>=stealth, bx/.style={draw, thick, minimum width=27mm, minimum height=13mm, align=center}, ax/.style={draw=acc, text=acc, thick, minimum width=27mm, minimum height=13mm, align=center}] \definecolor{acc}{HTML}{2348F2} \node[bx] (w) at (0,0) {\texttt{waveform}\\\texttt{160000}}; \node[bx] (s) at (3.6,0) {\texttt{STFT mag}\\\texttt{999 x 257}}; \node[ax] (m) at (7.2,0) {\texttt{log-mel}\\\texttt{999 x 80}}; \node[bx] (c) at (10.8,0) {\texttt{MFCC}\\\texttt{999 x 13}}; \draw[->, black, thick] (w) -- (s) node[midway, above, font=\footnotesize, align=center] {\texttt{frame}\\\texttt{+ DFT}}; \draw[->, black, thick] (s) -- (m) node[midway, above, font=\footnotesize, align=center] {\texttt{mel}\\\texttt{+ log}}; \draw[->, black, thick] (m) -- (c) node[midway, above, font=\footnotesize, align=center] {\texttt{DCT}\\\texttt{truncate}}; \node[black, anchor=north, font=\footnotesize] at (0,-1.0) {\texttt{1 axis}}; \node[black, anchor=north, font=\footnotesize] at (5.4,-1.0) {\texttt{time x freq}}; \node[acc, anchor=north, font=\footnotesize] at (7.2,-1.0) {\texttt{neural default}}; \node[black, anchor=north, font=\footnotesize] at (10.8,-1.0) {\texttt{classical}}; \end{tikzpicture} $$ ## Spectral front-ends The core tool is the **short-time Fourier transform** (STFT): chop the waveform into overlapping frames, window each, and take its discrete Fourier transform. Frame $m$ starts at hop $H \cdot m$ and spans a window $w[\cdot]$ of length $L$; its spectrum at frequency bin $k$ is $$ X[m, k] = \sum_{n=0}^{L-1} w[n]\, x[H m + n]\, e^{-\,i\, 2\pi k n / L}, \qquad k = 0, \dots, L-1 . $$ The window $w$ (Hann or Hamming) tapers each frame to zero at its edges, suppressing the spectral leakage a hard cutoff would introduce. Chopping the signal into frames is itself a multiplication by a rectangular window, and in the frequency domain multiplication becomes convolution with the window's spectrum. A rectangle's spectrum is a $\sinc$ with tall side-lobes, so a rectangular frame smears each true tone across many bins. The Hann window $w[n] = \tfrac{1}{2}\bigl(1 - \cos\tfrac{2\pi n}{L-1}\bigr)$ has a far more compact spectrum, so its convolution keeps each tone concentrated near its own bin. The **spectrogram** is the magnitude $\abs{X[m,k]}^2$, a two-dimensional image with time $m$ on one axis and frequency $k$ on the other. The frame count follows from the hop, not the window. With hop $H$ the number of frames is $M = 1 + \lfloor (N - L)/H \rfloor$. For the running clip, $N = 160{,}000$, $L = 400$, $H = 160$ give $M = 1 + \lfloor (160000 - 400)/160 \rfloor = 1 + 997 = 998$ frames, i.e. the $\approx 100$ frames per second promised above. Real-valued input makes the DFT conjugate-symmetric, so only $L/2 + 1 = 201$ of the $L$ bins are unique; a common choice of a $512$-point FFT on the $400$-sample window (zero-padded) yields $257$ bins, the number in the shape figure. The window sees $25\,\text{ms}$ of audio, long enough to resolve pitch yet short enough that the vocal tract is roughly stationary across it, the compromise the $25/10$ ms defaults encode. > **Definition (Spectrogram).** The spectrogram of $x$ is the matrix > $S[m,k] = \abs{X[m,k]}^2$ of short-time spectral energies. Typical settings are a > $25\,\text{ms}$ window ($L = 400$ at $16\,\text{kHz}$) and a $10\,\text{ms}$ hop > ($H = 160$), giving $100$ frames per second. $$ % caption: The front-end pipeline: a waveform is framed with overlap, each frame is % windowed and Fourier-transformed, and the stacked magnitudes form a spectrogram. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, thick, minimum width=20mm, minimum height=10mm, align=center}] \definecolor{acc}{HTML}{2348F2} % waveform sketch \draw[->, black, thick] (-0.2,0) -- (3.4,0); \draw[acc, very thick] plot[domain=0:3.1, samples=120] (\x, {0.45*sin(6*\x r) + 0.18*sin(23*\x r)}); \node[black, anchor=south] at (1.1,1.25) {\texttt{waveform} $x[n]$}; % framing brackets \draw[acc, thick] (0.3,-0.95) rectangle (1.3,0.95); \draw[acc, thick] (0.9,-1.1) rectangle (1.9,1.1); \node[black, font=\footnotesize, anchor=north] at (1.5,-1.25) {\texttt{overlapping frames}}; % arrow to window \draw[->, black, thick] (3.5,0) -- (4.5,0); \node[box, draw=acc, text=acc] (win) at (5.7,0) {\texttt{window}\\$w[n]$}; \draw[->, black, thick] (win) -- (7.0,0); \node[box, draw=acc, text=acc] (ft) at (8.2,0) {\texttt{DFT}}; \draw[->, black, thick] (ft) -- (9.5,0); % spectrogram grid \begin{scope}[xshift=98mm, yshift=-12mm] \draw[black, thick] (0,0) rectangle (2.4,2.4); \foreach \i in {1,...,5} { \draw[black] (0,\i*0.4) -- (2.4,\i*0.4); \draw[black] (\i*0.4,0) -- (\i*0.4,2.4); } \draw[draw=acc, fill=acc!18, thick] (0.4,0.4) rectangle (0.8,1.2); \draw[draw=acc, fill=acc!18, thick] (1.2,0.8) rectangle (1.6,1.6); \draw[draw=acc, fill=acc!18, thick] (1.6,0.4) rectangle (2.0,0.8); \draw[->, black, thick] (0,2.6) -- (0,3.0) node[left, font=\footnotesize] {\texttt{freq}}; \draw[->, black, thick] (2.6,0) -- (3.0,0) node[below, font=\footnotesize] {\texttt{time}}; \node[black, anchor=north, font=\footnotesize] at (1.2,-0.25) {\texttt{spectrogram}}; \end{scope} \end{tikzpicture} $$ ### Mel scale and MFCCs The linear frequency axis of the STFT wastes resolution: human pitch perception is roughly logarithmic, finely resolving low frequencies and coarsely resolving high ones. The **mel scale** warps hertz to a perceptual axis, $$ \mel(f) = 2595 \, \log_{10}\!\parens{1 + \frac{f}{700}} , $$ where the constants are chosen so $\mel(700) \approx 781$ and the map is nearly linear below $1\,\text{kHz}$ and logarithmic above. The warp is what a listener hears: the perceived jump from $200$ to $400\,\text{Hz}$ (one octave, mel $283 \to 509$, a gap of $226$) is close to the jump from $4000$ to $8000\,\text{Hz}$ (also an octave, mel $2146 \to 2840$, a gap of $694$ compressed onto far fewer bins), so equal musical intervals map to comparable mel distances while the raw hertz gap grows tenfold. A bank of $B$ triangular filters, equally spaced in mel, sums STFT energy into $B$ (typically $80$) perceptual bands. The **mel spectrogram** is the log of those band energies, $$ M[m, b] = \log \sum_{k} H_b[k]\, \abs{X[m,k]}^2 , \qquad b = 1, \dots, B , $$ where $H_b$ is the $b$-th triangular filter. The filters are narrow and dense at low frequency and broad and sparse at high frequency, the signature of the logarithmic mel warping. $$ % caption: A mel filterbank: triangular weights, narrow at low frequency and broad % at high, sum STFT energy into perceptual bands spaced evenly on the mel axis. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1cm, y=1cm] \definecolor{acc}{HTML}{2348F2} \draw[->, black, thick] (-0.1,0) -- (9.4,0) node[right, font=\footnotesize] {\texttt{frequency (Hz)}}; \draw[->, black, thick] (0,-0.1) -- (0,2.2) node[above, font=\footnotesize] {\texttt{weight}}; % triangular filters with increasing width (centers/edges grow) \draw[acc, very thick] (0.0,0) -- (0.7,1.7) -- (1.5,0); \draw[acc, very thick] (0.7,0) -- (1.5,1.7) -- (2.5,0); \draw[acc, very thick] (1.5,0) -- (2.5,1.7) -- (3.8,0); \draw[acc, very thick] (2.5,0) -- (3.8,1.7) -- (5.4,0); \draw[acc, very thick] (3.8,0) -- (5.4,1.7) -- (7.4,0); \draw[acc, very thick] (5.4,0) -- (7.4,1.7) -- (9.0,0); \node[black, font=\footnotesize, anchor=north] at (1.0,-0.2) {\texttt{narrow}}; \node[black, font=\footnotesize, anchor=north] at (7.4,-0.2) {\texttt{broad}}; \end{tikzpicture} $$ Taking a discrete cosine transform of $M[m, \cdot]$ and keeping the low-order coefficients gives the **MFCCs** (mel-frequency cepstral coefficients), a compact, decorrelated descriptor of the spectral envelope that dominated ASR front-ends for decades.[^chollet-spec] | Stage | Operation | Output dim per frame | | --- | --- | --- | | STFT | windowed DFT, magnitude | $L/2 + 1 \approx 257$ | | Mel filterbank | triangular bands on mel axis | $B \approx 80$ | | Log | compress dynamic range | $B$ | | DCT (truncated) | decorrelate, keep low orders | $\approx 13$ (MFCC) | > **Remark (Why log, why DCT).** The log compresses the spectrum's enormous dynamic > range and turns multiplicative channel gain into an additive offset; the DCT > approximates the optimal decorrelating transform for spectral envelopes, so the > retained coefficients are nearly uncorrelated, a convenience for the diagonal-covariance > Gaussians of classical systems. Neural front-ends keep the log-mel features and > drop the DCT, since a network does not need decorrelated inputs. ## The alignment problem After the front-end, the input is a frame sequence $x_{1:T}$ at roughly $100$ frames per second; the target is a label sequence $y_{1:U}$ over an alphabet of characters or word-pieces. Three facts define the difficulty. > **Definition (The ASR alignment problem).** Given frames $x_{1:T}$ and labels > $y_{1:U}$ with $T \gg U$, find a probabilistic model of $p(y_{1:U} \mid x_{1:T})$. > The alignment, which frames produce which label, is **unknown** (not annotated), > **monotonic** (speech does not reorder its transcript), and **many-to-one** (a > phoneme spans many frames). A frame-level classifier needs a label per frame, but the data gives only the unaligned transcript. The classical solution was a hidden Markov model with forced alignment; the modern solutions, CTC, attention seq2seq, and the transducer, learn the alignment as a latent variable instead. ## Connectionist temporal classification **CTC** sidesteps alignment by introducing a special **blank** symbol and marginalizing over every alignment consistent with the target.[^graves-ctc] Let the network emit, at each frame $t$, a distribution over the augmented alphabet $A' = A \cup \{\text{blank}\}$. A **path** (or alignment) $\pi \in A'^{\,T}$ is one label per frame. The **collapse map** $\mathcal{B}$ turns a path into a label sequence by first merging adjacent duplicates, then deleting blanks. > **Definition (Collapse map).** $\mathcal{B} : A'^{\,T} \to A^{\le T}$ removes > repeated consecutive symbols and then all blanks. For example > $\mathcal{B}(\texttt{a\,a\,-\,a\,b}) = \texttt{a\,a\,b}$ and > $\mathcal{B}(\texttt{-\,c\,c\,-\,a\,t}) = \texttt{c\,a\,t}$, writing $\texttt{-}$ > for blank. The blank is what lets a path emit the _same_ label twice in a row (separate the two by a blank) and what absorbs the silence and over-sampling between labels. The probability of a target is the total probability of all paths that collapse to it. > **Definition (CTC objective).** Assuming per-frame conditional independence given > $x$, a path has probability $p(\pi \mid x) = \prod_{t=1}^{T} p(\pi_t \mid x)$, and > $$ > p(y \mid x) = \sum_{\pi \,\in\, \mathcal{B}^{-1}(y)} \prod_{t=1}^{T} p(\pi_t \mid x) . > $$ > Training minimizes $-\log p(y \mid x)$ over the dataset. The sum is over exponentially many paths, so the definition is not directly computable. The saving structure is that $\mathcal{B}$ is a left-to-right, monotonic map, which makes the marginal a dynamic program identical in form to the HMM forward algorithm. $$ % caption: The CTC lattice for target "cat". Each column is a frame; blue dots mark % the blank-augmented label states a monotone path may occupy, with allowed moves. \begin{tikzpicture}[>=stealth, font=\footnotesize, x=1.55cm, y=0.92cm] \definecolor{acc}{HTML}{2348F2} % row labels (blank-augmented: -, c, -, a, -, t, -) \foreach \r/\lab in {0/{blank}, 1/c, 2/{blank}, 3/a, 4/{blank}, 5/t, 6/{blank}} \node[black, anchor=east, font=\footnotesize] at (-0.15, \r) {\texttt{\lab}}; % frame (column) labels \foreach \c in {1,2,3,4,5} \node[black, anchor=north, font=\scriptsize] at (\c, -0.55) {$t_{\c}$}; % nodes \foreach \c in {1,2,3,4,5} \foreach \r in {0,1,2,3,4,5,6} \node[circle, draw=acc, fill=acc!15, thick, inner sep=1.3pt] (n\c-\r) at (\c,\r) {}; % a few monotone transition arrows (right and right-up) on one representative path \draw[->, acc, thick] (n1-1) -- (n2-1); \draw[->, acc, thick] (n2-1) -- (n3-3); \draw[->, acc, thick] (n3-3) -- (n4-3); \draw[->, acc, thick] (n4-3) -- (n5-5); \node[black, anchor=west, font=\footnotesize] at (5.35, 3) {\texttt{one valid path}}; \node[black, anchor=west, font=\footnotesize] at (5.35, 5) {\texttt{collapses to cat}}; \end{tikzpicture} $$ ### The forward-backward recursion Expand the target with blanks into $y' = (\text{blank}, y_1, \text{blank}, y_2, \dots, y_U, \text{blank})$ of length $S = 2U + 1$. The **forward variable** $\alpha_t(s)$ is the total probability of all paths that have consumed the first $t$ frames and sit at position $s$ in $y'$: $$ \alpha_t(s) = \sum_{\pi_{1:t} \,:\, \mathcal{B}(\pi_{1:t}) = y'_{1:s}} \;\prod_{r=1}^{t} p(\pi_r \mid x) . $$ Monotonicity restricts the moves into state $s$ to three: stay at $s$, step from $s-1$, or skip from $s-2$ (the skip is permitted only when $y'_s$ is a non-blank label distinct from $y'_{s-2}$, so two identical letters cannot be merged). > **Theorem (CTC forward recursion).** With $p_t(\ell) = p(\pi_t = \ell \mid x)$, > $$ > \alpha_t(s) = p_t(y'_s)\,\bigl[\alpha_{t-1}(s) + \alpha_{t-1}(s-1) + \mathbb{1}[\,y'_s \neq \text{blank},\; y'_s \neq y'_{s-2}\,]\,\alpha_{t-1}(s-2)\bigr] , > $$ > with $\alpha_1(1) = p_1(\text{blank})$, $\alpha_1(2) = p_1(y_1)$, and $\alpha_1(s) = 0$ > for $s > 2$. The total likelihood is > $p(y \mid x) = \alpha_T(S) + \alpha_T(S-1)$. > **Proof (sketch).** Any path reaching state $s$ at frame $t$ emitted $y'_s$ on > frame $t$, contributing the factor $p_t(y'_s)$; before that it occupied $s$, $s-1$, > or $s-2$ at frame $t-1$, the only predecessors a monotone, at-most-unit step over > the blank-expanded label permits. The skip into $s$ is legal only across a blank, > i.e. when $y'_s$ is a label different from $y'_{s-2}$; otherwise collapsing would > wrongly merge two equal labels. Summing the predecessor masses and multiplying by > the emission gives the recursion; the two accepting states are the final label and > the trailing blank. $\qed$ The recursion fills a $T \times S$ table in $O(TS) = O(TU)$ time. The gradient of $-\log p(y \mid x)$ with respect to the network's frame logits is obtained from a symmetric **backward** variable $\beta_t(s)$, exactly as in the HMM Baum-Welch algorithm. The two variables combine into the posterior occupation $\gamma_t(s) = \alpha_t(s)\beta_t(s) / p(y \mid x)$, the probability that a path sits at state $s$ on frame $t$. Summing $\gamma_t(s)$ over the states that emit a given label $\ell$ gives the soft target the network is pulled toward, and the gradient of the loss with respect to the pre-softmax logit for $\ell$ at frame $t$ is the difference $$ \frac{\partial\,(-\log p(y \mid x))}{\partial\, a_t(\ell)} = p_t(\ell) - \frac{1}{p(y \mid x)} \sum_{s \,:\, y'_s = \ell} \alpha_t(s)\,\beta_t(s) , $$ softmax-output minus soft-alignment target, the same shape as an ordinary cross-entropy gradient with the CTC forward-backward supplying the target. > **Worked example (a two-frame lattice).** Take target $y = \texttt{a}$, so > $y' = (\texttt{-}, \texttt{a}, \texttt{-})$, $S = 3$, over $T = 2$ frames with > emission $p_t(\texttt{a}) = 0.6$, $p_t(\texttt{-}) = 0.4$ at both frames. Seed: > $\alpha_1(1) = p_1(\texttt{-}) = 0.4$, $\alpha_1(2) = p_1(\texttt{a}) = 0.6$, > $\alpha_1(3) = 0$. Step to $t = 2$: > $\alpha_2(1) = p_2(\texttt{-})\,\alpha_1(1) = 0.4 \cdot 0.4 = 0.16$; > $\alpha_2(2) = p_2(\texttt{a})\,[\alpha_1(2) + \alpha_1(1)] = 0.6 \cdot (0.6 + 0.4) = 0.60$; > $\alpha_2(3) = p_2(\texttt{-})\,[\alpha_1(3) + \alpha_1(2)] = 0.4 \cdot (0 + 0.6) = 0.24$. > The likelihood is $p(y \mid x) = \alpha_2(S) + \alpha_2(S-1) = 0.24 + 0.60 = 0.84$. > Checking directly, the four paths and their collapses are > $\texttt{a a} \to \texttt{a}$ ($0.36$), $\texttt{a -} \to \texttt{a}$ ($0.24$), > $\texttt{- a} \to \texttt{a}$ ($0.24$), $\texttt{- -} \to \varepsilon$ ($0.16$). > Exactly the first three collapse to $\texttt{a}$, and $0.36 + 0.24 + 0.24 = 0.84$. ```algorithm caption: $\textsc{CtcForward}(P, y)$ — marginal likelihood of target $y$ over all alignments $y' \gets (\text{blank}, y_1, \text{blank}, \dots, y_U, \text{blank})$; $S \gets 2U+1$ // blank-expand $\alpha_1(1) \gets P_1(\text{blank})$; $\alpha_1(2) \gets P_1(y_1)$ // seed first frame for $s \gets 3$ to $S$ do $\alpha_1(s) \gets 0$ for $t \gets 2$ to $T$ do for $s \gets 1$ to $S$ do $a \gets \alpha_{t-1}(s) + \alpha_{t-1}(s-1)$ // stay or step if $y'_s \neq \text{blank}$ and $y'_s \neq y'_{s-2}$ then // skip allowed across blank $a \gets a + \alpha_{t-1}(s-2)$ $\alpha_t(s) \gets P_t(y'_s) \cdot a$ // emit current symbol return $\alpha_T(S) + \alpha_T(S-1)$ // accept on final label or trailing blank ``` In practice the recursion runs in log-space with a log-sum-exp to avoid underflow, and the per-frame distribution $P_t$ comes from a softmax over an encoder, an [LSTM/GRU stack](/deep-learning/architectures/recurrent-networks) in the original deep-speech systems[^graves-rnn] and later a deep CNN or [Transformer](/deep-learning/architectures/the-transformer-architecture) encoder.[^amodei-ds2] > **Remark (The conditional-independence cost).** CTC assumes > $p(\pi \mid x) = \prod_t p(\pi_t \mid x)$: the per-frame outputs are independent > given the acoustics, with no dependence on previously emitted labels. The model has > no internal language model, so CTC systems lean on an external one at decode time. > The transducer and attention models below break this independence. ## Sequence-to-sequence ASR An alternative drops the blank and the marginalization entirely and treats ASR as ordinary conditional generation: an encoder reads the frames, and an attention decoder emits labels autoregressively, $p(y \mid x) = \prod_{u} p(y_u \mid y_{ **Definition (Listen, Attend and Spell).** A pyramidal BiLSTM **listener** encodes > $x_{1:T}$ into a shorter sequence of states $h_{1:T'}$ (each BiLSTM layer halves the > time resolution); an attention-equipped LSTM **speller** generates characters, > attending over $h$ at every output step. Training maximizes the log-likelihood of > the reference transcript. Because the decoder conditions each label on all previous labels, the model carries its own implicit language model, the structural advantage over CTC. The cost is that the soft attention is unconstrained: nothing enforces the monotonic, left-to-right alignment that speech actually obeys, so early attention models could attend out of order or skip ahead, especially on long utterances. Two pathologies result. **Looping**: the decoder re-attends a region it already transcribed and repeats a word. **Skipping**: attention jumps past a span and drops words. Both show up as high word-error rate on long inputs and were mitigated by adding a monotonic-attention penalty or by training with a joint CTC loss whose monotone lattice regularizes the attention alignment. $$ % caption: Attention seq2seq ASR dataflow. The encoder turns T mel frames into T' % states (pyramid or strided down-sampling); at each step the decoder attends over % all encoder states and emits the next label from its own history. \begin{tikzpicture}[font=\footnotesize,>=stealth, bx/.style={draw, thick, minimum width=24mm, minimum height=11mm, align=center}, ax/.style={draw=acc, text=acc, thick, minimum width=24mm, minimum height=11mm, align=center}, st/.style={draw, thick, minimum width=6mm, minimum height=6mm}] \definecolor{acc}{HTML}{2348F2} % input \node[bx] (in) at (0,0) {\texttt{mel frames}\\\texttt{T x 80}}; % encoder \node[bx] (enc) at (0,1.8) {\texttt{encoder}\\\texttt{down x4}}; \draw[->, black, thick] (in) -- (enc); % encoder states h \foreach \i in {0,1,2,3} \node[st, draw=acc, fill=acc!12] (h\i) at (\i*0.85+2.9,1.8) {}; \node[acc, anchor=west, font=\footnotesize] at (2.6,2.7) {\texttt{encoder states}}; \draw[->, black, thick] (enc) -- (h0); % attention box \node[ax] (att) at (5.3,-0.1) {\texttt{attention}}; \foreach \i in {0,1,2,3} \draw[->, acc, thick] (h\i) -- (att); % decoder \node[bx] (dec) at (8.7,-0.1) {\texttt{decoder}}; \draw[->, black, thick] (att) -- (dec) node[midway, above, font=\footnotesize] {\texttt{context}}; % emitted labels \node[st, draw, minimum width=8mm] (y1) at (8.1,-2.0) {\texttt{c}}; \node[st, draw, minimum width=8mm] (y2) at (9.3,-2.0) {\texttt{a}}; \node[st, draw, minimum width=8mm] (y3) at (10.5,-2.0) {\texttt{t}}; \draw[->, black, thick] (dec) -- (y1); \draw[->, black, thick] (y1) -- (y2); \draw[->, black, thick] (y2) -- (y3); \node[black, anchor=west, font=\footnotesize] at (11.0,-2.0) {\texttt{labels}}; \end{tikzpicture} $$ The third option, the **RNN transducer** (RNN-T), keeps CTC's blank-driven, monotonic, streaming structure but adds a prediction network over previous labels, so the output distribution depends on emission history while alignment stays strictly left-to-right. Where CTC scores one label per frame, the transducer scores a joint distribution $p(k \mid t, u)$ over the next symbol given both the acoustic step $t$ and the label position $u$, and its lattice is a two-dimensional $T \times U$ grid: emit a blank to advance time, emit a label to advance $u$. The forward-backward marginalization over this grid is the transducer analogue of the CTC recursion, still $O(TU)$, but now the label history $y_{