Speech Recognition: Front-Ends and Alignment

Text arrives pre-tokenized; speech does not. A spoken utterance is a pressure wave sampled tens of thousands of times a second, and the transcript it corresponds to is a handful of characters whose boundaries are nowhere marked in the signal. Automatic speech recognition (ASR) is the problem of mapping the long acoustic sequence $x_{1:T}$ to the short label sequence $y_{1:U}$ with $T\gg U$ and no given alignment between them.¹ Everything specific to the field, the spectral front-ends, the CTC loss, the transducer, follows from that one structural fact.

Three numbers define the problem. The input rate is high (tens of thousands of samples a second), the output rate is low (a few characters a second), and the map between them is unknown. A ten-second clip at $16\text{kHz}$ is $160,000$ samples; its transcript is maybe $120$ characters. No dataset tells you that samples $41,200$ through $43,800$ spell the word cat. A supervised frame-classifier would need exactly that annotation, and it does not exist at scale. The three model families in this lesson (CTC, attention seq2seq, and the transducer) are three ways to train through the missing alignment, and the representation learning at the end (wav2vec 2.0, HuBERT, Whisper) is three ways to train when even the transcript is scarce or noisy.

Audio as a sequence

A microphone produces a continuous pressure signal; sampling it at rate $f_s$ yields a discrete waveform $x[n]=x_a(n/f_s)$. The Nyquist theorem fixes the floor on $f_s$: to represent content up to frequency $f_{\max }$ without aliasing, $f_s>2f_{\max }$. Human speech carries information up to roughly $8\text{kHz}$, so telephone audio samples at $8\text{kHz}$ and wideband speech at $16\text{kHz}$.

The consequence is severe length. A ten-second utterance at $16\text{kHz}$ is $160,000$ samples, against a transcript of perhaps a hundred characters: a ratio of more than $1000:1$. A single sample carries almost no linguistic content; what matters is the local frequency content of the signal, how energy is distributed across pitches in a short window, and that is invisible in the raw amplitude. Two utterances of the same word, one high-pitched and one low, share almost no samples in common yet must map to the same characters. Feeding $160,000$ raw scalars to a network is both wasteful (the sequence is too long for attention, which costs $O(T^2)$) and statistically hard (the network must rediscover the Fourier basis from scratch), which is why classical and modern systems alike first transform the waveform into a time-frequency representation.

The transform trades one long axis for two shorter ones. Instead of $160,000$ samples on a single time axis, a spectrogram is roughly $1000$ frames on a time axis by a few hundred bins on a frequency axis. The frame rate ($\approx 100$ frames/s) is now only about $10\times$ the character rate rather than $1000\times$, and each frame is a rich vector describing the energy at every pitch, exactly the cue a phoneme classifier needs.

Reading the table top to bottom is the front-end pipeline: each row is the previous row after one more transform, and each transform trades length or channel count for a more compact, more linguistically salient description. The figure below tracks the tensor shape through every stage for the running ten-second, $16\text{kHz}$ clip.

Spectral front-ends

The core tool is the short-time Fourier transform (STFT): chop the waveform into overlapping frames, window each, and take its discrete Fourier transform. Frame $m$ starts at hop $H\cdot m$ and spans a window $w[\cdot ]$ of length $L$; its spectrum at frequency bin $k$ is

The window $w$ (Hann or Hamming) tapers each frame to zero at its edges, suppressing the spectral leakage a hard cutoff would introduce. Chopping the signal into frames is itself a multiplication by a rectangular window, and in the frequency domain multiplication becomes convolution with the window's spectrum. A rectangle's spectrum is a $\mathrm{sinc}$ with tall side-lobes, so a rectangular frame smears each true tone across many bins. The Hann window $w[n]=\frac{1}{2}(1-\cos \frac{2\pi n}{L-1})$ has a far more compact spectrum, so its convolution keeps each tone concentrated near its own bin. The spectrogram is the magnitude ${\left|X[m,k]\right|}^2$, a two-dimensional image with time $m$ on one axis and frequency $k$ on the other.

The frame count follows from the hop, not the window. With hop $H$ the number of frames is $M=1+\lfloor (N-L)/H\rfloor$. For the running clip, $N=160,000$, $L=400$, $H=160$ give $M=1+\lfloor (160000-400)/160\rfloor =1+997=998$ frames, i.e. the $\approx 100$ frames per second promised above. Real-valued input makes the DFT conjugate-symmetric, so only $L/2+1=201$ of the $L$ bins are unique; a common choice of a $512$-point FFT on the $400$-sample window (zero-padded) yields $257$ bins, the number in the shape figure. The window sees $25\text{ms}$ of audio, long enough to resolve pitch yet short enough that the vocal tract is roughly stationary across it, the compromise the $25/10$ ms defaults encode.

Mel scale and MFCCs

The linear frequency axis of the STFT wastes resolution: human pitch perception is roughly logarithmic, finely resolving low frequencies and coarsely resolving high ones. The mel scale warps hertz to a perceptual axis,

where the constants are chosen so $\mathrm{mel}(700)\approx 781$ and the map is nearly linear below $1\text{kHz}$ and logarithmic above. The warp is what a listener hears: the perceived jump from $200$ to $400\text{Hz}$ (one octave, mel $283\to 509$, a gap of $226$) is close to the jump from $4000$ to $8000\text{Hz}$ (also an octave, mel $2146\to 2840$, a gap of $694$ compressed onto far fewer bins), so equal musical intervals map to comparable mel distances while the raw hertz gap grows tenfold. A bank of $B$ triangular filters, equally spaced in mel, sums STFT energy into $B$ (typically $80$) perceptual bands. The mel spectrogram is the log of those band energies,

where $H_b$ is the $b$-th triangular filter. The filters are narrow and dense at low frequency and broad and sparse at high frequency, the signature of the logarithmic mel warping.

Taking a discrete cosine transform of $M[m,\cdot ]$ and keeping the low-order coefficients gives the MFCCs (mel-frequency cepstral coefficients), a compact, decorrelated descriptor of the spectral envelope that dominated ASR front-ends for decades.²

The alignment problem

After the front-end, the input is a frame sequence $x_{1:T}$ at roughly $100$ frames per second; the target is a label sequence $y_{1:U}$ over an alphabet of characters or word-pieces. Three facts define the difficulty.

A frame-level classifier needs a label per frame, but the data gives only the unaligned transcript. The classical solution was a hidden Markov model with forced alignment; the modern solutions, CTC, attention seq2seq, and the transducer, learn the alignment as a latent variable instead.

Connectionist temporal classification

CTC sidesteps alignment by introducing a special blank symbol and marginalizing over every alignment consistent with the target.³ Let the network emit, at each frame $t$, a distribution over the augmented alphabet $A^{\prime }=A\cup \{\text{blank}\}$. A path (or alignment) $\pi \in A^{\prime T}$ is one label per frame. The collapse map $\mathcal{B}$ turns a path into a label sequence by first merging adjacent duplicates, then deleting blanks.

The blank is what lets a path emit the same label twice in a row (separate the two by a blank) and what absorbs the silence and over-sampling between labels. The probability of a target is the total probability of all paths that collapse to it.

The sum is over exponentially many paths, so the definition is not directly computable. The saving structure is that $\mathcal{B}$ is a left-to-right, monotonic map, which makes the marginal a dynamic program identical in form to the HMM forward algorithm.

The forward-backward recursion

Expand the target with blanks into $y^{\prime }=(\text{blank},y_1,\text{blank},y_2,\dots ,y_U,\text{blank})$ of length $S=2U+1$. The forward variable${\alpha }_t(s)$ is the total probability of all paths that have consumed the first $t$ frames and sit at position $s$ in $y^{\prime }$:

Monotonicity restricts the moves into state $s$ to three: stay at $s$, step from $s-1$, or skip from $s-2$ (the skip is permitted only when $y_s^{\prime }$ is a non-blank label distinct from $y_{s-2}^{\prime }$, so two identical letters cannot be merged).

The recursion fills a $T\times S$ table in $O(TS)=O(TU)$ time. The gradient of $-\log p(y\mid x)$ with respect to the network's frame logits is obtained from a symmetric backward variable ${\beta }_t(s)$, exactly as in the HMM Baum-Welch algorithm. The two variables combine into the posterior occupation ${\gamma }_t(s)={\alpha }_t(s){\beta }_t(s)/p(y\mid x)$, the probability that a path sits at state $s$ on frame $t$. Summing ${\gamma }_t(s)$ over the states that emit a given label $\ell$ gives the soft target the network is pulled toward, and the gradient of the loss with respect to the pre-softmax logit for $\ell$ at frame $t$ is the difference

softmax-output minus soft-alignment target, the same shape as an ordinary cross-entropy gradient with the CTC forward-backward supplying the target.

In practice the recursion runs in log-space with a log-sum-exp to avoid underflow, and the per-frame distribution $P_t$ comes from a softmax over an encoder, an LSTM/GRU stack in the original deep-speech systems⁴ and later a deep CNN or Transformer encoder.⁵

Sequence-to-sequence ASR

An alternative drops the blank and the marginalization entirely and treats ASR as ordinary conditional generation: an encoder reads the frames, and an attention decoder emits labels autoregressively, $p(y\mid x)={\prod }_{u}p(y_u\mid y_{<u},x)$. The canonical instance is Listen, Attend and Spell (LAS).⁶

Because the decoder conditions each label on all previous labels, the model carries its own implicit language model, the structural advantage over CTC. The cost is that the soft attention is unconstrained: nothing enforces the monotonic, left-to-right alignment that speech actually obeys, so early attention models could attend out of order or skip ahead, especially on long utterances. Two pathologies result. Looping: the decoder re-attends a region it already transcribed and repeats a word. Skipping: attention jumps past a span and drops words. Both show up as high word-error rate on long inputs and were mitigated by adding a monotonic-attention penalty or by training with a joint CTC loss whose monotone lattice regularizes the attention alignment.

The third option, the RNN transducer (RNN-T), keeps CTC's blank-driven, monotonic, streaming structure but adds a prediction network over previous labels, so the output distribution depends on emission history while alignment stays strictly left-to-right. Where CTC scores one label per frame, the transducer scores a joint distribution $p(k\mid t,u)$ over the next symbol given both the acoustic step $t$ and the label position $u$, and its lattice is a two-dimensional $T\times U$ grid: emit a blank to advance time, emit a label to advance $u$. The forward-backward marginalization over this grid is the transducer analogue of the CTC recursion, still $O(TU)$, but now the label history $y_{<u}$ conditions every emission through the prediction network. This combines CTC's streaming and monotonicity with the attention decoder's implicit language model, at a heavier training cost (the joint network materializes a $T\times U\times |A^{\prime }|$ tensor). It is the dominant choice for on-device, streaming ASR.

This continues in Self-Supervised Speech Models and Synthesis, which trains the same encoders without transcripts (wav2vec 2.0, HuBERT), scales weak supervision to hundreds of thousands of web hours (Whisper), and runs the whole length mismatch backwards for text-to-speech.

Takeaways

• Audio is a long, high-rate sequence: $16\text{kHz}$ mono is $16,000$ samples per second, roughly $1000\times$ longer than its transcript, and a single sample is linguistically meaningless, so systems first compute a time-frequency front-end.
• The STFT frames and Fourier-transforms the waveform; the mel spectrogram warps frequency to the perceptual mel axis and takes a log; a truncated DCT yields the classical MFCCs. The frame count follows the hop, not the window: $M=1+\lfloor (N-L)/H\rfloor$.
• The defining difficulty is alignment: $T\gg U$, with the frame-to-label map unknown, monotonic, and many-to-one.
• CTC adds a blank symbol and a collapse map $\mathcal{B}$, then marginalizes $p(y\mid x)={\sum }_{\pi \in {\mathcal{B}}^{-1}(y)}{\prod }_{t}p({\pi }_t\mid x)$ via an $O(TU)$ forward-backward recursion (the two-frame example gives $0.84$ by both the recursion and direct path enumeration); it assumes per-frame independence and so leans on an external language model.
• Attention seq2seq (LAS) generates labels autoregressively with an implicit language model but an unconstrained alignment (looping, skipping); the RNN transducer keeps CTC's monotone streaming structure while adding label-history dependence, the dominant on-device choice.