--- title: Acoustic Phonetics module: Speech moduleNumber: 8 lessonNumber: 2 order: 802 summary: > Articulation is the cause; the acoustic signal is the effect, and the effect is all a microphone ever gets. This part follows the sound out of the mouth: waves, sampling and the Nyquist limit, F0 and the pitch track, the mel scale, the spectrum and Fourier analysis, the source-filter model that explains why each vowel carries its own formants, and the spectrogram the log-mel front end of every ASR system sits directly on top of — closing with neural TTS, wav2vec, HuBERT, and Whisper, where phonetics went in neural speech. topics: [Speech] sources: - book: Jurafsky ref: "§25.4 Acoustic Phonetics and Signals; §25.5 Phonetic Resources" --- This builds on [Phonetics](/natural-language-processing/speech/phonetics), which covered the articulatory and prosodic side of speech — how the vocal tract shapes airflow into consonants and vowels, and how stress and tune ride on top. That was the _cause_. This part is the _effect_: the pressure wave those movements produce, and everything a recognizer can measure in it. ## Acoustic phonetics and signals Articulation is the cause; the **acoustic signal** is the effect, and the effect is all a microphone or an ear ever gets.[^jm-acoustic] Acoustic phonetics is the study of that signal — and it is the immediate input to speech recognition. ### Waves, frequency, and amplitude The building block is the sine wave, $y = A \sin(2\pi f t)$, with two parameters. The **frequency** $f$ is how many cycles occur per second, measured in **hertz** (Hz); the **amplitude** $A$ is the peak height. The **period** $T = 1/f$ is the time for one cycle. $$ % caption: A pure sine wave. The amplitude A is the peak height; the period T is % the time for one cycle; the frequency f = 1/T counts cycles per second (Hz). % Here f is 4 Hz over the window shown. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \draw[->, black] (0,0) -- (9.6,0) node[anchor=north east, font=\scriptsize, text=black] {time}; \draw[black] (0,-1.5) -- (0,1.6); \node[anchor=south east, font=\scriptsize] at (-0.05,1.5) {A}; \draw[black, dashed] (0,1.2) -- (9.4,1.2); \draw[black, dashed] (0,-1.2) -- (9.4,-1.2); % sine \draw[acc, very thick] (0,0) \foreach \i in {1,...,96} { -- ({\i*0.098}, {1.2*sin(\i*15)}) }; % period bracket between two peaks (peak at 6 deg... use quarter offsets) \draw[<->, black] (0.6,-1.35) -- (3.0,-1.35) node[midway, below, font=\scriptsize] {period T}; \node[anchor=west, font=\scriptsize] at (9.45,1.2) {+A}; \node[anchor=west, font=\scriptsize] at (9.45,-1.2) {-A}; \end{tikzpicture} $$ A **speech waveform** plots air pressure against time. When a speaker's vocal folds and vocal tract push and pull the air, a microphone (or eardrum) records the resulting compression and rarefaction as a wiggling line. A vowel's waveform is not a pure sine, but it is **periodic** — it repeats — and the rate of that repetition carries the pitch. Turning that analog pressure into numbers is **analog-to-digital conversion**, in two steps. **Sampling** measures the amplitude at regular instants; the **sampling rate** is the number of samples per second. To capture a wave you need at least two samples per cycle, so the highest recoverable frequency is half the sampling rate — the **Nyquist frequency**. Most speech energy sits below 10 kHz, so 16 kHz sampling suffices for microphone speech; telephone speech, band-limited to 4 kHz, needs only 8 kHz. **Quantization** then rounds each sampled amplitude to an integer (commonly 16-bit), and the stream is stored in a format like `.wav`. ### Pitch, loudness, and their perceptual scales The rate at which a voiced waveform repeats is its **fundamental frequency**, F0 — the frequency of vocal-fold vibration.[^jm-acoustic] Each period corresponds to one opening of the folds. Plotting F0 over time gives a **pitch track**, whose shape recovers the tune from the prosody section: the pitch track of _"Three o'clock?"_ rises at the end. $$ % caption: A pitch track of the question "Three o'clock?" plotted below its % waveform. Each repetition of the waveform is one glottal cycle; the F0 curve % counts those cycles per second and rises at the end to mark the question. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} % waveform panel \draw[black] (0,2.2) rectangle (9,3.6); \node[anchor=south west, font=\scriptsize] at (0.05,3.62) {waveform}; \draw[acc, thick] (0,2.9) \foreach \i in {1,...,72} { -- ++(0.125, {0.55*sin(\i*52)*(0.35+0.65*abs(sin(\i*7)))}) }; % pitch panel \draw[black] (0,0) rectangle (9,1.8); \node[anchor=south west, font=\scriptsize] at (0.05,1.82) {F0 (pitch track)}; \node[anchor=east, font=\scriptsize] at (-0.1,1.65) {500 Hz}; \node[anchor=east, font=\scriptsize] at (-0.1,0.15) {0 Hz}; % F0 dotted trace: gap in the middle (unvoiced), rise at end \draw[acc, very thick] (0.5,0.75) .. controls (1.6,0.8) and (2.4,0.7) .. (3.0,0.75); \draw[acc, very thick] (5.2,0.7) .. controls (6.4,0.75) and (7.4,1.1) .. (8.6,1.5); \node[font=\scriptsize, text=black, anchor=north] at (4.1,0.6) {(unvoiced gap)}; \node[font=\scriptsize, anchor=north] at (1.6,-0.1) {three}; \node[font=\scriptsize, anchor=north] at (6.8,-0.1) {o'clock}; \end{tikzpicture} $$ Amplitude has its own summaries. Because raw amplitude swings positive and negative, its average is near zero, so we use the **RMS amplitude** — square, average, then square-root: $\sqrt{\frac{1}{N}\sum_i x_i^2}$. The related **power** is the mean square $\frac{1}{N}\sum_i x_i^2$, and **intensity** normalizes power against the auditory threshold pressure $P_0 = 2\times10^{-5}$ Pa and reports it in decibels, $10\log_{10}\!\big(\frac{1}{N P_0}\sum_i x_i^2\big)$. Two perceptual quantities ride on these physical ones. **Pitch** is the perceived correlate of F0, and **loudness** the perceived correlate of power — but neither map is linear. Pitch tracks frequency linearly only between roughly 100 Hz and 1000 Hz; above that, equal pitch steps span ever-larger frequency gaps. The **mel scale** captures this: a mel is a unit of pitch on which perceptually equal intervals are equal numbers of mels, and $$ m = 1127 \, \ln\!\left(1 + \frac{f}{700}\right). $$ This formula is the warping that the [log-mel front end](/natural-language-processing/speech/automatic-speech-recognition) applies to spectrogram frequencies before passing them to the recognizer. ### The spectrum and the frequency domain Reading phones off a raw waveform works only for coarse features — you can spot the regular voicing peaks of a vowel, the near-silence-then-burst of a stop, the noisy hiss of a fricative. Fine phonetic detail lives in the **frequency domain**. The insight of **Fourier analysis** is that any complex wave is a sum of pure sine waves of different frequencies. The **spectrum** of a signal lists those component frequencies and their amplitudes: sum a 10 Hz and a 100 Hz sine, and the spectrum shows exactly two spikes, at 10 Hz and 100 Hz. $$ % caption: A spectrum decomposes a signal into its component frequencies. The % waveform on the left is the sum of a 10 Hz and a 100 Hz sine; its spectrum on % the right is two spikes, one at each component frequency. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} % --- left: summed waveform --- \draw[black] (0,-1.4) rectangle (4.2,1.4); \node[anchor=south, font=\scriptsize] at (2.1,1.42) {waveform (10 Hz + 100 Hz)}; \draw[acc, thick] (0.1,0) \foreach \i in {1,...,80} { -- ({0.1+\i*0.05}, {0.55*sin(\i*18) + 0.45*sin(\i*180)}) }; % --- arrow --- \draw[->, acc, very thick] (4.7,0) -- (5.9,0) node[midway, above, font=\scriptsize, text=black] {Fourier}; % --- right: spectrum --- \draw[->, black] (6.4,-1.4) -- (6.4,1.4) node[anchor=south east, font=\scriptsize, text=black] {amp}; \draw[->, black] (6.4,-1.4) -- (11.0,-1.4); \node[anchor=south east, font=\scriptsize, text=black] at (11.0,-1.35) {freq (Hz)}; \draw[acc, very thick] (7.2,-1.4) -- (7.2,0.9); \draw[acc, very thick] (9.2,-1.4) -- (9.2,0.5); \node[font=\scriptsize, anchor=north] at (7.2,-1.45) {10}; \node[font=\scriptsize, anchor=north] at (9.2,-1.45) {100}; \end{tikzpicture} $$ Run the same decomposition on a vowel and the spectrum is richer: a set of peaks at characteristic frequencies. Those peaks are the phone's acoustic fingerprint — just as burning elements emit light at characteristic wavelengths, phones concentrate energy at characteristic frequencies, and reading them off the spectrum is how both the cochlea and a recognizer identify sounds. ### The source-filter model Why does a vowel's spectrum have peaks where it does? The **source-filter model** separates two contributions.[^jm-acoustic] The **source** is the glottal buzz: vocal-fold vibration at F0, which produces not just F0 but a whole ladder of **harmonics** at integer multiples ($2f$, $3f$, …), each weaker than the last. The **filter** is the vocal tract: a tube whose shape makes some frequencies resonate (amplify) and damps others. Moving the tongue changes the tube's shape, so it changes _which_ harmonics get amplified. $$ % caption: The source-filter model. The glottis emits a harmonic-rich source % (a fundamental plus a ladder of harmonics); the vocal tract acts as a filter % whose resonances amplify some frequencies; the output spectrum is the source % shaped by the filter, with peaks (formants) where the two coincide. \begin{tikzpicture}[>=stealth, font=\footnotesize, panel/.style={draw, minimum width=27mm, minimum height=20mm}] \definecolor{acc}{HTML}{2348F2} % --- source spectrum: falling harmonic ladder --- \node[panel] (src) at (0,0) {}; \foreach \i/\h in {0/1.6, 1/1.3, 2/1.05, 3/0.85, 4/0.68, 5/0.55, 6/0.44} { \draw[acc, thick] ({-1.15+\i*0.33},-0.95) -- ({-1.15+\i*0.33},{-0.95+\h}); } \node[font=\scriptsize, anchor=south] at (0,1.05) {source (glottis)}; \node[font=\scriptsize, text=black, anchor=north] at (0,-1.05) {F0 + harmonics}; % times \node[font=\small] at (1.9,0) {x}; % --- filter: resonance envelope --- \node[panel] (fil) at (3.8,0) {}; \draw[acc, thick] (2.65,-0.8) .. controls (3.1,1.0) and (3.3,-0.2) .. (3.7,0.7) .. controls (4.0,1.3) and (4.4,-0.4) .. (4.95,-0.7); \node[font=\scriptsize, anchor=south] at (3.8,1.05) {f\/ilter (vocal tract)}; \node[font=\scriptsize, text=black, anchor=north] at (3.8,-1.05) {resonances}; % equals \node[font=\small] at (5.7,0) {=}; % --- output: shaped spectrum with formant peaks --- \node[panel] (out) at (7.6,0) {}; \foreach \i/\h in {0/0.7, 1/1.5, 2/0.9, 3/0.55, 4/1.2, 5/0.5, 6/0.35} { \draw[acc, thick] ({6.45+\i*0.33},-0.95) -- ({6.45+\i*0.33},{-0.95+\h}); } \draw[black, thick] (6.45,-0.25) .. controls (6.9,0.9) and (7.1,-0.1) .. (7.5,0.55) .. controls (7.8,1.1) and (8.2,-0.3) .. (8.7,-0.6); \node[font=\scriptsize, anchor=south] at (7.6,1.05) {output spectrum}; \node[font=\scriptsize, text=black, anchor=north] at (7.6,-1.05) {peaks = formants}; \end{tikzpicture} $$ The amplified bands are **formants**: frequency regions where the vocal tract resonates. Each vowel, produced with a different tract shape, has a different pattern of formants — and that is why the vowel space earlier "correlates more with acoustics than with tongue position." The formants _are_ the acoustic identity of the vowel. > **Definition (Formant).** A frequency band amplified by the resonance of the > vocal tract, appearing as a peak in the spectrum and a dark bar in the > spectrogram. The lowest two, F1 and F2, largely determine vowel identity. ### Formants, F0, and the spectrogram A spectrum is a snapshot — the frequency content at one instant. Speech changes over time, so we need a movie: the **spectrogram**, which plots time on the x-axis, frequency on the y-axis, and amplitude as darkness. A spectrogram is a stack of spectra, each a thin vertical time-slice, placed side by side. This is the central representation, and the direct input to a recognizer. $$ % caption: The centerpiece. Top: the waveform of a vowel, air pressure over time, % periodic at the fundamental F0. Middle: its spectrogram, time by frequency with % darkness for energy. Bottom: the same spectrogram read as formants F1, F2, F3 — % the horizontal dark bands that identify the vowel. F0 (waveform periodicity) and % the formants (spectral bands) are independent: pitch versus vowel color. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % ---- waveform (top) ---- \draw[black] (0,5.0) rectangle (10,6.4); \node[anchor=south west, font=\scriptsize] at (0.05,6.42) {waveform: air pressure vs time}; \node[anchor=east, font=\scriptsize] at (-0.15,5.7) {amp}; \draw[acc, thick] (0,5.7) \foreach \i in {1,...,80} { -- ++(0.125, {0.55*sin(\i*40)*(0.4+0.6*abs(sin(\i*5)))}) }; \draw[<->, red] (0.9,4.7) -- (1.8,4.7); \node[anchor=west, font=\scriptsize, text=red] at (1.9,4.7) {1/F0 (one period)}; % ---- spectrogram (middle) ---- \draw[black] (0,2.1) rectangle (10,4.1); \node[anchor=south west, font=\scriptsize] at (0.05,4.12) {spectrogram: time x frequency, dark = energy}; \node[anchor=east, font=\scriptsize] at (-0.15,3.1) {freq}; % formant bands (dark horizontal-ish bars) with slight wobble \foreach \yc/\lab in {2.35/F1, 3.05/F2, 3.65/F3} { \foreach \k in {0,1,2,3,4,5,6,7,8,9} { \pgfmathsetmacro\xx{0.3+\k*0.98} \pgfmathsetmacro\yy{\yc + 0.05*sin(\k*60)} \draw[black!70, line width=2.2pt] (\xx,\yy) -- (\xx+0.7,\yy); } } % faint vertical striations (glottal pulses) \foreach \k in {0,...,26} { \draw[black, line width=0.4pt] ({0.2+\k*0.36},2.2) -- ({0.2+\k*0.36},4.0); } % ---- formant reading (bottom) ---- \draw[black] (0,0) rectangle (10,1.6); \node[anchor=east, font=\scriptsize] at (-0.15,0.8) {freq}; \draw[acc, very thick] (0.3,0.35) -- (9.7,0.35); \node[acc, font=\scriptsize, anchor=west] at (9.75,0.35) {F1}; \draw[acc, very thick] (0.3,0.85) -- (9.7,0.85); \node[acc, font=\scriptsize, anchor=west] at (9.75,0.85) {F2}; \draw[acc, very thick] (0.3,1.25) -- (9.7,1.25); \node[acc, font=\scriptsize, anchor=west] at (9.75,1.25) {F3}; \node[anchor=south west, font=\scriptsize] at (0.05,1.66) {formant reading: horizontal bands identify the vowel}; % time axis at very bottom \draw[<->, black] (0,-0.35) -- (10,-0.35) node[midway, below, font=\scriptsize] {time}; \end{tikzpicture} $$ Two independent quantities live in this picture, and confusing them is the single most common error. **F0** is the _periodicity of the waveform_ — the rate the vocal folds vibrate, the source, heard as pitch. The **formants** are the _horizontal bands in the spectrogram_ — the resonances of the tract, the filter, heard as vowel quality. You can raise F0 (say the vowel higher) while holding the formants fixed (same vowel), or hold F0 and change the formants (same pitch, different vowel). F1 and F2 do most of the work of telling vowels apart: `[iy]` has a low F1 (~540 Hz) and high F2 (~2580 Hz), while `[ae]` has a higher F1 (~930 Hz) and lower F2 (~1860 Hz). Formants also help identify nasals and the liquids `[l]` and `[r]`. Higher formants (F3 and up) depend more on the speaker's vocal-tract size than on the specific vowel. #### A worked formant reading Two numbers per vowel are enough to separate the corners of the vowel space, and they invert the articulatory picture. F1 falls as the tongue rises (high vowels have low F1) and F2 rises as the tongue moves forward (front vowels have high F2). Plotting a vowel at its $(F2, F1)$ coordinates — F2 on a reversed horizontal axis, F1 on a reversed vertical axis — reproduces the tongue- position quadrilateral from articulation, which is why the vowel space "correlates with acoustics." Take three measured English vowels from a canonical formant table: `[iy]` (_beet_) at F1 $270$ Hz, F2 $2290$ Hz; `[aa]` (_bot_) at F1 $730$ Hz, F2 $1090$ Hz; `[uw]` (_boot_) at F1 $300$ Hz, F2 $870$ Hz. `[iy]` lands high and front (low F1, high F2), `[uw]` high and back (low F1, low F2), `[aa]` low and central- back (high F1, mid F2) — exactly their tongue positions. $$ % caption: Three vowels plotted in the F1-F2 plane. F2 runs right-to-left (front to % back) and F1 runs top-to-bottom (high to low), so the acoustic plot mirrors the % articulatory vowel quadrilateral. iy is high front, uw is high back, aa is low. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} % axes: x = F2 reversed (2500 left .. 500 right), y = F1 reversed (200 top .. 800 bottom) \draw[->, black] (0,0) -- (0,4.4) node[anchor=south, font=\scriptsize, text=black] {F1 low}; \draw[->, black] (0,0) -- (8.2,0) node[anchor=north east, font=\scriptsize, text=black] {F2 low (back)}; \node[anchor=west, font=\scriptsize] at (0.15,4.0) {F1 200 Hz}; \node[anchor=west, font=\scriptsize] at (0.15,0.35) {F1 800 Hz}; \node[anchor=south west, font=\scriptsize] at (2.7,4.15) {F2 high (front)}; % map: xpos increases as F2 decreases; ypos increases as F1 decreases % iy F1 270 F2 2290 -> high, front (upper left) \fill[acc] (1.0,3.5) circle (2.2pt); \node[font=\scriptsize, text=acc, anchor=south west] at (1.05,3.55) {iy (270, 2290)}; % uw F1 300 F2 870 -> high, back (upper right) \fill[acc] (6.8,3.4) circle (2.2pt); \node[font=\scriptsize, text=acc, anchor=south east] at (6.75,3.45) {uw (300, 870)}; % aa F1 730 F2 1090 -> low, mid-back (lower right) \fill[acc] (5.9,0.6) circle (2.2pt); \node[font=\scriptsize, text=acc, anchor=north] at (5.9,0.5) {aa (730, 1090)}; \end{tikzpicture} $$ The pair $(F1, F2)$ is the coordinate a vowel classifier reads, and the whole point of the mel warping below is to space those frequency measurements the way the ear spaces them before the classifier ever sees them. $$ % caption: F0 versus formants, the two axes of a voiced sound. F0 (the source) is % how fast the folds vibrate and sets pitch; the formants (the filter) are the % tract resonances and set vowel identity. They vary independently. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=32mm, minimum height=16mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box, draw=red, text=red] (f0) at (0,0) {F0 (source)\\vocal-fold rate\\= pitch}; \node[box, draw=acc, text=acc] (fm) at (5.6,0) {formants (f\/ilter)\\tract resonances\\= vowel identity}; \node[font=\scriptsize, text=black, anchor=south] at (0,1.0) {waveform periodicity}; \node[font=\scriptsize, text=black, anchor=south] at (5.6,1.0){spectrogram bands}; \draw[<->, black, thick] (f0) -- (fm) node[midway, above, font=\scriptsize, text=black] {independent}; \end{tikzpicture} $$ ## Phonetic resources Computational phonetics draws on two kinds of resource.[^jm-resources] A **pronunciation dictionary** maps words to phone strings: the open-source **CMU Pronouncing Dictionary** has about 134,000 words with stress marks; CELEX adds syllabification and part-of-speech; UNISYN covers dozens of English dialects. A **phonetically annotated corpus** pairs recorded waveforms with hand-labeled phone strings. The canonical one is **TIMIT** — 6300 read sentences, 10 from each of 630 speakers, each phone hand-labeled and time-aligned to the waveform, yielding a **time-aligned transcription** where every phone carries a start and end time. The Switchboard Transcription Project and the Buckeye corpus of spontaneous speech play the same role for conversational speech. Tools like **Praat** compute the pitch, spectral, and formant analyses this lesson has been describing — and generated many of the figures in the source text. ## Where phonetics went in neural speech The source treats phonetics as the hand analysis a recognizer later mechanizes. Modern systems went further: they either learn the acoustic representation directly from waveforms, or generate the waveform from scratch. Both build on the physics this lesson laid out. **Neural text-to-speech generates the waveform.** Von Kempelen's bellows machine and later concatenative synthesizers stitched recorded units; neural TTS instead predicts the signal. **WaveNet** (van den Oord et al., _WaveNet: A Generative Model for Raw Audio_, 2016) is an autoregressive convolutional network that models the waveform one sample at a time, $P(x_t \mid x_1, \ldots, x_{t-1})$, with dilated causal convolutions to reach far back in time. It produced markedly more natural speech than the concatenative and parametric systems before it, at the cost of slow sample-by-sample generation — which later work (parallel WaveNet, WaveGlow, HiFi-GAN) sped up. **Tacotron 2** (Shen et al., _Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions_, ICASSP 2018) split the task exactly along this lesson's source-filter division: a sequence-to-sequence network first predicts a **mel spectrogram** from text — the same log-mel representation the vowel and formant analysis here produces — and a WaveNet-style vocoder then turns that spectrogram into audio. The mel spectrogram is the interface, which is why a TTS system is cut in two at the acoustic middle of this chapter.[^wavenet][^tacotron] $$ % caption: Neural TTS as this chapter's speech chain run in reverse. Text becomes a % mel spectrogram (the log-mel representation of the acoustic section), and a neural % vocoder turns that spectrogram into a waveform. Recognition runs the same chain % left to right; synthesis runs it right to left. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=25mm, minimum height=12mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[box] (txt) at (0,0) {text}; \node[box, draw=acc, text=acc] (mel) at (3.7,0) {mel\\spectrogram}; \node[box] (wav) at (7.4,0) {waveform}; \draw[->, acc, thick] (txt) -- (mel) node[midway, above, font=\scriptsize] {Tacotron}; \draw[->, acc, thick] (mel) -- (wav) node[midway, above, font=\scriptsize] {vocoder}; \node[font=\scriptsize, text=black, anchor=north] at (3.7,-0.95) {synthesis: text to sound}; \end{tikzpicture} $$ **Self-supervised models learn the representation from raw audio.** The log-mel front end is a fixed, hand-designed transform. **wav2vec 2.0** (Baevski et al., NeurIPS 2020) instead _learns_ a speech representation by masking spans of a latent audio encoding and training the model to pick the correct quantized latent from distractors — a contrastive objective on unlabeled speech, in the spirit of masked language modeling. **HuBERT** (Hsu et al., IEEE/ACM TASLP 2021) trains on masked prediction of cluster labels derived from the audio itself. Both pretrain on thousands of hours of unlabeled speech and then fine-tune on a small labeled set, which is what made high-accuracy recognition possible for low-resource languages. The representations they learn are not hand-specified formant tracks — but probing studies find that the physical categories of this chapter (voicing, place of articulation, formant structure) re-emerge in the learned features, because they are the information the signal actually carries.[^wav2vec][^hubert] **Whisper and weak supervision.** OpenAI's **Whisper** (Radford et al., 2022) trained an encoder-decoder on $680{,}000$ hours of weakly labeled multilingual audio scraped from the web, and showed that scale alone — the log-mel spectrogram front end kept intact — yields robust multilingual recognition without task-specific fine-tuning. The acoustic representation this chapter derives by hand stayed at the input, and the modeling above it got large enough to absorb everything else.[^whisper] [^jm-acoustic]: **Jurafsky & Martin**, §25.4 — Acoustic Phonetics and Signals: waves and sampling, F0 and the pitch track, the spectrum and Fourier analysis, the source-filter model, formants, and the spectrogram; the mel scale for perceived pitch. [^jm-resources]: **Jurafsky & Martin**, §25.5 — Phonetic Resources: pronunciation dictionaries (CMU, CELEX, UNISYN) and phonetically annotated corpora (TIMIT, Switchboard, Buckeye), plus the Praat analysis toolkit. [^wavenet]: **van den Oord et al. (2016)**, _WaveNet: A Generative Model for Raw Audio_ — an autoregressive dilated-convolution network modeling the speech waveform one sample at a time, more natural than concatenative/parametric TTS but slow to sample; later sped up by parallel WaveNet, WaveGlow, and HiFi-GAN vocoders. [^tacotron]: **Shen et al. (2018)**, _Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions_ (Tacotron 2), ICASSP 2018 — a sequence-to-sequence model predicting a mel spectrogram from text, followed by a neural vocoder producing the waveform; the mel spectrogram is the source-filter-style interface between the two stages. [^wav2vec]: **Baevski, Zhou, Mohamed & Auli (2020)**, _wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations_, NeurIPS 2020 — contrastive masked prediction over quantized latent audio units, pretrained on unlabeled speech and fine-tuned on small labeled sets. [^hubert]: **Hsu et al. (2021)**, _HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units_, IEEE/ACM TASLP 2021 — masked prediction of self-derived cluster labels; learned features in which voicing, place, and formant structure re-emerge under probing. [^whisper]: **Radford et al. (2022)**, _Robust Speech Recognition via Large-Scale Weak Supervision_ (Whisper) — an encoder-decoder trained on ~680,000 hours of weakly labeled multilingual web audio over an intact log-mel front end, giving robust multilingual recognition without task-specific fine-tuning.