Acoustic Phonetics

This builds on Phonetics, which covered the articulatory and prosodic side of speech — how the vocal tract shapes airflow into consonants and vowels, and how stress and tune ride on top. That was the cause. This part is the effect: the pressure wave those movements produce, and everything a recognizer can measure in it.

Acoustic phonetics and signals

Articulation is the cause; the acoustic signal is the effect, and the effect is all a microphone or an ear ever gets.¹ Acoustic phonetics is the study of that signal — and it is the immediate input to speech recognition.

Waves, frequency, and amplitude

The building block is the sine wave, $y=A\sin (2\pi ft)$, with two parameters. The frequency $f$ is how many cycles occur per second, measured in hertz (Hz); the amplitude $A$ is the peak height. The period $T=1/f$ is the time for one cycle.

A speech waveform plots air pressure against time. When a speaker's vocal folds and vocal tract push and pull the air, a microphone (or eardrum) records the resulting compression and rarefaction as a wiggling line. A vowel's waveform is not a pure sine, but it is periodic — it repeats — and the rate of that repetition carries the pitch.

Turning that analog pressure into numbers is analog-to-digital conversion, in two steps. Sampling measures the amplitude at regular instants; the sampling rate is the number of samples per second. To capture a wave you need at least two samples per cycle, so the highest recoverable frequency is half the sampling rate — the Nyquist frequency. Most speech energy sits below 10 kHz, so 16 kHz sampling suffices for microphone speech; telephone speech, band-limited to 4 kHz, needs only 8 kHz. Quantization then rounds each sampled amplitude to an integer (commonly 16-bit), and the stream is stored in a format like .wav.

Pitch, loudness, and their perceptual scales

The rate at which a voiced waveform repeats is its fundamental frequency, F0 — the frequency of vocal-fold vibration.¹ Each period corresponds to one opening of the folds. Plotting F0 over time gives a pitch track, whose shape recovers the tune from the prosody section: the pitch track of Three o'clock? rises at the end.

Amplitude has its own summaries. Because raw amplitude swings positive and negative, its average is near zero, so we use the RMS amplitude — square, average, then square-root: $\sqrt{\frac{1}{N}{\sum }_i{x}_i^2}$. The related power is the mean square $\frac{1}{N}{\sum }_i{x}_i^2$, and intensity normalizes power against the auditory threshold pressure $P_0=2\times 10^{-5}$ Pa and reports it in decibels, $10{\log }_{10}\left(\frac{1}{N{P}_0}{\sum }_i{x}_i^2\right)$.

Two perceptual quantities ride on these physical ones. Pitch is the perceived correlate of F0, and loudness the perceived correlate of power — but neither map is linear. Pitch tracks frequency linearly only between roughly 100 Hz and 1000 Hz; above that, equal pitch steps span ever-larger frequency gaps. The mel scale captures this: a mel is a unit of pitch on which perceptually equal intervals are equal numbers of mels, and

This formula is the warping that the log-mel front end applies to spectrogram frequencies before passing them to the recognizer.

The spectrum and the frequency domain

Reading phones off a raw waveform works only for coarse features — you can spot the regular voicing peaks of a vowel, the near-silence-then-burst of a stop, the noisy hiss of a fricative. Fine phonetic detail lives in the frequency domain. The insight of Fourier analysis is that any complex wave is a sum of pure sine waves of different frequencies. The spectrum of a signal lists those component frequencies and their amplitudes: sum a 10 Hz and a 100 Hz sine, and the spectrum shows exactly two spikes, at 10 Hz and 100 Hz.

Run the same decomposition on a vowel and the spectrum is richer: a set of peaks at characteristic frequencies. Those peaks are the phone's acoustic fingerprint — just as burning elements emit light at characteristic wavelengths, phones concentrate energy at characteristic frequencies, and reading them off the spectrum is how both the cochlea and a recognizer identify sounds.

The source-filter model

Why does a vowel's spectrum have peaks where it does? The source-filter model separates two contributions.¹ The source is the glottal buzz: vocal-fold vibration at F0, which produces not just F0 but a whole ladder of harmonics at integer multiples ($2f$, $3f$, …), each weaker than the last. The filter is the vocal tract: a tube whose shape makes some frequencies resonate (amplify) and damps others. Moving the tongue changes the tube's shape, so it changes which harmonics get amplified.

The amplified bands are formants: frequency regions where the vocal tract resonates. Each vowel, produced with a different tract shape, has a different pattern of formants — and that is why the vowel space earlier correlates more with acoustics than with tongue position. The formants are the acoustic identity of the vowel.

Formants, F0, and the spectrogram

A spectrum is a snapshot — the frequency content at one instant. Speech changes over time, so we need a movie: the spectrogram, which plots time on the x-axis, frequency on the y-axis, and amplitude as darkness. A spectrogram is a stack of spectra, each a thin vertical time-slice, placed side by side. This is the central representation, and the direct input to a recognizer.

Two independent quantities live in this picture, and confusing them is the single most common error. F0 is the periodicity of the waveform — the rate the vocal folds vibrate, the source, heard as pitch. The formants are the horizontal bands in the spectrogram — the resonances of the tract, the filter, heard as vowel quality. You can raise F0 (say the vowel higher) while holding the formants fixed (same vowel), or hold F0 and change the formants (same pitch, different vowel). F1 and F2 do most of the work of telling vowels apart: [iy] has a low F1 (~540 Hz) and high F2 (~2580 Hz), while [ae] has a higher F1 (~930 Hz) and lower F2 (~1860 Hz). Formants also help identify nasals and the liquids [l] and [r]. Higher formants (F3 and up) depend more on the speaker's vocal-tract size than on the specific vowel.

A worked formant reading

Two numbers per vowel are enough to separate the corners of the vowel space, and they invert the articulatory picture. F1 falls as the tongue rises (high vowels have low F1) and F2 rises as the tongue moves forward (front vowels have high F2). Plotting a vowel at its $(F2,F1)$ coordinates — F2 on a reversed horizontal axis, F1 on a reversed vertical axis — reproduces the tongue- position quadrilateral from articulation, which is why the vowel space correlates with acoustics. Take three measured English vowels from a canonical formant table: [iy] (beet) at F1 $270$ Hz, F2 $2290$ Hz; [aa] (bot) at F1 $730$ Hz, F2 $1090$ Hz; [uw] (boot) at F1 $300$ Hz, F2 $870$ Hz. [iy] lands high and front (low F1, high F2), [uw] high and back (low F1, low F2), [aa] low and central- back (high F1, mid F2) — exactly their tongue positions.

The pair $(F1,F2)$ is the coordinate a vowel classifier reads, and the whole point of the mel warping below is to space those frequency measurements the way the ear spaces them before the classifier ever sees them.

Phonetic resources

Computational phonetics draws on two kinds of resource.² A pronunciation dictionary maps words to phone strings: the open-source CMU Pronouncing Dictionary has about 134,000 words with stress marks; CELEX adds syllabification and part-of-speech; UNISYN covers dozens of English dialects. A phonetically annotated corpus pairs recorded waveforms with hand-labeled phone strings. The canonical one is TIMIT — 6300 read sentences, 10 from each of 630 speakers, each phone hand-labeled and time-aligned to the waveform, yielding a time-aligned transcription where every phone carries a start and end time. The Switchboard Transcription Project and the Buckeye corpus of spontaneous speech play the same role for conversational speech. Tools like Praat compute the pitch, spectral, and formant analyses this lesson has been describing — and generated many of the figures in the source text.

Where phonetics went in neural speech

The source treats phonetics as the hand analysis a recognizer later mechanizes. Modern systems went further: they either learn the acoustic representation directly from waveforms, or generate the waveform from scratch. Both build on the physics this lesson laid out.

Neural text-to-speech generates the waveform. Von Kempelen's bellows machine and later concatenative synthesizers stitched recorded units; neural TTS instead predicts the signal. WaveNet (van den Oord et al., WaveNet: A Generative Model for Raw Audio, 2016) is an autoregressive convolutional network that models the waveform one sample at a time, $P(x_t\mid x_1,\dots ,x_{t-1})$, with dilated causal convolutions to reach far back in time. It produced markedly more natural speech than the concatenative and parametric systems before it, at the cost of slow sample-by-sample generation — which later work (parallel WaveNet, WaveGlow, HiFi-GAN) sped up. Tacotron 2 (Shen et al., Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions, ICASSP 2018) split the task exactly along this lesson's source-filter division: a sequence-to-sequence network first predicts a mel spectrogram from text — the same log-mel representation the vowel and formant analysis here produces — and a WaveNet-style vocoder then turns that spectrogram into audio. The mel spectrogram is the interface, which is why a TTS system is cut in two at the acoustic middle of this chapter.³⁴

Self-supervised models learn the representation from raw audio. The log-mel front end is a fixed, hand-designed transform. wav2vec 2.0 (Baevski et al., NeurIPS 2020) instead learns a speech representation by masking spans of a latent audio encoding and training the model to pick the correct quantized latent from distractors — a contrastive objective on unlabeled speech, in the spirit of masked language modeling. HuBERT (Hsu et al., IEEE/ACM TASLP 2021) trains on masked prediction of cluster labels derived from the audio itself. Both pretrain on thousands of hours of unlabeled speech and then fine-tune on a small labeled set, which is what made high-accuracy recognition possible for low-resource languages. The representations they learn are not hand-specified formant tracks — but probing studies find that the physical categories of this chapter (voicing, place of articulation, formant structure) re-emerge in the learned features, because they are the information the signal actually carries.⁵⁶

Whisper and weak supervision. OpenAI's Whisper (Radford et al., 2022) trained an encoder-decoder on $680,000$ hours of weakly labeled multilingual audio scraped from the web, and showed that scale alone — the log-mel spectrogram front end kept intact — yields robust multilingual recognition without task-specific fine-tuning. The acoustic representation this chapter derives by hand stayed at the input, and the modeling above it got large enough to absorb everything else.⁷