---
title: What Is Natural Language Processing?
module: Foundations
moduleNumber: 1
lessonNumber: 1
order: 101
summary: >
  Natural language processing is the computational treatment of human language:
  reading it, representing it, and generating it. We set up why the problem is
  hard — ambiguity at every level, from sound to intent — trace the field from
  ELIZA's pattern-matching through statistical methods to today's neural models,
  lay out the linguistic levels and task families the course covers, and fix the
  vocabulary of tokens, types, and corpora the rest of the notes rely on.
topics: [Foundations]
sources:
  - book: Jurafsky & Martin
    ref: "Ch. 1 — Introduction; Ch. 2 opening — ELIZA and the pattern-matching tradition"
  - book: Jurafsky & Martin
    ref: "§2.2 Words — types, tokens, corpora, and the vocabulary $V$"
---

A five-year-old resolves the sentence "I made her duck" in the time it takes to
hear it, usually without noticing there was anything to resolve. A machine has to
be told that the sentence has at least five readings, and that choosing among them
draws on the sound stream, the words, the grammar, the meanings, and the situation
all at once. **Natural language processing** (NLP) is the discipline of getting
computers to do the resolving: to read, represent, and generate human language
with enough competence to translate it, answer questions from it, transcribe it,
and hold a conversation in it.

The field sits at the meeting point of linguistics, which describes the structure
of language, and machine learning, which supplies the methods for fitting models
to data. What makes it its own subject rather than a corner of either is the
particular character of the problem: language is generated by people for people,
under the assumption that the listener shares an enormous amount of unstated
context, and almost every choice a speaker makes is locally ambiguous and globally
resolved.

## Why language is hard: ambiguity at every level

In NLP, ambiguity is not an edge case; it is the normal condition of every
utterance. A sentence that seems to have one obvious meaning has that meaning only
because the reader has already, without noticing, discarded the alternatives.
Consider Jurafsky and Martin's example:[^jm-intro]

> I made her duck.

This four-word sentence is ambiguous at several distinct linguistic levels, and the
ambiguities compound:

- **I cooked waterfowl for her.** Here _duck_ is a noun (a bird), _her_ is the
  beneficiary, and _made_ means "cooked".
- **I cooked waterfowl belonging to her.** Same _duck_, but now _her_ is a
  possessor.
- **I created the (plaster?) duck she owns.** _made_ means "created", _duck_ is a
  noun.
- **I caused her to quickly lower her head or body.** Now _duck_ is a _verb_, _her_
  is the subject of that verb, and _made_ is causative.
- **I magically turned her into a duck.** _made_ is again causative, but _duck_ is
  a predicate.

Each reading turns on a decision at a different level of linguistic structure. Is
_duck_ a noun or a verb (a **lexical** ambiguity, in its part of speech)? Does _her_
attach as an indirect object or as the subject of an embedded verb (a **syntactic**
ambiguity, in the parse)? Does _make_ mean "cook", "create", or "cause" (a
**semantic** ambiguity, in word sense)? And in speech there is a level below all of
these: the string _made her_ and the string _mader_ are acoustically nearly
identical, so a speech recognizer faces a **phonetic** ambiguity before it can even
commit to a sequence of words.

$$
% caption: The ambiguity ladder for "I made her duck": each rung is a level of
% linguistic structure where the sentence forks, and every reading is a path that
% commits at each rung. Lower rungs feed the ones above.
\begin{tikzpicture}[>=stealth, font=\small,
  rung/.style={draw, minimum width=34mm, minimum height=9mm, align=center, font=\footnotesize},
  side/.style={font=\scriptsize, text=black, align=left, anchor=west}]
  \definecolor{acc}{HTML}{2348F2}
  \node[rung, draw=acc, text=acc] (prag) at (0,4.4) {pragmatic};
  \node[rung] (sem)  at (0,3.3) {semantic};
  \node[rung] (syn)  at (0,2.2) {syntactic};
  \node[rung] (lex)  at (0,1.1) {lexical};
  \node[rung] (pho)  at (0,0.0) {phonetic};
  \draw[->, acc, thick] (pho) -- (lex);
  \draw[->, acc, thick] (lex) -- (syn);
  \draw[->, acc, thick] (syn) -- (sem);
  \draw[->, acc, thick] (sem) -- (prag);
  \node[side] at (2.4,4.4) {what did she intend?};
  \node[side] at (2.4,3.3) {cook vs. create vs. cause};
  \node[side] at (2.4,2.2) {her = object or subject?};
  \node[side] at (2.4,1.1) {duck = noun or verb?};
  \node[side] at (2.4,0.0) {made her vs. mader};
\end{tikzpicture}
$$

The reason a human never notices the ambiguity is that resolution happens by
context, and context includes everything from the previous sentence to who is
speaking to what a duck is. The reason a machine struggles is that all of that
context has to be either supplied as data or learned from data. Every subfield of
NLP can be read as an attack on one rung of this ladder: part-of-speech tagging
resolves the lexical rung, parsing the syntactic one, word-sense disambiguation and
semantic parsing the semantic one, and coreference and discourse the pragmatic one.

> **Definition (Ambiguity).** An input is **ambiguous** if it admits more than one
> valid interpretation at some level of analysis. NLP systems must either commit to
> one interpretation (disambiguation) or carry a distribution over interpretations
> forward. Ambiguity is _local_ (present in a fragment) and often _resolvable_
> (settled by wider context), and managing the gap between the two is the recurring
> engineering problem of the field.

## The levels of linguistic structure

The ambiguity ladder is really a view of a deeper organizing idea: language is
structured in **levels**, each built on the one below, and NLP mirrors that
stack.[^jm-intro] A complete system that goes from a sound wave to an understood
meaning passes through every level, whether or not it does so with explicit modules.

$$
% caption: The levels of linguistic structure, sound at the bottom and discourse
% at the top. Each level takes the units below and combines them into larger
% units, and each is a source of ambiguity that NLP must resolve.
\begin{tikzpicture}[>=stealth, font=\small,
  lvl/.style={draw, minimum width=52mm, minimum height=9mm, align=center, font=\footnotesize},
  note/.style={font=\scriptsize, text=black, anchor=west}]
  \definecolor{acc}{HTML}{2348F2}
  \node[lvl] (phon) at (0,0.0) {phonology / phonetics};
  \node[lvl] (morph) at (0,1.05) {morphology};
  \node[lvl] (syn) at (0,2.1) {syntax};
  \node[lvl] (sem) at (0,3.15) {semantics};
  \node[lvl, draw=acc, text=acc] (prag) at (0,4.2) {pragmatics / discourse};
  \node[note] at (2.9,0.0) {sounds and their patterns};
  \node[note] at (2.9,1.05) {word structure: un-lock-able};
  \node[note] at (2.9,2.1) {how words form phrases};
  \node[note] at (2.9,3.15) {literal meaning};
  \node[note] at (2.9,4.2) {meaning in context and use};
  \draw[->, black, thick] (-3.3,-0.5) -- (-3.3,4.7);
  \node[rotate=90, anchor=south, font=\scriptsize, text=black] at (-3.55,2.1) {larger units, more context};
\end{tikzpicture}
$$

Each level names both a kind of structure and a family of tasks:

| Level | Unit of analysis | Question it answers | Course topic |
| --- | --- | --- | --- |
| Phonology / phonetics | sounds (phones) | which sounds, in what patterns? | [speech recognition](/natural-language-processing/speech/automatic-speech-recognition) |
| Morphology | morphemes | how are words built from stems and affixes? | [text normalization](/natural-language-processing/foundations/regex-and-text-normalization) |
| Syntax | phrases, clauses | how do words combine into grammatical structure? | [sequence labeling](/natural-language-processing/sequences/sequence-labeling) |
| Semantics | word and sentence meaning | what does it literally mean? | [vector semantics](/natural-language-processing/semantics/vector-semantics-and-embeddings) |
| Pragmatics / discourse | utterances in context | what did the speaker mean here? | [dialogue](/natural-language-processing/applications/dialogue-and-chatbots) |

The classical view treated these as a **pipeline**: run a tokenizer, then a
morphological analyzer, then a parser, then a semantic interpreter, each consuming
the previous stage's output. The modern neural view, which we reach later in the
course, largely dissolves the seams — a single model maps sound or text toward
meaning end-to-end — but the _levels_ remain the right way to understand what a
model must implicitly get right, and the right way to diagnose where it fails.

## A short history: from ELIZA to LLMs

The field's history is a story of the ambiguity problem being attacked with three
successive toolkits: hand-written rules, statistics over corpora, and learned
representations. Each era begins with the previous one's failures.

### ELIZA and the age of patterns (1960s)

The first program most people would call a natural language system was **ELIZA**,
written by Joseph Weizenbaum in 1966.[^jm-eliza] ELIZA imitated a Rogerian
psychotherapist — a style of therapy in which the therapist mostly reflects the
patient's statements back as questions — and it did so by matching patterns in the
input and rewriting them into responses.

> User: I am unhappy.
> ELIZA: DO YOU THINK COMING HERE WILL HELP YOU NOT TO BE UNHAPPY
> User: I need some help, that much seems certain.
> ELIZA: WHAT WOULD IT MEAN TO YOU IF YOU GOT SOME HELP
> User: Perhaps I could learn to get along with my mother.
> ELIZA: TELL ME MORE ABOUT YOUR FAMILY
> User: My mother takes care of me.
> ELIZA: WHO ELSE IN YOUR FAMILY TAKES CARE OF YOU

There is no understanding here. ELIZA works by a cascade of substitutions built out
of the same [regular expressions](/natural-language-processing/foundations/regex-and-text-normalization)
we study in the next lesson: a rule spots the pattern "I need X" and emits "WHAT
WOULD IT MEAN TO YOU IF YOU GOT X", swapping the pronouns so that _my_ becomes
_your_. The program keeps no model of the world, the conversation, or the user.

$$
% caption: ELIZA as a rewrite loop. The input is uppercased and pronouns are
% flipped, a ranked pattern matches, and the matched fragment is substituted into
% a template. No representation of meaning is ever built.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=27mm, minimum height=11mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (in) at (0,0) {user input};
  \node[box] (norm) at (3.6,0) {f\/lip pronouns\\(my $\to$ your)};
  \node[box, draw=acc, text=acc] (match) at (7.6,0) {match ranked\\pattern};
  \node[box] (out) at (11.4,0) {f\/ill template,\\reply};
  \draw[->, acc, thick] (in) -- (norm);
  \draw[->, acc, thick] (norm) -- (match);
  \draw[->, acc, thick] (match) -- (out);
\end{tikzpicture}
$$

What makes ELIZA a permanent lesson is that it fooled people. Many users came to
believe ELIZA genuinely understood them, and some kept believing it even after the
mechanism was explained.[^jm-eliza] Weizenbaum himself was disturbed by this. The
Rogerian setting is a rare genre in which a system can succeed while knowing
nothing about the world — the therapist is _supposed_ to reflect rather than
inform — so ELIZA's success reveals both how much fluency pattern-matching can fake
and how little that fluency has to do with comprehension. That gap, between
convincing surface behavior and actual understanding, is the through-line of the
whole subject, and it returns in the era of large language models.

### Measuring understanding: the imitation game

ELIZA forces a question the field has never fully settled: how would you _test_
whether a machine understands language? The oldest proposed answer predates NLP.
In 1950 Alan Turing set aside the unanswerable "can machines think?" and replaced it
with an operational **imitation game**: a human judge holds a text conversation with
an unseen partner that is either a person or a machine, and the machine passes if the
judge cannot reliably tell which is which.[^jm-eliza] The test is deliberately
behavioral — it grades what the system _does_ with language, not what happens inside
it. ELIZA is the cautionary counterexample. In its narrow Rogerian genre it fooled
people, yet it plainly understood nothing, which is the standing objection to a
purely behavioral test: fluency can be faked, and a judge's willingness to attribute
a mind is easy to exploit.

$$
% caption: The imitation game. A judge exchanges text with a hidden human and a
% hidden machine and must decide which is which; the machine passes if the judge
% cannot reliably tell. The test grades behavior, not internal understanding.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  party/.style={draw, minimum width=22mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[party] (judge) at (0,0) {judge};
  \node[party] (human) at (5,1.1) {hidden human};
  \node[party, draw=acc, text=acc] (machine) at (5,-1.1) {hidden machine};
  \draw[<->, black, thick] (judge) -- (human) node[midway, above, font=\scriptsize] {text only};
  \draw[<->, acc, thick] (judge) -- (machine) node[midway, below, font=\scriptsize] {text only};
  \node[black, anchor=west, font=\scriptsize, align=left] at (7.2,0)
    {machine passes if judge \\ cannot reliably tell \\ which is which};
\end{tikzpicture}
$$

The philosophical stakes were sharpened by Searle's **Chinese room** argument
(1980): imagine a person who speaks no Chinese, sealed in a room, following a rule
book that maps incoming Chinese symbols to outgoing ones. From outside the room the
answers look fluent, yet no one and nothing in the room understands Chinese — the
symbols are manipulated by their _shape_, never connected to meaning. Searle's claim
is that symbol manipulation alone, however competent, is not comprehension. Whether
that argument holds against modern systems is contested; what matters for the course
is the distinction it names. A system can be measured on **task performance** — does
it translate, answer, summarize correctly? — without any claim that it "understands,"
and almost all of NLP is evaluated the first way. The gap ELIZA opened, between
fluent behavior and grounded meaning, is what the imitation game cannot see
past, and it is why the field measures systems on concrete tasks with concrete
metrics rather than on whether they seem to understand.

### The statistical turn (1980s–2000s)

Hand-written rules do not scale. English has too many words, too many patterns, and
too many exceptions for a person to enumerate, and every new domain breaks the
rules tuned for the last one. The **statistical** turn replaced the rule-writer with
the corpus: rather than tell the machine that _duck_ is more often a noun than a
verb, count it in a large body of text and let the frequencies decide. The next two
lessons are the origin of this era — count word occurrences to estimate
[n-gram probabilities](/natural-language-processing/foundations/n-gram-language-models),
and train a [naive Bayes](/natural-language-processing/classification/naive-bayes-and-sentiment)
or [logistic-regression](/natural-language-processing/classification/logistic-regression)
classifier on labeled examples. Ambiguity becomes a probability distribution rather
than a hard obstacle: the parser does not need the one true parse, only the most
likely one given the data.

### The neural and LLM era (2010s–present)

Statistical models still leaned on features a human designed — this word, that
suffix, this preceding tag. The **neural** turn automated the features too. Words
become dense [vectors learned from context](/natural-language-processing/semantics/vector-semantics-and-embeddings)
so that meaning becomes geometry; a network composes those vectors into the meaning
of a phrase and a sentence. Then the
[transformer](/natural-language-processing/transformers/transformers-and-attention),
by letting every word attend to every other, scaled cleanly with data and compute,
and trained on enough text it became the
[large language model](/natural-language-processing/transformers/large-language-models):
one architecture that translates, summarizes, answers, and converses.

$$
% caption: Three eras, distinguished by where the knowledge comes from. Rules are
% written by hand; statistical models count features a human chose; neural models
% learn the features themselves.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  era/.style={draw, minimum width=30mm, minimum height=15mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[era] (rule) at (0,0) {rules\\(ELIZA, 1966)\\human writes patterns};
  \node[era] (stat) at (4.2,0) {statistics\\(n-grams, 1990s)\\count chosen features};
  \node[era, draw=acc, text=acc, thick] (neu) at (8.4,0) {neural\\(transformers, 2017-)\\learn the features};
  \draw[->, acc, thick] (rule) -- (stat);
  \draw[->, acc, thick] (stat) -- (neu);
  \draw[->, black, thick] (-1.6,-1.5) -- (10.0,-1.5);
  \node[font=\scriptsize, text=black, anchor=north] at (4.2,-1.55) {less hand-engineering, more learning from data};
\end{tikzpicture}
$$

The arc of the field is a steady transfer of work from the human to the data. ELIZA's
designer wrote every pattern; the statistical modeler chose every feature and let the
counts fill in the weights; the neural modeler specifies an architecture and a loss
and lets gradient descent discover the representation. The course follows that arc in
order — from regexes and n-grams to embeddings and transformers — so that each new
tool is legible as a response to the previous one's ceiling.

## The task families this course covers

Underneath the applications, NLP problems fall into a small number of shapes. Naming
them early makes the rest of the course navigable, because most systems are one of
these shapes, or a composition of a few.

$$
% caption: The major task families. Most NLP applications reduce to one of these
% shapes or a pipeline of several; the same modeling toolkit serves all of them.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  root/.style={draw, thick, minimum width=30mm, minimum height=9mm, align=center},
  leaf/.style={draw, minimum width=32mm, minimum height=12mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[root, draw=acc, text=acc] (r) at (0,0.6) {NLP tasks};
  \node[leaf] (cls)  at (-6.6,-2.2) {classi\/f\/ication\\(one label per text)};
  \node[leaf] (seq)  at (-3.3,-2.2) {sequence labeling\\(one tag per token)};
  \node[leaf] (parse) at (0,-2.2) {parsing\\(build structure)};
  \node[leaf] (gen)  at (3.3,-2.2) {generation\\(translate, summarize)};
  \node[leaf] (qa)   at (6.6,-2.2) {QA and dialogue\\(respond to input)};
  \draw[->, acc, thick] (r) -- (cls);
  \draw[->, acc, thick] (r) -- (seq);
  \draw[->, acc, thick] (r) -- (parse);
  \draw[->, acc, thick] (r) -- (gen);
  \draw[->, acc, thick] (r) -- (qa);
\end{tikzpicture}
$$

| Family | Shape of the problem | Example task | Course lesson |
| --- | --- | --- | --- |
| Classification | text $\to$ one of $k$ labels | sentiment analysis, spam | [naive Bayes](/natural-language-processing/classification/naive-bayes-and-sentiment) |
| Sequence labeling | token sequence $\to$ tag sequence | part-of-speech, named entities | [sequence labeling](/natural-language-processing/sequences/sequence-labeling) |
| Parsing | sentence $\to$ tree / graph | constituency, dependency structure | (later modules) |
| Translation | text $\to$ text, one language to another | machine translation | [machine translation](/natural-language-processing/applications/machine-translation) |
| Question answering | question $+$ context $\to$ answer | factoid QA, reading comprehension | [question answering](/natural-language-processing/applications/question-answering) |
| Dialogue | conversation history $\to$ next turn | chatbots, assistants | [dialogue](/natural-language-processing/applications/dialogue-and-chatbots) |
| Speech | audio $\to$ text | automatic speech recognition | [speech recognition](/natural-language-processing/speech/automatic-speech-recognition) |

Two families do most of the structural work. **Classification** assigns a single
label to a whole span of text, and **sequence labeling** assigns a label to each
token in order; much of NLP reduces to one of these two. The generation tasks —
translation, summarization, dialogue — share the harder shape of producing text
rather than choosing from a fixed set, and they are where neural sequence models
and transformers are needed.

## The pipeline-to-end-to-end shift

For most of the field's history a system was a **pipeline**: a chain of components,
each solving one level of the linguistic stack and passing its output to the next.
Tokenize, then tag parts of speech, then parse, then resolve references, then extract
meaning. The appeal is modularity — each stage is separately buildable and
debuggable — but the cost is that errors compound. A tokenizer mistake corrupts the
tagger's input, a tagging mistake corrupts the parser's, and no downstream stage can
recover information an upstream stage discarded.

$$
% caption: Two ways to build a system. The classical pipeline chains hand-defined
% stages, each a place errors can enter; the end-to-end model maps input to output
% through learned representations with no fixed seams.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=19mm, minimum height=9mm, align=center, font=\scriptsize},
  wide/.style={draw, minimum width=44mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  % --- pipeline (top) ---
  \node[box] (t) at (0,1.6) {tokenize};
  \node[box] (p) at (2.6,1.6) {tag};
  \node[box] (q) at (5.2,1.6) {parse};
  \node[box] (s) at (7.8,1.6) {interpret};
  \draw[->, black, thick] (t) -- (p);
  \draw[->, black, thick] (p) -- (q);
  \draw[->, black, thick] (q) -- (s);
  \node[font=\scriptsize, text=black, anchor=west] at (8.9,1.6) {errors compound};
  \node[font=\footnotesize, anchor=east] at (-0.9,1.6) {pipeline:};
  % --- end-to-end (bottom) ---
  \node[box] (in2) at (0,-0.6) {text};
  \node[wide, draw=acc, text=acc, thick] (model) at (4.0,-0.6) {one learned model};
  \node[box] (out2) at (7.9,-0.6) {output};
  \draw[->, acc, thick] (in2) -- (model);
  \draw[->, acc, thick] (model) -- (out2);
  \node[font=\footnotesize, anchor=east] at (-0.9,-0.6) {end-to-end:};
\end{tikzpicture}
$$

The neural era replaced the pipeline with **end-to-end** models: a single network
maps the raw input toward the final output, learning whatever intermediate
representations serve the task rather than being handed human-defined ones. This
has been the dominant recipe on translation, speech, and question answering, where
the hand-built pipeline has all but disappeared. The trade is real — end-to-end
models need far more data and compute, and they are harder to inspect — but on the
tasks where enough data exists, jointly optimizing every stage against one final
objective wins. We keep the levels as a _conceptual_ map even when the model no
longer has separate modules for them.

## Notation: tokens, types, and corpora

Before any modeling, we need to count words, and counting forces a distinction the
rest of the course relies on.[^jm-words] A **corpus** (plural _corpora_) is a
computer-readable body of text or speech — the raw material NLP learns from. Given a
corpus, take a sentence like

> They picnicked by the pool, then lay back on the grass and looked at the stars.

Ignoring punctuation, this sentence has 16 **tokens**: the running words in order,
counted with repetition. But it has only 14 **types**: the distinct words. The two
occurrences of _the_ are two tokens of one type. When we ask "how many words is
this?", the answer depends on which we mean.

> **Definition (Token vs. type).** A **token** is an individual occurrence of a word
> in running text; a **type** is a distinct word form. A corpus of $N$ tokens draws
> on a **vocabulary** of $|V|$ types, where $V$ is the set of distinct types and
> $|V|$ its size. Repetition inflates the token count $N$ but never the type count
> $|V|$.

We write $N$ for the number of tokens in a corpus and $V$ for its **vocabulary**,
the set of types, with size $|V|$. These two numbers grow at very different rates.
As a corpus gets larger, new tokens keep arriving long after most common types have
appeared, so $N$ climbs steeply while $|V|$ climbs slowly. In a long book, by the
halfway point almost every common word has appeared, so the second half adds many
tokens but few genuinely new types. Their relationship is empirically
a power law, **Herdan's Law** (also Heaps' Law),

$$
|V| \;=\; k\,N^{\beta},
\qquad 0 < \beta < 1,
$$

with $k$ and $\beta$ positive constants: the vocabulary grows as a sublinear power
of the token count. In real corpora the gap is large: the Brown corpus has about
1 million tokens but only ~38 thousand types, while the far larger Google n-grams
corpus has ~1 trillion tokens and ~13 million types.[^jm-words] The vocabulary $V$,
the token count $N$, and the corpus are the three quantities every later lesson is
measured against: n-gram models estimate probabilities over $V$, classifiers turn a
document into counts over $V$, and embeddings assign a learned vector to each type in
$V$.

### Zipf's law and the long tail

The vocabulary does not just grow slowly; its _frequencies_ are wildly uneven, and
that unevenness shapes every method built on top of counts. Rank the types of a
corpus from most to least frequent, giving the most common type rank $1$, the next
rank $2$, and so on. **Zipf's law** is the empirical observation that a type's
frequency is roughly inversely proportional to its rank:[^jm-zipf]

$$
f_r \;\approx\; \frac{C}{r},
\qquad\text{equivalently}\qquad
f_r \cdot r \;\approx\; C,
$$

for a constant $C$ near the corpus size. The most frequent word appears about twice
as often as the second, three times as often as the third. Take logs of both sides
and the law becomes a straight line, $\log f_r \approx \log C - \log r$, with slope
$-1$: a frequency-versus-rank plot on log–log axes is nearly linear.

$$
% caption: Zipf's law on log-log axes. Plotting log frequency against log rank
% gives a near-straight line of slope about -1: a handful of types account for most
% tokens (the head), and a very long tail of types each appear only once or twice.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \draw[->, black] (-0.2,0) -- (6.6,0) node[right, black, font=\scriptsize] {log rank};
  \draw[->, black] (0,-0.2) -- (0,4.2) node[above, black, font=\scriptsize] {log freq};
  % the Zipf line, slope about -1
  \draw[acc, very thick] (0.4,3.9) -- (6.0,0.35);
  \node[acc, anchor=west, font=\scriptsize] at (4.3,1.55) {slope about -1};
  % head vs tail annotations
  \draw[black, dashed] (1.4,0) -- (1.4,3.05);
  \node[black, anchor=south, font=\scriptsize] at (0.85,3.15) {head:};
  \node[black, anchor=south, font=\scriptsize] at (0.85,2.75) {the, of, and};
  \node[black, anchor=west, font=\scriptsize] at (2.3,0.35) {tail: rare types};
\end{tikzpicture}
$$

The consequence is a **long tail**: a small set of function words (_the_, _of_,
_and_) accounts for a large share of all tokens, while most of the vocabulary
consists of types that appear only once or twice. In the Brown corpus about half the
types are **hapax legomena** — words that occur exactly once. For example, in that
corpus _the_ occurs roughly 69{,}971 times, so with a
corpus of about a million tokens $f_1 \cdot 1 \approx 70{,}000$; the word ranked
around 35th, near _at_, occurs on the order of $2{,}000$ times, and $f_{35}\cdot 35
\approx 70{,}000$ holds to within the noise. The product $f_r \cdot r$ stays close to
constant across three orders of magnitude of rank.

Two consequences follow, and both recur through the rest of the course. First,
most of the vocabulary will be **rare**, so a model that only works on words
it has seen many times will fail on most of the words it meets — the motivation for
smoothing in [n-gram models](/natural-language-processing/foundations/n-gram-language-models)
and for the subword tokenization the next lesson builds. Second, the head is cheap
and the tail is expensive: a few thousand common types cover most running text, but
capturing the tail well is where the modeling effort goes.

## The pretrain-and-prompt era

The SLP3 draft was written as the field's center of gravity moved a second time,
and the move is worth stating on its own because it reframes what "an NLP system" is.
For most of the statistical and early-neural period, you built one model per task:
collect labeled data for sentiment, train a sentiment classifier; collect labeled
data for translation, train a translation model. The **pretraining** paradigm breaks
that one-model-per-task assumption. A single large network is trained once, on a
generic objective over enormous unlabeled text, and the same pretrained model is then
adapted to many tasks.

The pivot was **BERT** (Devlin et al., 2019, NAACL),[^devlin-bert] which pretrained a
transformer on a _masked language modeling_ objective — predict words blanked out of a sentence —
over billions of words, then **fine-tuned** the result with a small labeled set per
downstream task. Fine-tuned BERT set new state-of-the-art numbers across a broad
suite of tasks at once (the GLUE benchmark, question answering, named-entity
recognition), which established the recipe: pretrain a general language
representation, adapt it cheaply. The work of designing task-specific features, which
the statistical era spent most of its effort on, largely disappeared into the
pretrained weights.

$$
% caption: The pretrain-then-adapt recipe. One transformer is pretrained once on a
% generic objective over unlabeled text; the same model is then adapted to many
% tasks, either by fine-tuning on labeled data or by prompting with no weight update.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=11mm, align=center, font=\scriptsize},
  leaf/.style={draw, minimum width=24mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box, draw=acc, text=acc, thick] (pre) at (0,0) {pretrain once\\(unlabeled text)};
  \node[box] (model) at (4.2,0) {one large model};
  \node[leaf] (t1) at (8.6,1.5)  {sentiment};
  \node[leaf] (t2) at (8.6,0.0)  {translation};
  \node[leaf] (t3) at (8.6,-1.5) {question answering};
  \draw[->, acc, thick] (pre) -- (model);
  \draw[->, black, thick] (model) -- (t1);
  \draw[->, black, thick] (model) -- (t2);
  \draw[->, black, thick] (model) -- (t3);
  \node[black, anchor=south, font=\scriptsize] at (6.4,0.1) {adapt};
\end{tikzpicture}
$$

The next step removed even the fine-tuning. **GPT-3** (Brown et al., 2020,
NeurIPS)[^brown-gpt3] scaled an autoregressive language model to 175 billion
parameters and showed that a
model that large, trained only to predict the next word, can perform a new task from
a natural-language description and a few examples placed in its input — **in-context
learning**, with no gradient update to the weights at all. Ask it to translate by
showing three example translations and then a fourth source sentence, and it
continues the pattern. The task specification moved out of the training loop and into
the **prompt**. This is the mechanism behind the instruction-following assistants
that followed; the same one architecture translates, summarizes, answers, and
converses, selected among those behaviors by what you write to it.

Two caveats apply, and both trace back to ELIZA. First, next-word
prediction optimizes fluency, not truth: these models produce confident,
well-formed text that is sometimes simply wrong, a failure the literature calls
**hallucination**. The gap between convincing surface behavior and grounded
correctness that Weizenbaum warned about is now a first-order engineering problem, not
a curiosity. Second, scale changed what is possible but not the underlying levels of
linguistic structure — a large model that gets syntax and semantics right most of the
time still gets them right by having implicitly learned the same distinctions this
lesson laid out, and it still fails at the same rungs of the ambiguity ladder when it
fails. The course reaches these models in the
[transformers](/natural-language-processing/transformers/transformers-and-attention)
and [large language models](/natural-language-processing/transformers/large-language-models)
modules; the point here is only that the field's answer to "what is an NLP system"
has shifted from _a pipeline you build per task_ to _one model you pretrain and then
prompt_.

[^jm-intro]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed. draft), Ch. 1 — Introduction: language processing as the resolution of ambiguity that pervades every level of linguistic structure, illustrated by the multiply-ambiguous sentence "I made her duck".
[^jm-eliza]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed. draft), Ch. 2 opening: ELIZA (Weizenbaum, 1966) as a pattern-matching Rogerian-therapist chatbot built from a cascade of regular-expression substitutions, and the observation that users believed it understood them despite it modeling nothing about the world. The imitation game is A. M. Turing, "Computing Machinery and Intelligence," _Mind_ 59 (1950), and the Chinese room is J. Searle, "Minds, Brains, and Programs," _Behavioral and Brain Sciences_ 3 (1980).
[^jm-words]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed. draft), §2.2 — Words: the token/type distinction, the vocabulary $V$ and token count $N$, corpora, and Herdan's / Heaps' Law $|V| = kN^{\beta}$ relating vocabulary size to corpus size.
[^jm-zipf]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed. draft), §2.2 — Word frequencies and the observation, due to G. K. Zipf, _The Psycho-Biology of Language_ (1935) and _Human Behavior and the Principle of Least Effort_ (1949), that frequency is roughly inversely proportional to rank, $f_r \approx C/r$, producing a long tail of rare types (roughly half of them hapax legomena in the Brown corpus).
[^brown-gpt3]: **T. Brown, B. Mann, N. Ryder, et al.**, "Language Models are Few-Shot Learners," _Advances in Neural Information Processing Systems_ (NeurIPS) 33, 2020 — GPT-3, a 175-billion-parameter autoregressive language model, and the demonstration of in-context (few-shot) learning: performing a new task from a natural-language description and a few examples in the prompt, with no gradient update.
[^devlin-bert]: **J. Devlin, M.-W. Chang, K. Lee, K. Toutanova**, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," _Proceedings of NAACL-HLT_, 2019 — the masked-language-modeling pretraining objective and fine-tuning recipe that set state-of-the-art results across the GLUE benchmark, question answering, and named-entity recognition.