What Is Natural Language Processing?

A five-year-old resolves the sentence I made her duck in the time it takes to hear it, usually without noticing there was anything to resolve. A machine has to be told that the sentence has at least five readings, and that choosing among them draws on the sound stream, the words, the grammar, the meanings, and the situation all at once. Natural language processing (NLP) is the discipline of getting computers to do the resolving: to read, represent, and generate human language with enough competence to translate it, answer questions from it, transcribe it, and hold a conversation in it.

The field sits at the meeting point of linguistics, which describes the structure of language, and machine learning, which supplies the methods for fitting models to data. What makes it its own subject rather than a corner of either is the particular character of the problem: language is generated by people for people, under the assumption that the listener shares an enormous amount of unstated context, and almost every choice a speaker makes is locally ambiguous and globally resolved.

Why language is hard: ambiguity at every level

In NLP, ambiguity is not an edge case; it is the normal condition of every utterance. A sentence that seems to have one obvious meaning has that meaning only because the reader has already, without noticing, discarded the alternatives. Consider Jurafsky and Martin's example:¹

I made her duck.

This four-word sentence is ambiguous at several distinct linguistic levels, and the ambiguities compound:

• I cooked waterfowl for her. Here duck is a noun (a bird), her is the beneficiary, and made means cooked.
• I cooked waterfowl belonging to her. Same duck, but now her is a possessor.
• I created the (plaster?) duck she owns. made means created, duck is a noun.
• I caused her to quickly lower her head or body. Now duck is a verb, her is the subject of that verb, and made is causative.
• I magically turned her into a duck. made is again causative, but duck is a predicate.

Each reading turns on a decision at a different level of linguistic structure. Is duck a noun or a verb (a lexical ambiguity, in its part of speech)? Does her attach as an indirect object or as the subject of an embedded verb (a syntactic ambiguity, in the parse)? Does make mean cook, create, or cause (a semantic ambiguity, in word sense)? And in speech there is a level below all of these: the string made her and the string mader are acoustically nearly identical, so a speech recognizer faces a phonetic ambiguity before it can even commit to a sequence of words.

The reason a human never notices the ambiguity is that resolution happens by context, and context includes everything from the previous sentence to who is speaking to what a duck is. The reason a machine struggles is that all of that context has to be either supplied as data or learned from data. Every subfield of NLP can be read as an attack on one rung of this ladder: part-of-speech tagging resolves the lexical rung, parsing the syntactic one, word-sense disambiguation and semantic parsing the semantic one, and coreference and discourse the pragmatic one.

The levels of linguistic structure

The ambiguity ladder is really a view of a deeper organizing idea: language is structured in levels, each built on the one below, and NLP mirrors that stack.¹ A complete system that goes from a sound wave to an understood meaning passes through every level, whether or not it does so with explicit modules.

Each level names both a kind of structure and a family of tasks:

The classical view treated these as a pipeline: run a tokenizer, then a morphological analyzer, then a parser, then a semantic interpreter, each consuming the previous stage's output. The modern neural view, which we reach later in the course, largely dissolves the seams — a single model maps sound or text toward meaning end-to-end — but the levels remain the right way to understand what a model must implicitly get right, and the right way to diagnose where it fails.

A short history: from ELIZA to LLMs

The field's history is a story of the ambiguity problem being attacked with three successive toolkits: hand-written rules, statistics over corpora, and learned representations. Each era begins with the previous one's failures.

ELIZA and the age of patterns (1960s)

The first program most people would call a natural language system was ELIZA, written by Joseph Weizenbaum in 1966.² ELIZA imitated a Rogerian psychotherapist — a style of therapy in which the therapist mostly reflects the patient's statements back as questions — and it did so by matching patterns in the input and rewriting them into responses.

User: I am unhappy. ELIZA: DO YOU THINK COMING HERE WILL HELP YOU NOT TO BE UNHAPPY User: I need some help, that much seems certain. ELIZA: WHAT WOULD IT MEAN TO YOU IF YOU GOT SOME HELP User: Perhaps I could learn to get along with my mother. ELIZA: TELL ME MORE ABOUT YOUR FAMILY User: My mother takes care of me. ELIZA: WHO ELSE IN YOUR FAMILY TAKES CARE OF YOU

There is no understanding here. ELIZA works by a cascade of substitutions built out of the same regular expressions we study in the next lesson: a rule spots the pattern I need X and emits WHAT WOULD IT MEAN TO YOU IF YOU GOT X, swapping the pronouns so that my becomes your. The program keeps no model of the world, the conversation, or the user.

What makes ELIZA a permanent lesson is that it fooled people. Many users came to believe ELIZA genuinely understood them, and some kept believing it even after the mechanism was explained.² Weizenbaum himself was disturbed by this. The Rogerian setting is a rare genre in which a system can succeed while knowing nothing about the world — the therapist is supposed to reflect rather than inform — so ELIZA's success reveals both how much fluency pattern-matching can fake and how little that fluency has to do with comprehension. That gap, between convincing surface behavior and actual understanding, is the through-line of the whole subject, and it returns in the era of large language models.

Measuring understanding: the imitation game

ELIZA forces a question the field has never fully settled: how would you test whether a machine understands language? The oldest proposed answer predates NLP. In 1950 Alan Turing set aside the unanswerable can machines think? and replaced it with an operational imitation game: a human judge holds a text conversation with an unseen partner that is either a person or a machine, and the machine passes if the judge cannot reliably tell which is which.² The test is deliberately behavioral — it grades what the system does with language, not what happens inside it. ELIZA is the cautionary counterexample. In its narrow Rogerian genre it fooled people, yet it plainly understood nothing, which is the standing objection to a purely behavioral test: fluency can be faked, and a judge's willingness to attribute a mind is easy to exploit.

The philosophical stakes were sharpened by Searle's Chinese room argument (1980): imagine a person who speaks no Chinese, sealed in a room, following a rule book that maps incoming Chinese symbols to outgoing ones. From outside the room the answers look fluent, yet no one and nothing in the room understands Chinese — the symbols are manipulated by their shape, never connected to meaning. Searle's claim is that symbol manipulation alone, however competent, is not comprehension. Whether that argument holds against modern systems is contested; what matters for the course is the distinction it names. A system can be measured on task performance — does it translate, answer, summarize correctly? — without any claim that it understands, and almost all of NLP is evaluated the first way. The gap ELIZA opened, between fluent behavior and grounded meaning, is what the imitation game cannot see past, and it is why the field measures systems on concrete tasks with concrete metrics rather than on whether they seem to understand.

The statistical turn (1980s–2000s)

Hand-written rules do not scale. English has too many words, too many patterns, and too many exceptions for a person to enumerate, and every new domain breaks the rules tuned for the last one. The statistical turn replaced the rule-writer with the corpus: rather than tell the machine that duck is more often a noun than a verb, count it in a large body of text and let the frequencies decide. The next two lessons are the origin of this era — count word occurrences to estimate n-gram probabilities, and train a naive Bayes or logistic-regression classifier on labeled examples. Ambiguity becomes a probability distribution rather than a hard obstacle: the parser does not need the one true parse, only the most likely one given the data.

The neural and LLM era (2010s–present)

Statistical models still leaned on features a human designed — this word, that suffix, this preceding tag. The neural turn automated the features too. Words become dense vectors learned from context so that meaning becomes geometry; a network composes those vectors into the meaning of a phrase and a sentence. Then the transformer, by letting every word attend to every other, scaled cleanly with data and compute, and trained on enough text it became the large language model: one architecture that translates, summarizes, answers, and converses.

The arc of the field is a steady transfer of work from the human to the data. ELIZA's designer wrote every pattern; the statistical modeler chose every feature and let the counts fill in the weights; the neural modeler specifies an architecture and a loss and lets gradient descent discover the representation. The course follows that arc in order — from regexes and n-grams to embeddings and transformers — so that each new tool is legible as a response to the previous one's ceiling.

The task families this course covers

Underneath the applications, NLP problems fall into a small number of shapes. Naming them early makes the rest of the course navigable, because most systems are one of these shapes, or a composition of a few.

Two families do most of the structural work. Classification assigns a single label to a whole span of text, and sequence labeling assigns a label to each token in order; much of NLP reduces to one of these two. The generation tasks — translation, summarization, dialogue — share the harder shape of producing text rather than choosing from a fixed set, and they are where neural sequence models and transformers are needed.

The pipeline-to-end-to-end shift

For most of the field's history a system was a pipeline: a chain of components, each solving one level of the linguistic stack and passing its output to the next. Tokenize, then tag parts of speech, then parse, then resolve references, then extract meaning. The appeal is modularity — each stage is separately buildable and debuggable — but the cost is that errors compound. A tokenizer mistake corrupts the tagger's input, a tagging mistake corrupts the parser's, and no downstream stage can recover information an upstream stage discarded.

The neural era replaced the pipeline with end-to-end models: a single network maps the raw input toward the final output, learning whatever intermediate representations serve the task rather than being handed human-defined ones. This has been the dominant recipe on translation, speech, and question answering, where the hand-built pipeline has all but disappeared. The trade is real — end-to-end models need far more data and compute, and they are harder to inspect — but on the tasks where enough data exists, jointly optimizing every stage against one final objective wins. We keep the levels as a conceptual map even when the model no longer has separate modules for them.

Notation: tokens, types, and corpora

Before any modeling, we need to count words, and counting forces a distinction the rest of the course relies on.³ A corpus (plural corpora) is a computer-readable body of text or speech — the raw material NLP learns from. Given a corpus, take a sentence like

They picnicked by the pool, then lay back on the grass and looked at the stars.

Ignoring punctuation, this sentence has 16 tokens: the running words in order, counted with repetition. But it has only 14 types: the distinct words. The two occurrences of the are two tokens of one type. When we ask how many words is this?, the answer depends on which we mean.

We write $N$ for the number of tokens in a corpus and $V$ for its vocabulary, the set of types, with size $|V|$. These two numbers grow at very different rates. As a corpus gets larger, new tokens keep arriving long after most common types have appeared, so $N$ climbs steeply while $|V|$ climbs slowly. In a long book, by the halfway point almost every common word has appeared, so the second half adds many tokens but few genuinely new types. Their relationship is empirically a power law, Herdan's Law (also Heaps' Law),

with $k$ and $\beta$ positive constants: the vocabulary grows as a sublinear power of the token count. In real corpora the gap is large: the Brown corpus has about 1 million tokens but only ~38 thousand types, while the far larger Google n-grams corpus has ~1 trillion tokens and ~13 million types.³ The vocabulary $V$, the token count $N$, and the corpus are the three quantities every later lesson is measured against: n-gram models estimate probabilities over $V$, classifiers turn a document into counts over $V$, and embeddings assign a learned vector to each type in $V$.

Zipf's law and the long tail

The vocabulary does not just grow slowly; its frequencies are wildly uneven, and that unevenness shapes every method built on top of counts. Rank the types of a corpus from most to least frequent, giving the most common type rank $1$, the next rank $2$, and so on. Zipf's law is the empirical observation that a type's frequency is roughly inversely proportional to its rank:⁴

for a constant $C$ near the corpus size. The most frequent word appears about twice as often as the second, three times as often as the third. Take logs of both sides and the law becomes a straight line, $\log f_r\approx \log C-\log r$, with slope $-1$: a frequency-versus-rank plot on log–log axes is nearly linear.

The consequence is a long tail: a small set of function words (the, of, and) accounts for a large share of all tokens, while most of the vocabulary consists of types that appear only once or twice. In the Brown corpus about half the types are hapax legomena — words that occur exactly once. For example, in that corpus the occurs roughly 69{,}971 times, so with a corpus of about a million tokens $f_1\cdot 1\approx 70,000$; the word ranked around 35th, near at, occurs on the order of $2,000$ times, and $f_{35}\cdot 35\approx 70,000$ holds to within the noise. The product $f_r\cdot r$ stays close to constant across three orders of magnitude of rank.

Two consequences follow, and both recur through the rest of the course. First, most of the vocabulary will be rare, so a model that only works on words it has seen many times will fail on most of the words it meets — the motivation for smoothing in n-gram models and for the subword tokenization the next lesson builds. Second, the head is cheap and the tail is expensive: a few thousand common types cover most running text, but capturing the tail well is where the modeling effort goes.

The pretrain-and-prompt era

The SLP3 draft was written as the field's center of gravity moved a second time, and the move is worth stating on its own because it reframes what an NLP system is. For most of the statistical and early-neural period, you built one model per task: collect labeled data for sentiment, train a sentiment classifier; collect labeled data for translation, train a translation model. The pretraining paradigm breaks that one-model-per-task assumption. A single large network is trained once, on a generic objective over enormous unlabeled text, and the same pretrained model is then adapted to many tasks.

The pivot was BERT (Devlin et al., 2019, NAACL),⁵ which pretrained a transformer on a masked language modeling objective — predict words blanked out of a sentence — over billions of words, then fine-tuned the result with a small labeled set per downstream task. Fine-tuned BERT set new state-of-the-art numbers across a broad suite of tasks at once (the GLUE benchmark, question answering, named-entity recognition), which established the recipe: pretrain a general language representation, adapt it cheaply. The work of designing task-specific features, which the statistical era spent most of its effort on, largely disappeared into the pretrained weights.

The next step removed even the fine-tuning. GPT-3 (Brown et al., 2020, NeurIPS)⁶ scaled an autoregressive language model to 175 billion parameters and showed that a model that large, trained only to predict the next word, can perform a new task from a natural-language description and a few examples placed in its input — in-context learning, with no gradient update to the weights at all. Ask it to translate by showing three example translations and then a fourth source sentence, and it continues the pattern. The task specification moved out of the training loop and into the prompt. This is the mechanism behind the instruction-following assistants that followed; the same one architecture translates, summarizes, answers, and converses, selected among those behaviors by what you write to it.

Two caveats apply, and both trace back to ELIZA. First, next-word prediction optimizes fluency, not truth: these models produce confident, well-formed text that is sometimes simply wrong, a failure the literature calls hallucination. The gap between convincing surface behavior and grounded correctness that Weizenbaum warned about is now a first-order engineering problem, not a curiosity. Second, scale changed what is possible but not the underlying levels of linguistic structure — a large model that gets syntax and semantics right most of the time still gets them right by having implicitly learned the same distinctions this lesson laid out, and it still fails at the same rungs of the ambiguity ladder when it fails. The course reaches these models in the transformers and large language models modules; the point here is only that the field's answer to what is an NLP system has shifted from a pipeline you build per task to one model you pretrain and then prompt.