---
title: Word Senses and Disambiguation
module: Linguistic Structure
moduleNumber: 6
lessonNumber: 5
order: 605
summary: >
  A word is not an atom of meaning: "bass" names a fish, a voice, and an
  instrument, and one static embedding blurs them into a single point. This
  lesson pulls those senses apart. We define polysemy and the relations that
  organize senses — synonymy, antonymy, hyponymy, meronymy — build them into
  WordNet's synset graph, measure similarity along that graph, and then solve the
  core of word sense disambiguation: the most-frequent-sense baseline, the Lesk
  gloss-overlap algorithm, feature-based classifiers, and the nearest-neighbor
  method over BERT embeddings. WSD variants, embeddings, and evaluation follow in
  the companion lesson.
topics: [Structure]
sources:
  - book: Jurafsky
    ref: "Ch. 18 — Word Senses and WordNet; §18.1 Word Senses; §18.2 Relations Between Senses"
  - book: Jurafsky
    ref: "§18.3 WordNet; §18.4 Word Sense Disambiguation"
---

The [vector-semantics](/natural-language-processing/semantics/vector-semantics-and-embeddings)
model gives every word exactly one vector. That is its blind spot. The word
_bass_ sits at a single point in embedding space, yet it names a fish, a singing
voice, and a stringed instrument — three meanings a static vector is forced to
average into one blur. A sentence about _playing bass_ and a sentence about
_grilled bass_ pull that vector in incompatible directions, and word2vec ends up
placing it somewhere in the middle, near neither.[^jm-senses]

This lesson takes meaning apart at the level below the word: the **sense**. We
say what senses are and how a word comes to have several; we catalogue the
relations that tie senses to one another (a fish _is-a_ animal, a wheel is
_part-of_ a car); we assemble those relations into WordNet, the lexical database
that makes them machine-readable; and then we solve the central task, **word
sense disambiguation** — given a word in context, pick the sense the writer
meant.

## Word senses and polysemy

> **Definition (Word sense).** A discrete representation of one aspect of the
> meaning of a word. A word with more than one sense is **polysemous**.

Lexicographic tradition writes senses with a superscript. The word _bank_ has at
least $\text{bank}^1$, a financial institution, and $\text{bank}^2$, the sloping
land beside a river; _mouse_ has $\text{mouse}^1$ the rodent and
$\text{mouse}^2$ the pointing device.[^jm-senses] In context the intended sense
is usually obvious to a reader — "a bank can hold the investments in a custodial
account" versus "as agriculture burgeons on the east bank, the river" — but the
disambiguation the reader does automatically is what a program must be
taught to do.

The extreme case is a word like _bass_, whose noun form carries eight distinct
senses in WordNet. Laying them out shows both the scale of the ambiguity and its
internal structure — the eight senses fall into two families that a bare list
hides.

$$
% caption: The sense inventory for the noun $\textit{bass}$ splits into two
% unrelated families — musical low-pitch senses and edible-fish senses — that a
% single static embedding is forced to average into one point.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  root/.style={draw, thick, minimum width=16mm, minimum height=8mm, align=center},
  fam/.style={draw, minimum width=22mm, minimum height=8mm, align=center, font=\scriptsize},
  s/.style={draw, minimum width=30mm, minimum height=7mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[root, draw=acc, text=acc] (r) at (0,0) {bass (noun)};
  \node[fam] (mus) at (-3.4,-1.7) {MUSIC / PITCH};
  \node[fam] (fish) at (3.4,-1.7) {ANIMAL / FOOD};
  \draw[->, acc] (r) -- (mus);
  \draw[->, red] (r) -- (fish);
  \node[s] (m1) at (-3.4,-2.9) {bass-1: lowest musical range};
  \node[s] (m2) at (-3.4,-3.7) {bass-3: lowest male voice};
  \node[s] (m3) at (-3.4,-4.5) {bass-7: lowest instrument};
  \node[s] (f1) at (3.4,-2.9) {bass-4: sea-bass f\/lesh};
  \node[s] (f2) at (3.4,-3.7) {bass-5: freshwater f\/ish};
  \node[s] (f3) at (3.4,-4.5) {bass-8: edible marine f\/ish};
  \draw[acc] (mus.south) -- (m1.north);
  \draw[acc] (mus.west) to[out=-90, in=180] (m2.west);
  \draw[acc] (mus.west) to[out=-90, in=180] (m3.west);
  \draw[red] (fish.south) -- (f1.north);
  \draw[red] (fish.east) to[out=-90, in=0] (f2.east);
  \draw[red] (fish.east) to[out=-90, in=0] (f3.east);
\end{tikzpicture}
$$

The two families are pronounced differently — [b ey s] for the music senses, [b
ae s] for the fish — yet share a spelling, which is why WordNet lists them under
one lemma. When two senses of a word are historically and semantically unrelated
like this, the traditional name is **homonymy**; when they are related in some
structured way (the _bank_ that is an institution versus the _building_ that
houses it) the name is **structured polysemy**. We follow Jurafsky & Martin in using
_polysemy_ for any sense ambiguity.[^jm-poly]

### How many senses does a word have?

Dictionaries slice meaning finely; computation usually needs coarser cuts. What
counts as a genuinely distinct sense? Three heuristics apply. Two uses are
distinct if they have **independent truth conditions**, different **syntactic
behavior**, or antagonistic meanings. The sharpest test is **zeugma** —
conjoining two uses in one sentence and listening for oddness:

> (a) Which of those flights _serve_ breakfast?
> (b) Does Air France _serve_ Philadelphia?
> (c) ?Does Air France _serve_ breakfast and Philadelphia?

The strangeness of (c) shows that no single sense of _serve_ covers both
objects, which is evidence that _serve_ has (at least) two senses here. Because
dictionary senses are often too fine for a model to tell apart, a common
preprocessing step clusters them into coarser groups — a theme that returns when
we reach the Word-in-Context task.

## Relations between senses

Senses do not float independently; they are organized by a small set of
relations, and much of a thesaurus's usefulness is in recording them.[^jm-rel]

**Synonymy.** Two senses (of two different lemmas) that are identical or nearly
identical: _couch_/_sofa_, _car_/_automobile_, _filbert_/_hazelnut_. Synonymy is
a relation between _senses_, not words — _big_ and _large_ are synonymous in "how
_big_/_large_ is that plane?" but not in "a _big_ sister," because _big_ has a
grown-up sense that _large_ lacks.

**Antonymy.** Senses with opposite meanings: _long_/_short_, _fast_/_slow_,
_up_/_down_, _rise_/_fall_. Antonyms differ completely along one axis — position
on a scale, or direction of change (**reversives** like _rise_/_fall_) — while
sharing almost everything else, and that near-total overlap is what makes them
hard to separate automatically. In a word2vec space the nearest neighbor of _up_ is often _down_,
because the two occur in near-identical contexts; a thesaurus that records the
opposition is what keeps a dialogue agent from turning the music _down_ when
asked to turn it _up_.

**Taxonomic relations.** A sense is a **hyponym** of another if it is more
specific: _car_ is a hyponym of _vehicle_, _mango_ of _fruit_, _dog_ of _animal_.
The reverse is **hypernymy** (also **superordinate**): _vehicle_ is a hypernym of
_car_. Formally, $A$ is a hyponym of $B$ when everything that is an $A$ is also a
$B$ — being an $A$ **entails** being a $B$:

$$
A \sqsubseteq B \iff \forall x\;\big(A(x) \Rightarrow B(x)\big).
$$

The relation is transitive, so hyponymy chains stack into an **IS-A hierarchy**:
$\text{dog} \sqsubseteq \text{mammal} \sqsubseteq \text{animal} \sqsubseteq
\text{organism}$.

**Meronymy.** The part–whole relation: a _wheel_ is a **meronym** of _car_, and
_car_ is a **holonym** of _wheel_; a _leg_ is a meronym of _chair_. Meronymy is
orthogonal to the IS-A axis — a wheel is a _part_ of a car, not a _kind_ of car.

$$
% caption: Two orthogonal axes organize senses: the vertical IS-A
% (hypernym/hyponym) hierarchy runs from general to specific, while the
% horizontal PART-OF (meronymy) relation ties a whole to its components.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  n/.style={draw, minimum width=17mm, minimum height=7mm, align=center, font=\scriptsize},
  p/.style={draw, minimum width=15mm, minimum height=6mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % IS-A chain (vertical)
  \node[n, draw=acc, text=acc] (veh) at (0,3.0)  {vehicle};
  \node[n] (car) at (0,1.5)  {car};
  \node[n] (sport) at (0,0.0) {sports car};
  \draw[->, acc, thick] (car) -- (veh) node[midway, right, font=\scriptsize] {IS-A};
  \draw[->, acc, thick] (sport) -- (car) node[midway, right, font=\scriptsize] {IS-A};
  % PART-OF (horizontal, off the car node)
  \node[p] (wheel) at (4.2,2.1) {wheel};
  \node[p] (engine) at (4.2,0.9) {engine};
  \draw[->, red, thick] (wheel) -- (car) node[midway, above, font=\scriptsize] {PART-OF};
  \draw[->, red, thick] (engine) -- (car) node[midway, below, font=\scriptsize] {PART-OF};
  % axis labels
  \node[acc, anchor=south, font=\scriptsize] at (-2.0,1.5) {general};
  \node[acc, anchor=north, font=\scriptsize] at (-2.0,1.4) {speci\/f\/ic};
  \draw[acc, ->] (-2.0,0.2) -- (-2.0,2.8);
\end{tikzpicture}
$$

Structured polysemy adds one more layer: **metonymy**, using one aspect of an
entity to stand for another. BUILDING $\leftrightarrow$ ORGANIZATION (the _bank_
on the corner versus the _bank_ that holds your account), AUTHOR $\leftrightarrow$
WORKS ("I love Jane Austen"), and FRUIT-TREE $\leftrightarrow$ FRUIT ("plums have
blossoms" / "I ate a plum") are systematic patterns that recur across whole
classes of words.

## WordNet

> **Definition (WordNet).** A large lexical database of English (with versions in
> many languages) that records lemmas, their senses, and the relations among
> those senses. It has three separate databases — nouns, verbs, and
> adjectives/adverbs — and excludes closed-class words.

WordNet 3.0 holds 117,798 nouns, 11,529 verbs, 22,479 adjectives, and 4,481
adverbs.[^jm-wn] The average noun has 1.23 senses and the average verb 2.16 — verbs
are more polysemous, which will matter for disambiguation.

The primitive unit is not the sense but the **synset** (synonym set): the set of
near-synonymous senses that all express one concept, together with a **gloss** (a
dictionary-style definition) and often example sentences. One synset for the
concept "a person easy to take advantage of" is

$$
\{\text{chump}^1,\ \text{fool}^2,\ \text{gull}^1,\ \text{mark}^9,\ \text{patsy}^1,\ \text{sucker}^1\},
$$

with the gloss _a person who is gullible and easy to take advantage of._ The
gloss is a property of the synset, so every sense in it inherits the same
definition. It is **synsets, not words**, that participate in the
lexical relations — hypernymy connects one synset to another, not one string to
another.

$$
% caption: A fragment of WordNet centered on the synset $\{\textit{car},
% \textit{auto}\}$: each box is a synset with a gloss; solid blue edges are IS-A
% (hypernym) links and dashed red edges are PART-OF (meronym) links.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  syn/.style={draw, minimum width=30mm, minimum height=9mm, align=center, font=\scriptsize},
  part/.style={draw, minimum width=20mm, minimum height=7mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[syn] (veh)  at (0,3.2)  {synset: vehicle\\a conveyance};
  \node[syn, draw=acc, text=acc, thick] (car) at (0,1.4)  {synset: car, auto\\a wheeled motor vehicle};
  \node[syn] (cab)  at (-3.0,-0.4) {synset: cab, taxi\\a car for hire};
  \node[syn] (sedan) at (0.8,-0.4) {synset: sedan\\an enclosed car};
  \node[part] (wheel) at (4.4,1.9) {synset: wheel};
  \node[part] (engine) at (4.4,0.6) {synset: engine};
  \draw[->, acc, thick] (car) -- (veh) node[midway, right, font=\scriptsize] {hypernym};
  \draw[->, acc, thick] (cab) -- (car);
  \draw[->, acc, thick] (sedan) -- (car);
  \draw[->, red, thick, dashed] (wheel) -- (car);
  \draw[->, red, thick, dashed] (engine) -- (car) node[midway, below, font=\scriptsize] {meronym};
\end{tikzpicture}
$$

Following hypernym links upward produces a chain from a specific synset to the
most abstract. The chains for two senses of _bass_ show how disambiguation could
work in principle — the senses live in entirely separate regions of the
hierarchy and only meet at the root.

$$
% caption: Hypernym chains for two senses of $\textit{bass}$. The singer sense
% ($\text{bass}^3$) climbs through PERSON, the instrument sense
% ($\text{bass}^7$) through ARTIFACT; they meet only at the abstract root
% ENTITY.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  n/.style={draw, minimum width=32mm, minimum height=6.5mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[n, draw=acc, text=acc] (b3) at (-2.4,4.2) {bass-3: adult male singer};
  \node[n] (sing) at (-2.4,3.2) {singer, vocalist};
  \node[n] (mus)  at (-2.4,2.2) {musician, performer};
  \node[n] (pers) at (-2.4,1.2) {person, individual};
  \node[n, draw=red, text=red] (b7) at (2.6,4.2) {bass-7: lowest instrument};
  \node[n] (inst) at (2.6,3.2) {musical instrument};
  \node[n] (dev)  at (2.6,2.2) {device};
  \node[n] (art)  at (2.6,1.2) {artifact};
  \node[n, draw, thick] (ent) at (0.1,0.0) {entity};
  \draw[->, acc] (b3) -- (sing); \draw[->, acc] (sing) -- (mus);
  \draw[->, acc] (mus) -- (pers); \draw[->, acc] (pers) -- (ent);
  \draw[->, red] (b7) -- (inst); \draw[->, red] (inst) -- (dev);
  \draw[->, red] (dev) -- (art); \draw[->, red] (art) -- (ent);
\end{tikzpicture}
$$

WordNet also tags each synset with a **supersense** — one of 26 coarse
lexicographic categories for nouns (ANIMAL, ARTIFACT, FOOD, PERSON, …) plus 15
for verbs — useful when the full sense inventory is too fine-grained. Verbs get
their own relations: **troponymy** (a manner-of-doing subtype: _stroll_ is a
troponym of _walk_) plays the role hyponymy plays for nouns, and **entailment**
links _snore_ to _sleep_.

## Word similarity from the hierarchy

Because WordNet is a graph, the distance between two senses in it is a usable
measure of similarity — a thesaurus-based alternative to the embedding cosine of
the previous chapter. The simplest version is **path length**.[^jm-sim]

> **Definition (Path similarity).** Let $\text{pathlen}(c_1, c_2)$ be the number
> of edges on the shortest hypernym/hyponym path between synsets $c_1$ and $c_2$.
> Then $\text{sim}_{\text{path}}(c_1, c_2) = \dfrac{1}{1 + \text{pathlen}(c_1,
> c_2)}$, so identical senses score $1$ and distance shrinks similarity toward $0$.

$$
% caption: Path-based similarity counts edges to the lowest common subsumer:
% $\textit{nickel}$ and $\textit{dime}$ are close (both under COIN), while
% $\textit{nickel}$ and $\textit{money}$ are farther, separated by more edges.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  n/.style={draw, minimum width=20mm, minimum height=6.5mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[n] (std)   at (0,4.2) {standard};
  \node[n] (med)   at (0,3.0) {medium of exchange};
  \node[n] (money) at (-2.6,1.8) {money};
  \node[n, draw=acc, text=acc] (coin)  at (2.2,1.8) {coin};
  \node[n] (cash)  at (-2.6,0.5) {cash};
  \node[n] (nickel) at (0.9,0.5) {nickel};
  \node[n] (dime)  at (3.6,0.5) {dime};
  \draw (med) -- (std);
  \draw (money) -- (med); \draw (coin) -- (med);
  \draw (cash) -- (money);
  \draw[acc, thick] (nickel) -- (coin); \draw[acc, thick] (dime) -- (coin);
  \node[acc, anchor=west, font=\scriptsize] at (3.4,1.8) {lowest common subsumer};
\end{tikzpicture}
$$

Path length has a flaw: it treats every edge as equal, but the step from
_entity_ to _physical-object_ near the root spans a huge conceptual gap, while
_nickel_ to _dime_ near the leaves is tiny. **Information-content** measures fix
this by weighting nodes with corpus probabilities. Let $P(c)$ be the probability
that a randomly drawn word is an instance of synset $c$ (or any of its
descendants), so $P$ increases monotonically toward the root, where $P(\text{root})
= 1$. The information content of a synset is $\text{IC}(c) = -\log P(c)$, and the
**lowest common subsumer** $\text{LCS}(c_1, c_2)$ is the most specific synset that
is a hypernym of both.

> **Definition (Resnik similarity).** The similarity of two senses is the
> information content of their lowest common subsumer:
> $\text{sim}_{\text{Resnik}}(c_1, c_2) = -\log P(\text{LCS}(c_1, c_2))$. A more
> specific (rarer, higher-IC) shared ancestor means more shared meaning.

Resnik ignores how far below the LCS each sense lies. **Lin similarity**
normalizes by the information content of the two senses themselves, so it
rewards a specific common ancestor _and_ penalizes senses that are individually
very specific:

$$
\text{sim}_{\text{Lin}}(c_1, c_2)
  = \frac{2 \,\log P\!\big(\text{LCS}(c_1, c_2)\big)}{\log P(c_1) + \log P(c_2)}.
$$

The numerator is the shared information, the denominator the total; the ratio
lands in $[0, 1]$, equalling $1$ when the two senses coincide. Both measures need
only WordNet plus a corpus to estimate $P(c)$ — no labeled sense data at all.

## Word sense disambiguation

> **Definition (Word sense disambiguation).** Given a word in context and a fixed
> inventory of its possible senses, output the sense intended in that context.

The sense inventory depends on the application: WordNet senses for general text,
supersenses for a coarser grain, MeSH headings for medical indexing, or the set
of foreign-language translations for machine translation (the animal _bat_ is
Spanish _murciélago_, the baseball _bat_ is _bate_). Two task shapes recur. In a
**lexical sample** task a small fixed set of target words is disambiguated, and
simple supervised classifiers do very well. In the harder **all-words** task
every content word in a text must be tagged — like part-of-speech tagging, but
with an enormous tag set, since every lemma carries its own sense inventory, so
data sparseness bites hard.

$$
% caption: The all-words WSD task maps each content word $x_i$ (input) to one
% WordNet sense $y_i$ (output). Words like $\textit{guitar}$ have a single sense;
% $\textit{bass}$ and $\textit{player}$ are ambiguous, and context selects among
% their senses.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  w/.style={draw, minimum width=15mm, minimum height=6.5mm, align=center, font=\scriptsize},
  s/.style={draw, minimum width=17mm, minimum height=6mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % input words
  \node[w, draw=acc, text=acc] (guitar) at (0,0)   {guitar};
  \node[w, draw=acc, text=acc] (bass)   at (0,-1.5) {bass};
  \node[w, draw=acc, text=acc] (player) at (0,-3.0) {player};
  % candidate senses
  \node[s] (g1) at (4.2,0)    {guitar-1: instrument};
  \node[s] (b1) at (4.2,-1.0) {bass-1: low range};
  \node[s] (b7) at (4.2,-1.9) {bass-7: instrument};
  \node[s] (p2) at (4.2,-2.8) {player-2: musician};
  \node[s] (p3) at (4.2,-3.7) {player-3: actor};
  % correct choices (blue), distractors (black)
  \draw[->, acc, thick] (guitar) -- (g1);
  \draw[->, acc, thick] (bass) -- (b7);
  \draw[->, black] (bass) -- (b1);
  \draw[->, acc, thick] (player) -- (p2);
  \draw[->, black] (player) -- (p3);
  \node[acc, anchor=south, font=\scriptsize] at (0,0.5) {input words};
  \node[anchor=south, font=\scriptsize] at (4.2,0.5) {WordNet senses};
\end{tikzpicture}
$$

Supervised WSD trains on a **semantic concordance** — a corpus where each
open-class word is hand-tagged with its WordNet sense. The largest is **SemCor**,
a 226,036-word subset of the Brown Corpus. A SemCor sentence looks like

$$
\text{You will find}^{9}_{v}\; \text{that avocado}^{1}_{n}\; \text{is}^{1}_{v}\;
\text{unlike}^{1}_{j}\; \text{other}^{1}_{j}\; \text{fruit}^{1}_{n}\ldots
$$

where the superscript is the WordNet sense number and the subscript the part of
speech. For _fruit_ the task is to pick $\text{fruit}^1_n$ (the ripened body of a
seed plant) over $\text{fruit}^2_n$ (an amount of product) and $\text{fruit}^3_n$
(the consequence of an action). Systems are evaluated **intrinsically**, by $F_1$
against the held-out gold tags.

### Baselines

Two baselines set the bar. The **most-frequent-sense** baseline picks, for every
occurrence, the sense that was most common in a labeled corpus. It performs
well for how little it assumes, and in WordNet it is trivial to apply, because WordNet
orders each word's senses from most to least frequent — the most-frequent sense
is just **sense 1**. Any real system must beat "always guess sense 1" to justify
itself. The **one-sense-per-discourse** heuristic observes that a word repeated
within one document usually keeps the same sense throughout; it holds best for
coarse, unrelated senses and is used as a soft bias rather than a standalone
baseline.

### The Lesk algorithm

Labeled corpora are expensive, so **knowledge-based** methods that use only a
dictionary are valuable. The oldest is **Lesk**, which chooses the sense whose
gloss shares the most words with the target's context.[^jm-lesk]

```algorithm
caption: $\textsc{Simplified-Lesk}(word, sentence)$ — pick the sense whose gloss best overlaps the context
$best\text{-}sense \gets$ most frequent sense for $word$
$max\text{-}overlap \gets 0$
$context \gets$ set of words in $sentence$
for each $sense$ in senses of $word$ do
  $signature \gets$ set of words in the gloss and examples of $sense$
  $overlap \gets \textsc{Compute-Overlap}(signature, context)$ // count shared non-stopwords
  if $overlap > max\text{-}overlap$ then
    $max\text{-}overlap \gets overlap$
    $best\text{-}sense \gets sense$
return $best\text{-}sense$
```

The $\textsc{Compute-Overlap}$ function counts shared words, ignoring stopwords.
For example, run it on _bank_ in

> The **bank** can guarantee _deposits_ will eventually cover future tuition
> costs because it invests in adjustable-rate _mortgage_ securities.

against the two WordNet senses.

$$
% caption: Simplified Lesk on $\textit{bank}$: the context shares
% $\{\textit{deposits}, \textit{mortgage}\}$ with the gloss of $\text{bank}^1$
% (overlap $2$) and nothing with $\text{bank}^2$ (overlap $0$), so
% $\text{bank}^1$ wins.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  gl/.style={draw, minimum width=52mm, minimum height=13mm, align=left, font=\scriptsize, inner sep=4pt},
  sc/.style={draw, circle, minimum size=8mm, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[gl, draw=acc, text=acc] (g1) at (0,1.2)
    {bank-1: f\/inancial institution that accepts\\
     \textbf{deposits} and channels money into\\
     lending; ex. holds the \textbf{mortgage} on my home};
  \node[gl] (g2) at (0,-1.6)
    {bank-2: sloping land, especially the slope\\
     beside a body of water; ex. pulled the\\
     canoe up on the bank of the river};
  \node[sc, draw=acc, text=acc] (o1) at (4.4,1.2) {2};
  \node[sc] (o2) at (4.4,-1.6) {0};
  \draw[->, acc, thick] (g1) -- (o1) node[midway, above, font=\scriptsize] {overlap};
  \draw[->, black] (g2) -- (o2);
  \node[acc, anchor=west, font=\scriptsize] at (5.0,1.2) {chosen};
\end{tikzpicture}
$$

Sense $\text{bank}^1$ overlaps the context on _deposits_ and _mortgage_ (overlap
2); $\text{bank}^2$ overlaps on nothing (overlap 0); Lesk returns $\text{bank}^1$,
correctly.

The _bank_ case is easy because one sense scores two and the other zero. A second
trace shows what happens when the margins are thin and why the raw count needs the
IDF fix. Disambiguate _cone_ in

> The **pine cone** dropped by the tree fell onto the forest floor near a cluster of
> spruce and fir seedlings.

against three WordNet senses, each with the gloss reduced to its content words after
stoplist removal:

- $\text{cone}^1$ — _seed-bearing organ of a **pine**, **fir**, or other conifer_ — content set $\{\textit{seed}, \textit{pine}, \textit{fir}, \textit{conifer}\}$.
- $\text{cone}^2$ — _a **cone**-shaped **cell** in the retina of the eye, sensitive to color_ — content set $\{\textit{cell}, \textit{retina}, \textit{eye}, \textit{color}\}$.
- $\text{cone}^3$ — _a **solid** with a **circular** base tapering to a **point**_ — content set $\{\textit{solid}, \textit{circular}, \textit{base}, \textit{point}\}$.

The context content words (after stoplist) are $\{\textit{pine}, \textit{dropped},
\textit{tree}, \textit{fell}, \textit{forest}, \textit{floor}, \textit{cluster},
\textit{spruce}, \textit{fir}, \textit{seedlings}\}$. Computing raw overlap sense by
sense:

$$
% caption: A three-way Simplified Lesk trace on $\textit{cone}$. The conifer sense
% $\text{cone}^1$ overlaps the context on $\textit{pine}$ and $\textit{fir}$ (overlap
% $2$); the retina and geometry senses overlap on nothing (overlap $0$), so
% $\text{cone}^1$ wins.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  gl/.style={draw, minimum width=48mm, minimum height=11mm, align=left, font=\scriptsize, inner sep=4pt},
  sc/.style={draw, circle, minimum size=8mm, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[gl, draw=acc, text=acc] (g1) at (0,2.4)
    {cone-1 (conifer): seed pine\\f\/ir conifer};
  \node[gl] (g2) at (0,0.0)
    {cone-2 (retina): cell retina\\eye color};
  \node[gl] (g3) at (0,-2.4)
    {cone-3 (geometry): solid\\circular base point};
  \node[sc, draw=acc, text=acc] (o1) at (4.6,2.4) {2};
  \node[sc] (o2) at (4.6,0.0)  {0};
  \node[sc] (o3) at (4.6,-2.4) {0};
  \draw[->, acc, thick] (g1) -- (o1) node[midway, above, font=\scriptsize] {overlap};
  \draw[->, black] (g2) -- (o2);
  \draw[->, black] (g3) -- (o3);
  \node[acc, anchor=west, font=\scriptsize] at (5.2,2.4) {chosen};
\end{tikzpicture}
$$

$\text{cone}^1$ shares _pine_ and _fir_ with the context, overlap 2; the retina and
geometry senses share nothing, overlap 0. Lesk picks the conifer sense. The
tie-break rule matters when two senses draw: the algorithm keeps its running
`best-sense`, initialized to the most frequent sense, and only replaces it on a
_strict_ improvement (`overlap > max-overlap`), so a later sense that merely ties the
current best is rejected and the frequency prior breaks the tie.

The raw count also overweights common words. Suppose the context had matched a sense
on the word _thing_ as well as on _pine_. Both count $1$ toward the overlap, yet _pine_
is far more diagnostic of the conifer sense than _thing_ is of anything. The IDF fix
weights each shared word by $\text{idf}(w) = \log(N / \text{df}(w))$, where $\text{df}(w)$
is the number of glosses (or documents) containing $w$: _thing_, appearing in
thousands of glosses, contributes almost nothing, while _pine_, appearing in a
handful, dominates the score. The overlap becomes a sum of IDF weights rather than a
bare count, and the diagnostic words carry the decision.

Extensions improve further on the count: best-performing is to replace word overlap
with the cosine between an embedding of the gloss and an embedding of the context,
which catches _deposits_/_savings_ matches that exact overlap misses — the precursor
of the gloss-aware neural methods a later section takes up.

### Feature-based WSD

Before contextual embeddings, the strongest supervised systems were plain
classifiers over hand-built features of the surrounding words. The IMS system
feeds an SVM three feature families: **part-of-speech tags** in a window around
the target, **collocations** (specific words or $n$-grams at specific offsets),
and a distance-weighted **average of the neighbors' embeddings**. For _bass_ in
"An electric guitar and _bass_ player stand off to one side," a two-word window
yields a vector like

$$
[\,\text{guitar}, \text{NN}, \text{and}, \text{CC}, \text{player}, \text{NN}, \text{stand}, \text{VB},\ \ldots,\ g(E)\,],
$$

pairing each neighbor with its tag, adding the bigram collocations _guitar-and_
and _player-stand_, and appending the pooled embedding $g$. These remain within a
point or two of neural systems.

### Contextual-embedding WSD

The [vector-semantics](/natural-language-processing/semantics/vector-semantics-and-embeddings)
problem — one vector per word — is the one a **contextual** embedding from a
transformer solves: BERT gives a _different_ vector to _bass_ in "grilled bass"
than in "bass guitar," because each token's representation is a function of its
whole sentence. That reopens WSD as a nearest-neighbor problem.[^jm-nn]

At training time, run every SemCor sentence through BERT and collect a contextual
vector $v_i$ for each labeled token (pooling the last four layers is common).
Then, for each sense $s$, average the vectors of all its tagged tokens into a
single **sense embedding**:

$$
v_s = \frac{1}{n} \sum_{i \,\in\, \text{tokens}(s)} v_i.
$$

At test time, embed the target token $t$ in its context and return the sense whose
embedding is closest by cosine:

$$
\text{sense}(t) = \argmax_{s \,\in\, \text{senses}(t)} \ \cos(t, v_s).
$$

$$
% caption: Nearest-neighbor WSD. Each sense of $\textit{find}$ has a precomputed
% sense embedding (blue); the target token $\textit{found}$ is encoded in its
% sentence and assigned the sense whose embedding is closest by cosine —
% $\text{find}^9$ here.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tok/.style={draw, minimum width=12mm, minimum height=6mm, font=\scriptsize},
  enc/.style={draw, thick, minimum width=58mm, minimum height=7mm, align=center},
  se/.style={circle, draw=acc, fill=acc!8, minimum size=8mm, font=\scriptsize, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % sentence
  \node[tok] (t1) at (-3.2,3.4) {I};
  \node[tok] (t2) at (-1.6,3.4) {found};
  \node[tok] (t3) at (0.0,3.4)  {the};
  \node[tok] (t4) at (1.6,3.4)  {jar};
  \node[tok] (t5) at (3.2,3.4)  {empty};
  \node[enc, draw=acc, text=acc] (enc) at (0,2.1) {ENCODER (BERT)};
  \foreach \t in {t1,t2,t3,t4,t5} \draw[->, black] (\t) -- (enc);
  % target contextual embedding
  \node[circle, draw=red, fill=red!8, minimum size=8mm, font=\scriptsize, inner sep=0pt] (found) at (-1.6,0.7) {found};
  \draw[->, red, thick] (enc.south -| found) -- (found);
  % candidate sense embeddings
  \node[se] (f4) at (1.4,0.9)  {f\/ind-4};
  \node[se] (f5) at (2.9,0.3)  {f\/ind-5};
  \node[se] (f9) at (2.1,-0.9) {f\/ind-9};
  % cosine comparisons; nearest = find-9
  \draw[red!55, dashed] (found) -- (f4);
  \draw[red!55, dashed] (found) -- (f5);
  \draw[red, thick, ->] (found) -- (f9) node[midway, below, font=\scriptsize] {max cosine};
\end{tikzpicture}
$$

This ties WSD directly to the transformer machinery of
[fine-tuning and prompting](/natural-language-processing/transformers/fine-tuning-and-prompting):
the encoder is a pretrained BERT, and no task-specific training is needed beyond
averaging labeled vectors. The gap it leaves is coverage — SemCor tags only a
sliver of WordNet's senses, so many senses have no $v_s$. The fix imputes the
missing embeddings **bottom-up** through the taxonomy: give a synset the average
of its sense embeddings, a hypernym the average of its synsets, and a supersense
the average of all its synsets. Because every supersense has _some_ SemCor data,
the back-off is guaranteed to reach an embedding eventually — coarse, but never
empty.

$$
% caption: Imputing a missing sense embedding by backing off up the WordNet
% taxonomy: if the sense itself has no labeled tokens, average its synset
% siblings; failing that, its hypernym's synsets; failing that, its supersense.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  lvl/.style={draw, minimum width=30mm, minimum height=7mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[lvl] (sup)  at (0,3.3) {supersense (coarse)};
  \node[lvl] (hyp)  at (0,2.0) {hypernym synsets};
  \node[lvl] (syn)  at (0,0.7) {synset siblings};
  \node[lvl, draw=acc, text=acc] (miss) at (0,-0.6) {missing sense};
  \draw[->, acc, thick] (syn) -- (miss) node[midway, right, font=\scriptsize] {try f\/irst};
  \draw[->, black] (hyp) -- (syn) node[midway, right, font=\scriptsize] {else};
  \draw[->, black] (sup) -- (hyp) node[midway, right, font=\scriptsize] {else};
\end{tikzpicture}
$$

## Where this continues

We have taken meaning apart below the word. Senses are the discrete meanings a form
carries; the sense relations — synonymy, antonymy, hyponymy, meronymy — organize them
into WordNet's synset graph; path- and information-content measures read similarity
off that graph; and word sense disambiguation picks the intended sense in context,
from the most-frequent-sense baseline and Lesk gloss overlap to feature classifiers
and the nearest-neighbor method over contextual BERT embeddings.

That leaves the variants and the loose ends: the sense-inventory-free Word-in-Context
task, retrofitting embeddings to a thesaurus, discovering senses without a fixed
inventory (word sense induction), the gloss-aware and bi-encoder neural systems, and
evaluation. They continue in
[WSD in practice and word sense induction](/natural-language-processing/linguistic-structure/wsd-in-practice-and-induction).

[^jm-senses]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), §18.1 — Word Senses: the definition of a word sense, the superscript notation, and glosses as human-readable but circular sense definitions.
[^jm-poly]: **Jurafsky & Martin**, §18.1 — the footnote distinguishing homonymy (unrelated senses) from structured polysemy (related senses), and the choice to use _polysemy_ for any sense ambiguity; §18.1.2 for the zeugma test and how many senses a word has.
[^jm-rel]: **Jurafsky & Martin**, §18.2 — Relations Between Senses: synonymy, antonymy and reversives, hyponymy/hypernymy with the entailment definition and IS-A transitivity, meronymy/holonymy, and structured polysemy / metonymy.
[^jm-wn]: **Jurafsky & Martin**, §18.3 — WordNet: the three databases and their sizes, synsets and glosses as the primitive unit, hypernym chains for _bass_, supersenses, and the noun/verb relation tables (troponymy, entailment).
[^jm-sim]: **Jurafsky & Martin**, §18.3–18.4 and Ch. 6 — thesaurus-based word similarity built on the WordNet hierarchy: path length, information content $-\log P(c)$, the lowest common subsumer, and the Resnik and Lin similarity measures.
[^jm-lesk]: **Jurafsky & Martin**, §18.5.2 — the Lesk algorithm: the Simplified Lesk procedure, the _bank_ gloss-overlap worked example, and extensions (IDF weighting, embedding-cosine overlap).
[^jm-nn]: **Jurafsky & Martin**, §18.4.2 — the nearest-neighbor WSD algorithm with contextual embeddings: sense embeddings as averaged BERT vectors, the argmax-cosine decision rule, and bottom-up imputation of missing senses through the WordNet taxonomy.