Word Senses and Disambiguation

The vector-semantics model gives every word exactly one vector. That is its blind spot. The word bass sits at a single point in embedding space, yet it names a fish, a singing voice, and a stringed instrument — three meanings a static vector is forced to average into one blur. A sentence about playing bass and a sentence about grilled bass pull that vector in incompatible directions, and word2vec ends up placing it somewhere in the middle, near neither.¹

This lesson takes meaning apart at the level below the word: the sense. We say what senses are and how a word comes to have several; we catalogue the relations that tie senses to one another (a fish is-a animal, a wheel is part-of a car); we assemble those relations into WordNet, the lexical database that makes them machine-readable; and then we solve the central task, word sense disambiguation — given a word in context, pick the sense the writer meant.

Word senses and polysemy

Lexicographic tradition writes senses with a superscript. The word bank has at least ${\text{bank}}^1$, a financial institution, and ${\text{bank}}^2$, the sloping land beside a river; mouse has ${\text{mouse}}^1$ the rodent and ${\text{mouse}}^2$ the pointing device.¹ In context the intended sense is usually obvious to a reader — a bank can hold the investments in a custodial account versus as agriculture burgeons on the east bank, the river — but the disambiguation the reader does automatically is what a program must be taught to do.

The extreme case is a word like bass, whose noun form carries eight distinct senses in WordNet. Laying them out shows both the scale of the ambiguity and its internal structure — the eight senses fall into two families that a bare list hides.

The two families are pronounced differently — b ey s for the music senses, b ae s for the fish — yet share a spelling, which is why WordNet lists them under one lemma. When two senses of a word are historically and semantically unrelated like this, the traditional name is homonymy; when they are related in some structured way (the bank that is an institution versus the building that houses it) the name is structured polysemy. We follow Jurafsky & Martin in using polysemy for any sense ambiguity.²

How many senses does a word have?

Dictionaries slice meaning finely; computation usually needs coarser cuts. What counts as a genuinely distinct sense? Three heuristics apply. Two uses are distinct if they have independent truth conditions, different syntactic behavior, or antagonistic meanings. The sharpest test is zeugma — conjoining two uses in one sentence and listening for oddness:

(a) Which of those flights serve breakfast? (b) Does Air France serve Philadelphia? (c) ?Does Air France serve breakfast and Philadelphia?

The strangeness of (c) shows that no single sense of serve covers both objects, which is evidence that serve has (at least) two senses here. Because dictionary senses are often too fine for a model to tell apart, a common preprocessing step clusters them into coarser groups — a theme that returns when we reach the Word-in-Context task.

Relations between senses

Senses do not float independently; they are organized by a small set of relations, and much of a thesaurus's usefulness is in recording them.³

Synonymy. Two senses (of two different lemmas) that are identical or nearly identical: couch/sofa, car/automobile, filbert/hazelnut. Synonymy is a relation between senses, not words — big and large are synonymous in how big/large is that plane? but not in a big sister, because big has a grown-up sense that large lacks.

Antonymy. Senses with opposite meanings: long/short, fast/slow, up/down, rise/fall. Antonyms differ completely along one axis — position on a scale, or direction of change (reversives like rise/fall) — while sharing almost everything else, and that near-total overlap is what makes them hard to separate automatically. In a word2vec space the nearest neighbor of up is often down, because the two occur in near-identical contexts; a thesaurus that records the opposition is what keeps a dialogue agent from turning the music down when asked to turn it up.

Taxonomic relations. A sense is a hyponym of another if it is more specific: car is a hyponym of vehicle, mango of fruit, dog of animal. The reverse is hypernymy (also superordinate): vehicle is a hypernym of car. Formally, $A$ is a hyponym of $B$ when everything that is an $A$ is also a $B$ — being an $A$ entails being a $B$:

The relation is transitive, so hyponymy chains stack into an IS-A hierarchy: $\text{dog}⊑\text{mammal}⊑\text{animal}⊑\text{organism}$.

Meronymy. The part–whole relation: a wheel is a meronym of car, and car is a holonym of wheel; a leg is a meronym of chair. Meronymy is orthogonal to the IS-A axis — a wheel is a part of a car, not a kind of car.

Structured polysemy adds one more layer: metonymy, using one aspect of an entity to stand for another. BUILDING $\leftrightarrow$ ORGANIZATION (the bank on the corner versus the bank that holds your account), AUTHOR $\leftrightarrow$ WORKS (I love Jane Austen), and FRUIT-TREE $\leftrightarrow$ FRUIT (plums have blossoms / I ate a plum) are systematic patterns that recur across whole classes of words.

WordNet

WordNet 3.0 holds 117,798 nouns, 11,529 verbs, 22,479 adjectives, and 4,481 adverbs.⁴ The average noun has 1.23 senses and the average verb 2.16 — verbs are more polysemous, which will matter for disambiguation.

The primitive unit is not the sense but the synset (synonym set): the set of near-synonymous senses that all express one concept, together with a gloss (a dictionary-style definition) and often example sentences. One synset for the concept a person easy to take advantage of is

with the gloss a person who is gullible and easy to take advantage of. The gloss is a property of the synset, so every sense in it inherits the same definition. It is synsets, not words, that participate in the lexical relations — hypernymy connects one synset to another, not one string to another.

Following hypernym links upward produces a chain from a specific synset to the most abstract. The chains for two senses of bass show how disambiguation could work in principle — the senses live in entirely separate regions of the hierarchy and only meet at the root.

WordNet also tags each synset with a supersense — one of 26 coarse lexicographic categories for nouns (ANIMAL, ARTIFACT, FOOD, PERSON, …) plus 15 for verbs — useful when the full sense inventory is too fine-grained. Verbs get their own relations: troponymy (a manner-of-doing subtype: stroll is a troponym of walk) plays the role hyponymy plays for nouns, and entailment links snore to sleep.

Word similarity from the hierarchy

Because WordNet is a graph, the distance between two senses in it is a usable measure of similarity — a thesaurus-based alternative to the embedding cosine of the previous chapter. The simplest version is path length.⁵

Path length has a flaw: it treats every edge as equal, but the step from entity to physical-object near the root spans a huge conceptual gap, while nickel to dime near the leaves is tiny. Information-content measures fix this by weighting nodes with corpus probabilities. Let $P(c)$ be the probability that a randomly drawn word is an instance of synset $c$ (or any of its descendants), so $P$ increases monotonically toward the root, where $P(\text{root})=1$. The information content of a synset is $\text{IC}(c)=-\log P(c)$, and the lowest common subsumer $\text{LCS}(c_1,c_2)$ is the most specific synset that is a hypernym of both.

Resnik ignores how far below the LCS each sense lies. Lin similarity normalizes by the information content of the two senses themselves, so it rewards a specific common ancestor and penalizes senses that are individually very specific:

The numerator is the shared information, the denominator the total; the ratio lands in $[0,1]$, equalling $1$ when the two senses coincide. Both measures need only WordNet plus a corpus to estimate $P(c)$ — no labeled sense data at all.

Word sense disambiguation

The sense inventory depends on the application: WordNet senses for general text, supersenses for a coarser grain, MeSH headings for medical indexing, or the set of foreign-language translations for machine translation (the animal bat is Spanish murciélago, the baseball bat is bate). Two task shapes recur. In a lexical sample task a small fixed set of target words is disambiguated, and simple supervised classifiers do very well. In the harder all-words task every content word in a text must be tagged — like part-of-speech tagging, but with an enormous tag set, since every lemma carries its own sense inventory, so data sparseness bites hard.

Supervised WSD trains on a semantic concordance — a corpus where each open-class word is hand-tagged with its WordNet sense. The largest is SemCor, a 226,036-word subset of the Brown Corpus. A SemCor sentence looks like

where the superscript is the WordNet sense number and the subscript the part of speech. For fruit the task is to pick ${\text{fruit}}_n^1$ (the ripened body of a seed plant) over ${\text{fruit}}_n^2$ (an amount of product) and ${\text{fruit}}_n^3$ (the consequence of an action). Systems are evaluated intrinsically, by $F_1$ against the held-out gold tags.

Baselines

Two baselines set the bar. The most-frequent-sense baseline picks, for every occurrence, the sense that was most common in a labeled corpus. It performs well for how little it assumes, and in WordNet it is trivial to apply, because WordNet orders each word's senses from most to least frequent — the most-frequent sense is just sense 1. Any real system must beat always guess sense 1 to justify itself. The one-sense-per-discourse heuristic observes that a word repeated within one document usually keeps the same sense throughout; it holds best for coarse, unrelated senses and is used as a soft bias rather than a standalone baseline.

The Lesk algorithm

Labeled corpora are expensive, so knowledge-based methods that use only a dictionary are valuable. The oldest is Lesk, which chooses the sense whose gloss shares the most words with the target's context.⁶

The $\text{Compute-Overlap}$ function counts shared words, ignoring stopwords. For example, run it on bank in

The bank can guarantee deposits will eventually cover future tuition costs because it invests in adjustable-rate mortgage securities.

against the two WordNet senses.

Sense ${\text{bank}}^1$ overlaps the context on deposits and mortgage (overlap 2); ${\text{bank}}^2$ overlaps on nothing (overlap 0); Lesk returns ${\text{bank}}^1$, correctly.

The bank case is easy because one sense scores two and the other zero. A second trace shows what happens when the margins are thin and why the raw count needs the IDF fix. Disambiguate cone in

The pine cone dropped by the tree fell onto the forest floor near a cluster of spruce and fir seedlings.

against three WordNet senses, each with the gloss reduced to its content words after stoplist removal:

• ${\text{cone}}^1$ — seed-bearing organ of a pine, fir, or other conifer — content set $\{\text{seed},\text{pine},\text{fir},\text{conifer}\}$.
• ${\text{cone}}^2$ — a cone-shaped cell in the retina of the eye, sensitive to color — content set $\{\text{cell},\text{retina},\text{eye},\text{color}\}$.
• ${\text{cone}}^3$ — a solid with a circular base tapering to a point — content set $\{\text{solid},\text{circular},\text{base},\text{point}\}$.

The context content words (after stoplist) are $\{\text{pine},\text{dropped},\text{tree},\text{fell},\text{forest},\text{floor},\text{cluster},\text{spruce},\text{fir},\text{seedlings}\}$. Computing raw overlap sense by sense:

${\text{cone}}^1$ shares pine and fir with the context, overlap 2; the retina and geometry senses share nothing, overlap 0. Lesk picks the conifer sense. The tie-break rule matters when two senses draw: the algorithm keeps its running best-sense, initialized to the most frequent sense, and only replaces it on a strict improvement (overlap > max-overlap), so a later sense that merely ties the current best is rejected and the frequency prior breaks the tie.

The raw count also overweights common words. Suppose the context had matched a sense on the word thing as well as on pine. Both count $1$ toward the overlap, yet pine is far more diagnostic of the conifer sense than thing is of anything. The IDF fix weights each shared word by $\text{idf}(w)=\log (N/\text{df}(w))$, where $\text{df}(w)$ is the number of glosses (or documents) containing $w$: thing, appearing in thousands of glosses, contributes almost nothing, while pine, appearing in a handful, dominates the score. The overlap becomes a sum of IDF weights rather than a bare count, and the diagnostic words carry the decision.

Extensions improve further on the count: best-performing is to replace word overlap with the cosine between an embedding of the gloss and an embedding of the context, which catches deposits/savings matches that exact overlap misses — the precursor of the gloss-aware neural methods a later section takes up.

Feature-based WSD

Before contextual embeddings, the strongest supervised systems were plain classifiers over hand-built features of the surrounding words. The IMS system feeds an SVM three feature families: part-of-speech tags in a window around the target, collocations (specific words or $n$-grams at specific offsets), and a distance-weighted average of the neighbors' embeddings. For bass in An electric guitar and bass player stand off to one side, a two-word window yields a vector like

pairing each neighbor with its tag, adding the bigram collocations guitar-and and player-stand, and appending the pooled embedding $g$. These remain within a point or two of neural systems.

Contextual-embedding WSD

The vector-semantics problem — one vector per word — is the one a contextual embedding from a transformer solves: BERT gives a different vector to bass in grilled bass than in bass guitar, because each token's representation is a function of its whole sentence. That reopens WSD as a nearest-neighbor problem.⁷

At training time, run every SemCor sentence through BERT and collect a contextual vector $v_i$ for each labeled token (pooling the last four layers is common). Then, for each sense $s$, average the vectors of all its tagged tokens into a single sense embedding:

At test time, embed the target token $t$ in its context and return the sense whose embedding is closest by cosine:

This ties WSD directly to the transformer machinery of fine-tuning and prompting: the encoder is a pretrained BERT, and no task-specific training is needed beyond averaging labeled vectors. The gap it leaves is coverage — SemCor tags only a sliver of WordNet's senses, so many senses have no $v_s$. The fix imputes the missing embeddings bottom-up through the taxonomy: give a synset the average of its sense embeddings, a hypernym the average of its synsets, and a supersense the average of all its synsets. Because every supersense has some SemCor data, the back-off is guaranteed to reach an embedding eventually — coarse, but never empty.

Where this continues

We have taken meaning apart below the word. Senses are the discrete meanings a form carries; the sense relations — synonymy, antonymy, hyponymy, meronymy — organize them into WordNet's synset graph; path- and information-content measures read similarity off that graph; and word sense disambiguation picks the intended sense in context, from the most-frequent-sense baseline and Lesk gloss overlap to feature classifiers and the nearest-neighbor method over contextual BERT embeddings.

That leaves the variants and the loose ends: the sense-inventory-free Word-in-Context task, retrofitting embeddings to a thesaurus, discovering senses without a fixed inventory (word sense induction), the gloss-aware and bi-encoder neural systems, and evaluation. They continue in WSD in practice and word sense induction.