WSD in Practice and Word Sense Induction

This builds on word senses and disambiguation, which defined senses and their relations, assembled them into WordNet, and solved the core of word sense disambiguation — from the most-frequent-sense baseline to the nearest-neighbor method over BERT embeddings. Here we take up the variants that loosen WSD's fixed-inventory assumption, the ways senses reshape embeddings, and the neural systems and evaluations that finish the picture.

The Word-in-Context task

WSD requires a sense label, which forces a fixed inventory and fine-grained distinctions. The Word-in-Context (WiC) task strips that away: given two sentences containing the same target word, decide only whether the word means the same thing in both — a binary judgment, no sense inventory required.¹

WordNet senses are too fine to make the same/different call cleanly, so WiC first clusters senses into coarser groups — two uses count as the same if their senses are first-degree neighbors in the WordNet graph or share a supersense. The baseline is a thresholded cosine. For target word $w$ in sentences $s_1,s_2$, let ${\mathbf{c}}_1,{\mathbf{c}}_2$ be its BERT contextual vectors; predict same-sense iff

with $\tau$ tuned on held-out data. WiC is the cleanest probe of whether a contextual model separates senses rather than merely tracking the most frequent one.

Retrofitting embeddings to a thesaurus

WordNet is not only a target for disambiguation; it can also repair the static embeddings this chapter opened by criticizing. Recall the specific failure: because up and down occur in near-identical contexts, a static embedding places them close, and in GloVe the nearest neighbors of expensive include its antonym cheap, the neighbors of east include west, the neighbors of British include London and BBC. The distributional signal that a good embedding captures — words that appear in the same contexts — is the very thing that confuses antonyms. A thesaurus records the difference, and can be used to adjust the embeddings accordingly.²

There are two families of fix. The first retrains: fold thesaurus relations (synonymy, antonymy, supersenses) into the embedding loss itself, whether the static word2vec objective or a contextual model's pretraining. The second is lighter and runs after training — retrofitting (and its antonym-aware cousin counterfitting). It learns a second mapping that shifts already-trained vectors so that thesaurus synonyms are pulled closer together and antonyms pushed further apart, leaving the distributional geometry otherwise intact.

The retrofitting objective states this directly. Let ${\hat{q}}_i$ be the original trained vector for word $i$ and $q_i$ the retrofitted vector we solve for; let $E$ be the set of synonym edges from the thesaurus. Retrofitting minimizes a trade-off between two terms — staying near the original vector, and moving toward thesaurus neighbors:

The first term (${\alpha }_i$-weighted) anchors each retrofitted vector to what distributional training learned; the second (${\beta }_{ij}$-weighted, over synonym edges $E$) drags synonyms together. Counterfitting adds a third term over the antonym set $A$ that repels each pair, $+{\sum }_{(i,j)\in A}{\gamma }_{ij}\max (0,\delta -\Vert q_i-q_j\Vert )$, enforcing a margin $\delta$. The result, on the same queries: expensive now neighbors costly and overpriced instead of cheaper — the antonym is pushed out of the neighborhood.

Word sense induction

Every WSD method so far leans on a human-built sense inventory — WordNet, a supersense list, a set of translations. That inventory is the expensive part to build and label. Word sense induction (WSI) drops it: instead of assigning a word to one of a fixed list of senses, it discovers the senses automatically from how the word is used, then labels new occurrences with those discovered senses.³

The method, due in outline to Schütze, is clustering over context vectors. In training, for a word $w$:

Each of the $k$ clusters is a sense of $w$, and the centroid $s_j$ represents it. To disambiguate a new token $t$ with context vector $c(t)$, assign the nearest sense vector, $\hat{s}=\arg {\max }_j\cos (c(t),s_j)$. The method needs only a clustering algorithm and a distance metric; a common choice is agglomerative clustering, which starts every occurrence in its own cluster and repeatedly merges the two most similar clusters until a target count is reached.

WSI is cheap and needs no annotation; the difficulty appears at evaluation. The discovered senses have no names, so there is no gold set to score $F_1$ against directly. Intrinsic evaluation must first map each induced cluster onto a predefined sense (by choosing the gold sense it overlaps most), and no such mapping metric has become standard. The cleaner option is extrinsic evaluation — plug the induced senses into an end-to-end system and see if it improves, as when induced senses were used to diversify web-search results. The freedom from a human inventory is WSI's strength and, at scoring time, its weakness.

Gloss-aware and bi-encoder WSD

The nearest-neighbor method built its sense embeddings from labeled tokens alone: average the BERT vectors of every SemCor occurrence of a sense, and match a new token to the nearest average. That throws away a signal the sense already carries — its gloss. WordNet defines ${\text{bank}}^1$ as a financial institution that accepts deposits; that sentence is a free, human-written description of the sense, and the line of work past Jurafsky & Martin's treatment is about feeding it to the encoder rather than ignoring it. Three systems mark the progression.

GlossBERT (Huang, Sun, Qiu, and Huang, EMNLP 2019) reframes WSD as sentence-pair classification, the same shape BERT uses for natural-language inference. For a target word in context, form one pair per candidate sense: sentence A is the context with the target marked, sentence B is that sense's gloss. BERT reads the pair and outputs a single yes/no score for whether the gloss matches the use; the sense whose pair scores highest wins. The gloss is now input to the model, not a bag of words to intersect, so the match is robust to paraphrase — a context about savings can select a gloss about deposits with no shared word.⁴

GlossBERT re-encodes the target for every candidate sense, which is accurate but slow. The bi-encoder (Blevins and Zettlemoyer, ACL 2020) splits the work into two towers that share one embedding space. A context encoder $f$ embeds the target word in its sentence, once; a gloss encoder $g$ embeds each sense definition, once and offline. The sense score is a dot product, so disambiguation is nearest-neighbor over precomputed gloss vectors:

The sense vectors now come from glosses, not labeled tokens, so a sense with no training example still has a vector.⁵

The bi-encoder's headline result is about the long tail. Word senses follow a steep Zipfian distribution — a few senses of each word are common and most are rare or absent from SemCor — and the token-averaging method has no vector for a sense it never saw. Because the bi-encoder derives sense vectors from glosses, it labels rare and zero-shot senses far better than token-only methods, and its overall gain over the previous state of the art comes predominantly from rare senses, exactly the part of the distribution the most-frequent-sense baseline ignores.⁵

ESCHER (Barba, Pasini, and Navigli, NAACL 2021) pushes the framing one step further, casting WSD as extractive span comprehension: concatenate the sentence with all candidate glosses and have the model select the span of text that best fits the target's use, the same machinery a reading-comprehension model uses to point at an answer in a passage. This lets every gloss inform the decision jointly rather than scoring one sense at a time, and it set a new state of the art on English all-words WSD.⁶

One idea runs across these systems: treat the gloss as text to be read rather than a lookup key. Reading it with a transformer carried WSD past the nearest-neighbor method Jurafsky & Martin present.

Evaluation and where this goes

Supervised WSD is scored intrinsically with $F_1$ against SemCor or the SENSEVAL / SemEval gold sets, always against the most-frequent-sense baseline. WiC is scored by binary accuracy. Word sense induction, as just seen, has no standard intrinsic metric and is best judged extrinsically inside a downstream task.

The through-line from vector semantics is now complete. Static embeddings gave one point per word and conflated senses; contextual embeddings give one point per occurrence and pull the senses apart; WordNet supplies the discrete labels those points get matched to. The next step is meaning above the word — who did what to whom — which semantic roles and information extraction takes up, building predicate-argument structure on top of the disambiguated words this lesson produces.