--- title: "WSD in Practice and Word Sense Induction" module: Linguistic Structure moduleNumber: 6 lessonNumber: 6 order: 606 summary: > Beyond core word sense disambiguation lie the variants and loose ends: the sense-inventory-free Word-in-Context task, retrofitting static embeddings to a thesaurus, discovering senses without a fixed inventory (word sense induction), the gloss-aware and bi-encoder neural systems that hold the state of the art, and how WSD and its cousins are evaluated. Together they connect one-vector-per-word embeddings to sense-aware contextual representations. topics: [Structure] sources: - book: Jurafsky ref: "§18.5 Alternate WSD Algorithms and Tasks; §18.6 Using Thesauruses to Improve Embeddings; §18.7 Word Sense Induction" --- This builds on [word senses and disambiguation](/natural-language-processing/linguistic-structure/word-senses-and-wsd), which defined senses and their relations, assembled them into WordNet, and solved the core of word sense disambiguation — from the most-frequent-sense baseline to the nearest-neighbor method over BERT embeddings. Here we take up the variants that loosen WSD's fixed-inventory assumption, the ways senses reshape embeddings, and the neural systems and evaluations that finish the picture. ## The Word-in-Context task WSD requires a sense _label_, which forces a fixed inventory and fine-grained distinctions. The **Word-in-Context** (WiC) task strips that away: given two sentences containing the same target word, decide only whether the word means the _same thing_ in both — a binary judgment, no sense inventory required.[^jm-wic] $$ % caption: Word-in-Context pairs. Each row shares a target word across two % sentences; the label is T when the two uses share a sense and F when they % diverge. \begin{tikzpicture}[>=stealth, font=\footnotesize, lab/.style={draw, circle, minimum size=6mm, font=\scriptsize}, txt/.style={anchor=west, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[lab, draw=red, text=red] (l1) at (0,2.0) {F}; \node[txt] at (0.7,2.0) {trash on the \textbf{bed} of the river / a glass of water next to my \textbf{bed}}; \node[lab, draw=red, text=red] (l2) at (0,1.0) {F}; \node[txt] at (0.7,1.0) {\textbf{justify} the margins / the end \textbf{justi\/fies} the means}; \node[lab, draw=acc, text=acc] (l3) at (0,0.0) {T}; \node[txt] at (0.7,0.0) {\textbf{air} pollution / open a window and let in some \textbf{air}}; \node[lab, draw=acc, text=acc] (l4) at (0,-1.0) {T}; \node[txt] at (0.7,-1.0) {the expanded \textbf{window} / a two-hour \textbf{window} of clear weather}; \end{tikzpicture} $$ WordNet senses are too fine to make the same/different call cleanly, so WiC first clusters senses into coarser groups — two uses count as the "same" if their senses are first-degree neighbors in the WordNet graph or share a supersense. The baseline is a thresholded cosine. For target word $w$ in sentences $s_1, s_2$, let $\mathbf{c}_1, \mathbf{c}_2$ be its BERT contextual vectors; predict same-sense iff $$ \cos(\mathbf{c}_1, \mathbf{c}_2) = \frac{\mathbf{c}_1 \cdot \mathbf{c}_2}{\lVert \mathbf{c}_1 \rVert \, \lVert \mathbf{c}_2 \rVert} \ge \tau, $$ with $\tau$ tuned on held-out data. WiC is the cleanest probe of whether a contextual model separates senses rather than merely tracking the most frequent one. ## Retrofitting embeddings to a thesaurus WordNet is not only a target for disambiguation; it can also repair the [static embeddings](/natural-language-processing/semantics/vector-semantics-and-embeddings) this chapter opened by criticizing. Recall the specific failure: because _up_ and _down_ occur in near-identical contexts, a static embedding places them close, and in GloVe the nearest neighbors of _expensive_ include its antonym _cheap_, the neighbors of _east_ include _west_, the neighbors of _British_ include _London_ and _BBC_. The distributional signal that a good embedding captures — words that appear in the same contexts — is the very thing that confuses antonyms. A thesaurus records the difference, and can be used to adjust the embeddings accordingly.[^jm-retro] There are two families of fix. The first **retrains**: fold thesaurus relations (synonymy, antonymy, supersenses) into the embedding loss itself, whether the static word2vec objective or a contextual model's pretraining. The second is lighter and runs _after_ training — **retrofitting** (and its antonym-aware cousin **counterfitting**). It learns a second mapping that shifts already-trained vectors so that thesaurus synonyms are pulled closer together and antonyms pushed further apart, leaving the distributional geometry otherwise intact. The retrofitting objective states this directly. Let $\hat{q}_i$ be the original trained vector for word $i$ and $q_i$ the retrofitted vector we solve for; let $E$ be the set of synonym edges from the thesaurus. Retrofitting minimizes a trade-off between two terms — staying near the original vector, and moving toward thesaurus neighbors: $$ \Psi(Q) = \sum_i \Big[ \alpha_i \, \lVert q_i - \hat{q}_i \rVert^2 + \sum_{(i,j)\in E} \beta_{ij}\, \lVert q_i - q_j \rVert^2 \Big]. $$ The first term ($\alpha_i$-weighted) anchors each retrofitted vector to what distributional training learned; the second ($\beta_{ij}$-weighted, over synonym edges $E$) drags synonyms together. Counterfitting adds a third term over the antonym set $A$ that _repels_ each pair, $+\sum_{(i,j)\in A} \gamma_{ij} \max(0, \delta - \lVert q_i - q_j \rVert)$, enforcing a margin $\delta$. The result, on the same queries: _expensive_ now neighbors _costly_ and _overpriced_ instead of _cheaper_ — the antonym is pushed out of the neighborhood. $$ % caption: Retrofitting pulls a trained embedding toward its thesaurus synonyms while % an anchor term holds it near its original position; counterfitting adds a push away % from antonyms. The dashed circle marks the original vector, the arrows the two % competing forces. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % original position (dashed) and retrofitted (solid) \node[circle, draw, dashed, minimum size=6mm, inner sep=0pt, font=\scriptsize] (orig) at (0,0) {}; \node[anchor=north, font=\scriptsize, text=black] at (0,-0.45) {original}; \node[circle, draw=acc, fill=acc!10, minimum size=8mm, inner sep=0pt, font=\scriptsize, text=acc] (rf) at (2.2,0.9) {word}; % synonyms (pull toward) \node[circle, draw, fill=black!4, minimum size=8mm, inner sep=0pt, font=\scriptsize] (s1) at (4.8,1.9) {costly}; \node[circle, draw, fill=black!4, minimum size=8mm, inner sep=0pt, font=\scriptsize] (s2) at (5.0,0.2) {pricey}; % antonym (push away) \node[circle, draw=red, fill=red!6, minimum size=8mm, inner sep=0pt, font=\scriptsize, text=red] (a1) at (-1.4,2.0) {cheap}; % forces \draw[->, black, thick] (rf) -- (orig) node[midway, right, font=\scriptsize] {anchor}; \draw[->, acc, thick] (rf) -- (s1) node[midway, above, font=\scriptsize] {pull}; \draw[->, acc, thick] (rf) -- (s2); \draw[->, red, thick] (rf) -- (a1) node[midway, above, font=\scriptsize] {push}; \end{tikzpicture} $$ ## Word sense induction Every WSD method so far leans on a human-built sense inventory — WordNet, a supersense list, a set of translations. That inventory is the expensive part to build and label. **Word sense induction** (WSI) drops it: instead of assigning a word to one of a fixed list of senses, it _discovers_ the senses automatically from how the word is used, then labels new occurrences with those discovered senses.[^jm-wsi] > **Definition (Word sense induction).** The unsupervised task of learning a word's > senses from a corpus with no human sense labels, by clustering the contexts in > which the word occurs; each cluster is taken to be a sense, and the cluster's > centroid is the sense vector. The method, due in outline to Schütze, is clustering over context vectors. In **training**, for a word $w$: ```algorithm caption: $\textsc{Induce-Senses}(w, \text{corpus})$ — cluster contexts into senses for each token $w_i$ of $w$ in the corpus do $c_i \gets$ context vector for $w_i$ cluster the vectors $c_i$ into $k$ groups for each cluster $j$ do $s_j \gets$ centroid of the vectors in cluster $j$ // the sense vector ``` Each of the $k$ clusters _is_ a sense of $w$, and the centroid $s_j$ represents it. To **disambiguate** a new token $t$ with context vector $c(t)$, assign the nearest sense vector, $\hat{s} = \arg\max_j \cos(c(t), s_j)$. The method needs only a clustering algorithm and a distance metric; a common choice is **agglomerative clustering**, which starts every occurrence in its own cluster and repeatedly merges the two most similar clusters until a target count is reached. $$ % caption: Word sense induction for "bass". Each occurrence becomes a context vector; % clustering separates the tokens into groups (a music cluster and a fish cluster), % and each cluster centroid becomes an induced sense with no human-given name. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % music cluster (blue) points \foreach \x/\y in {0.2/2.1, 0.7/2.6, 0.5/1.6, 1.1/2.2, 0.9/1.7} { \fill[acc] (\x,\y) circle (2pt); } \node[circle, draw=acc, thick, minimum size=10mm, inner sep=0pt] (c1) at (0.68,2.05) {}; \node[anchor=south, font=\scriptsize, text=acc] at (0.68,2.75) {sense 1}; % fish cluster (red) points \foreach \x/\y in {3.6/0.5, 4.1/0.9, 3.9/0.2, 4.4/0.6, 3.7/1.0} { \fill[red] (\x,\y) circle (2pt); } \node[circle, draw=red, thick, minimum size=10mm, inner sep=0pt] (c2) at (3.94,0.64) {}; \node[anchor=north, font=\scriptsize, text=red] at (3.94,-0.05) {sense 2}; % centroids marked \node[star, star points=5, draw=acc, fill=acc!25, minimum size=4mm, inner sep=0pt] at (0.68,2.05) {}; \node[star, star points=5, draw=red, fill=red!25, minimum size=4mm, inner sep=0pt] at (3.94,0.64) {}; \node[anchor=west, font=\scriptsize, text=black] at (5.0,1.3) {each star = a}; \node[anchor=west, font=\scriptsize, text=black] at (5.0,0.9) {cluster centroid}; \node[anchor=west, font=\scriptsize, text=black] at (5.0,0.5) {= induced sense}; \end{tikzpicture} $$ WSI is cheap and needs no annotation; the difficulty appears at evaluation. The discovered "senses" have no names, so there is no gold set to score $F_1$ against directly. Intrinsic evaluation must first map each induced cluster onto a predefined sense (by choosing the gold sense it overlaps most), and no such mapping metric has become standard. The cleaner option is **extrinsic** evaluation — plug the induced senses into an end-to-end system and see if it improves, as when induced senses were used to diversify web-search results. The freedom from a human inventory is WSI's strength and, at scoring time, its weakness. ## Gloss-aware and bi-encoder WSD The nearest-neighbor method built its sense embeddings from _labeled tokens_ alone: average the BERT vectors of every SemCor occurrence of a sense, and match a new token to the nearest average. That throws away a signal the sense already carries — its **gloss**. WordNet defines $\text{bank}^1$ as _a financial institution that accepts deposits_; that sentence is a free, human-written description of the sense, and the line of work past Jurafsky & Martin's treatment is about feeding it to the encoder rather than ignoring it. Three systems mark the progression. **GlossBERT** (Huang, Sun, Qiu, and Huang, EMNLP 2019) reframes WSD as **sentence-pair classification**, the same shape BERT uses for natural-language inference. For a target word in context, form one pair per candidate sense: sentence A is the context with the target marked, sentence B is that sense's gloss. BERT reads the pair and outputs a single yes/no score for whether the gloss matches the use; the sense whose pair scores highest wins. The gloss is now _input to the model_, not a bag of words to intersect, so the match is robust to paraphrase — a context about _savings_ can select a gloss about _deposits_ with no shared word.[^gloss-wsd] $$ % caption: GlossBERT frames WSD as sentence-pair classification. Each candidate sense % of the target contributes one (context, gloss) pair; BERT scores the pair yes/no, % and the highest-scoring gloss selects the sense. Here $\text{bank}^1$ (finance) % beats $\text{bank}^2$ (riverside). \begin{tikzpicture}[>=stealth, font=\footnotesize, ctx/.style={draw, minimum width=40mm, minimum height=6.5mm, align=center, font=\scriptsize}, gloss/.style={draw, minimum width=40mm, minimum height=6.5mm, align=center, font=\scriptsize}, enc/.style={draw=acc, text=acc, thick, minimum width=20mm, minimum height=8mm, align=center, font=\scriptsize}, sco/.style={draw, circle, minimum size=7mm, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % pair 1 \node[ctx] (c1) at (0,3.0) {context: ...open an account at the bank...}; \node[gloss, draw=acc, text=acc] (g1) at (0,2.1) {gloss bank-1: f\/inancial institution}; \node[enc] (b1) at (5.2,2.55) {BERT}; \node[sco, draw=acc, text=acc] (s1) at (7.6,2.55) {0.94}; \draw[->, black] (c1.east) -- (b1.west); \draw[->, acc] (g1.east) -- (b1.west); \draw[->, acc, thick] (b1) -- (s1); % pair 2 \node[ctx] (c2) at (0,0.6) {context: ...open an account at the bank...}; \node[gloss] (g2) at (0,-0.3) {gloss bank-2: sloping riverside land}; \node[enc] (b2) at (5.2,0.15) {BERT}; \node[sco] (s2) at (7.6,0.15) {0.07}; \draw[->, black] (c2.east) -- (b2.west); \draw[->, black] (g2.east) -- (b2.west); \draw[->, black] (b2) -- (s2); \node[acc, anchor=west, font=\scriptsize] at (8.2,2.55) {argmax}; \end{tikzpicture} $$ GlossBERT re-encodes the target for every candidate sense, which is accurate but slow. The **bi-encoder** (Blevins and Zettlemoyer, ACL 2020) splits the work into two towers that share one embedding space. A **context encoder** $f$ embeds the target word in its sentence, once; a **gloss encoder** $g$ embeds each sense definition, once and offline. The sense score is a dot product, so disambiguation is nearest-neighbor over precomputed gloss vectors: $$ \hat{s} = \arg\max_{s \in \text{senses}(w)} f(w, \text{context}) \cdot g(\text{gloss}(s)). $$ The sense vectors now come from _glosses_, not labeled tokens, so a sense with no training example still has a vector.[^biencoder-wsd] $$ % caption: The gloss-informed bi-encoder. A context encoder embeds the target word in % context; a gloss encoder embeds each sense definition into the same space; the sense % whose gloss embedding has the highest dot product with the context embedding is % chosen. Gloss embeddings are precomputed offline. \begin{tikzpicture}[>=stealth, font=\footnotesize, enc/.style={draw=acc, text=acc, thick, minimum width=30mm, minimum height=8mm, align=center, font=\scriptsize}, vec/.style={circle, draw, minimum size=8mm, font=\scriptsize, inner sep=0pt}, gv/.style={circle, draw=acc, fill=acc!8, minimum size=8mm, font=\scriptsize, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % context tower \node[font=\scriptsize] (ctx) at (0,3.6) {"...at the bank..."}; \node[enc] (ce) at (0,2.6) {context encoder}; \node[vec, draw=red, text=red] (cv) at (0,1.4) {c}; \draw[->, black] (ctx) -- (ce); \draw[->, red] (ce) -- (cv); % gloss tower (offline) \node[font=\scriptsize] (gl) at (6.4,3.6) {each sense gloss}; \node[enc] (ge) at (6.4,2.6) {gloss encoder}; \node[gv] (g1) at (5.2,1.4) {g1}; \node[gv] (g2) at (6.4,1.4) {g2}; \node[gv] (g3) at (7.6,1.4) {g3}; \draw[->, black] (gl) -- (ge); \draw[->, acc] (ge) -- (g1); \draw[->, acc] (ge) -- (g2); \draw[->, acc] (ge) -- (g3); % scoring \node[font=\scriptsize] (dot) at (2.6,0.55) {dot product, pick max}; \draw[red!55, dashed] (cv) to[out=-30,in=250] (g1.south); \draw[red, thick, ->] (cv) to[out=-55,in=250] (g2.south); \draw[red!55, dashed] (cv) to[out=-45,in=250] (g3.south); \node[anchor=south, font=\scriptsize, text=black] at (6.4,4.0) {precomputed of\/f\/line}; \end{tikzpicture} $$ The bi-encoder's headline result is about the **long tail**. Word senses follow a steep Zipfian distribution — a few senses of each word are common and most are rare or absent from SemCor — and the token-averaging method has no vector for a sense it never saw. Because the bi-encoder derives sense vectors from glosses, it labels rare and zero-shot senses far better than token-only methods, and its overall gain over the previous state of the art comes _predominantly from rare senses_, exactly the part of the distribution the most-frequent-sense baseline ignores.[^biencoder-wsd] **ESCHER** (Barba, Pasini, and Navigli, NAACL 2021) pushes the framing one step further, casting WSD as **extractive span comprehension**: concatenate the sentence with all candidate glosses and have the model select the span of text that best fits the target's use, the same machinery a reading-comprehension model uses to point at an answer in a passage. This lets every gloss inform the decision jointly rather than scoring one sense at a time, and it set a new state of the art on English all-words WSD.[^escher-wsd] | System | Framing | Gloss encoded | Rare/zero-shot senses | | --- | --- | --- | --- | | GlossBERT | (context, gloss) sentence-pair classification | per candidate, re-encoded | no vector without a labeled token | | Bi-encoder | dot product of context and gloss towers | once, offline | vector from gloss alone | | ESCHER | extractive span over concatenated glosses | jointly, all at once | vector from gloss alone | One idea runs across these systems: treat the gloss as text to be read rather than a lookup key. Reading it with a transformer carried WSD past the nearest-neighbor method Jurafsky & Martin present. ## Evaluation and where this goes Supervised WSD is scored intrinsically with $F_1$ against SemCor or the SENSEVAL / SemEval gold sets, always against the most-frequent-sense baseline. WiC is scored by binary accuracy. Word sense induction, as just seen, has no standard intrinsic metric and is best judged extrinsically inside a downstream task. The through-line from [vector semantics](/natural-language-processing/semantics/vector-semantics-and-embeddings) is now complete. Static embeddings gave one point per word and conflated senses; contextual embeddings give one point per _occurrence_ and pull the senses apart; WordNet supplies the discrete labels those points get matched to. The next step is meaning above the word — who did what to whom — which [semantic roles and information extraction](/natural-language-processing/linguistic-structure/semantic-roles-and-information-extraction) takes up, building predicate-argument structure on top of the disambiguated words this lesson produces. [^jm-wic]: **Jurafsky & Martin**, §18.5.3 — Word-in-Context evaluation: the same/different-sense judgment, sense clustering into coarse groups, and the thresholded-cosine baseline; §18.4.1 for the SemCor / all-words setup and evaluation. [^jm-retro]: **Jurafsky & Martin**, §18.6 — Using Thesauruses to Improve Embeddings: the antonym problem in static embeddings (GloVe neighbors of _expensive_, _east_, _British_); the retrain family (modifying the word2vec or contextual loss) and the post-hoc family — retrofitting (Faruqui et al. 2015) and counterfitting (Mrkšić et al. 2016) — that shift trained vectors to pull synonyms together and push antonyms apart (Fig. 18.12). [^jm-wsi]: **Jurafsky & Martin**, §18.7 — Word Sense Induction: unsupervised sense learning by clustering context vectors (Schütze 1992b, 1998); the three-step training (context vector, cluster, centroid = sense vector) and disambiguation (nearest sense vector); agglomerative clustering; and evaluation difficulty (no standard intrinsic metric, extrinsic evaluation via search-result diversification). [^gloss-wsd]: **Huang, Sun, Qiu, and Huang**, "GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge," EMNLP-IJCNLP 2019. WSD recast as sentence-pair classification: each (context, sense-gloss) pair is scored by BERT, and the highest-scoring gloss selects the sense. Feeding the gloss into the encoder (rather than intersecting it as a bag of words, as Lesk does) lets the match survive paraphrase. [^escher-wsd]: **Barba, Pasini, and Navigli**, "ESC: Redesigning WSD with Extractive Sense Comprehension" (ESCHER), NAACL 2021. WSD framed as extractive span comprehension — the sentence is concatenated with all candidate glosses and the model selects the span best fitting the target use, letting every gloss inform the decision jointly. Set a new state of the art on English all-words WSD. [^biencoder-wsd]: **Blevins and Zettlemoyer**, "Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders," ACL 2020. A bi-encoder independently embeds the target-in-context and each sense gloss into a shared space, scoring by dot product; the two encoders are jointly trained. Because sense vectors derive from glosses rather than labeled tokens, the model represents rare and unseen senses, and its improvement over the prior state of the art comes predominantly from those rare senses — the long tail the most-frequent-sense baseline ignores.