---
title: "Static Word Embeddings: word2vec and After"
module: Semantics
moduleNumber: 3
lessonNumber: 2
order: 302
summary: >
  Count-based vectors are long and sparse; embeddings are the short, dense
  alternative. This lesson builds them with word2vec's skip-gram and negative
  sampling — a classifier whose learned weights are the vectors — derives its
  gradient, and works one update by hand. It then reads relations off the analogy
  parallelogram, surveys the papers that framed the static-embedding era
  (word2vec, GloVe, the SGNS-as-PPMI equivalence, fastText, ELMo), and closes on
  the biases embeddings inherit and the single-vector-per-word ceiling that
  contextual models break.
topics: [Semantics]
sources:
  - book: Jurafsky
    ref: "Ch. 6 — Vector Semantics and Embeddings; §6.8 Word2vec; §6.9 Other Static Embeddings"
  - book: Jurafsky
    ref: "§6.10 Semantic Properties of Embeddings; §6.11 Bias and Embeddings"
---

This builds on [Vector Semantics and Embeddings](/natural-language-processing/semantics/vector-semantics-and-embeddings),
which represented each word as a long, sparse vector of context counts, weighted
by tf-idf or PPMI. Those vectors work, but they carry one dimension per
vocabulary word and are almost all zeros. Here we compress them: instead of
counting the contexts a word appears in, we train a small classifier to _predict_
them, and keep the weights it learns as the word's dense vector. Same
distributional idea, a far smaller and more useful representation.

## Dense embeddings and word2vec

The vectors so far are **sparse** and **long**: dimension $|V|$, mostly zeros. The
alternative is a **dense** vector — **short** (50 to 1000 dimensions), with
real-valued entries that can be negative and no per-dimension interpretation.
These are **embeddings**.[^jm-w2v] Dense vectors work better on essentially every
NLP task. A classifier over 300 dense dimensions has far fewer weights to fit than
one over 50,000 sparse dimensions, so it generalizes better; and a dense space can
capture synonymy that a sparse one misses, where _car_ and _automobile_ occupy
distinct, unrelated dimensions.[^jm-w2v]

> **Definition (Embedding).** A short, dense, real-valued vector $\mathbf{v}_w \in
> \mathbb{R}^d$ ($d \approx 50$–$1000$) representing a word's meaning, learned so
> that words with similar distributions receive nearby vectors. _Static_
> embeddings assign one fixed vector per word type.

**Word2vec** is the method that made dense embeddings routine.[^jm-w2v] Rather than
computing anything about counts directly, it trains a
binary classifier on a pretext task — _is context word $c$ likely to occur near target
word $w$?_ — then discards the predictions and keeps the learned _weights_
as the embeddings. Because the
training data is just running text (a word and its actual neighbors are positive
examples, needing no human labels), this is **self-supervision** — the same idea
that later scales up to language-model pretraining.

### Skip-gram with negative sampling

The specific algorithm is **skip-gram with negative sampling** (SGNS). Take a
target word $w$ and a window of $\pm L$ words around it as its context. Each
(target, true-context) pair is a **positive** example. To teach the classifier what
_non_-neighbors look like, we manufacture **negative** examples: for each positive
pair, sample $k$ **noise words** $c_{\text{neg}}$ at random from the vocabulary,
each paired with the same target.[^jm-sgns]

$$
% caption: The skip-gram setup for target $w = apricot$ with a $\pm 2$ window.
% The four in-window words are positive contexts; $k$ random noise words (here
% $k=2$ per positive, e.g. $aardvark$, $seven$) are negative contexts the classifier learns to reject.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  ctx/.style={draw, minimum width=15mm, minimum height=7mm, align=center, font=\footnotesize},
  tgt/.style={draw=acc, text=acc, thick, minimum width=15mm, minimum height=7mm, align=center},
  noise/.style={draw=red, text=red, minimum width=15mm, minimum height=7mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % context window row
  \node[ctx] (c1) at (0,0)   {tablespoon};
  \node[ctx] (c2) at (1.8,0) {of};
  \node[tgt] (w)  at (3.6,0) {apricot};
  \node[ctx] (c3) at (5.4,0) {jam};
  \node[ctx] (c4) at (7.2,0) {a};
  \node[anchor=south, font=\scriptsize] at (0,0.45)   {c1};
  \node[anchor=south, font=\scriptsize] at (1.8,0.45) {c2};
  \node[acc, anchor=south, font=\scriptsize] at (3.6,0.45) {target w};
  \node[anchor=south, font=\scriptsize] at (5.4,0.45) {c3};
  \node[anchor=south, font=\scriptsize] at (7.2,0.45) {c4};
  % positive bracket
  \draw[acc, thick] (-0.85,-0.55) -- (8.05,-0.55);
  \node[acc, anchor=north, font=\scriptsize] at (3.6,-0.62) {positive: (apricot, c) for each window word};
  % negative row
  \node[noise] (n1) at (2.1,-2.3) {aardvark};
  \node[noise] (n2) at (5.1,-2.3) {seven};
  \node[red, anchor=east, font=\scriptsize] at (1.15,-2.3) {noise:};
  \draw[->, red] (3.6,-1.25) .. controls (3.0,-1.7) and (2.6,-1.8) .. (n1.north);
  \draw[->, red] (3.6,-1.25) .. controls (4.2,-1.7) and (4.8,-1.8) .. (n2.north);
  \node[red, anchor=north, font=\scriptsize] at (3.6,-2.75) {negative: (apricot, noise), k random words};
\end{tikzpicture}
$$

The classifier decides by embedding similarity. It keeps two vectors per
word — a **target** embedding $\mathbf{w}$ and a **context** embedding
$\mathbf{c}$ — and models the probability that $c$ is a real context of $w$ as the
sigmoid of their dot product:[^jm-sgns]

$$
P(+ \mid w, c) \;=\; \sigma(\mathbf{c} \cdot \mathbf{w}) \;=\; \frac{1}{1 + \exp(-\mathbf{c} \cdot \mathbf{w})} ,
\qquad P(- \mid w, c) \;=\; \sigma(-\mathbf{c} \cdot \mathbf{w}) .
$$

The dot product is a raw similarity; the sigmoid squashes it into a probability.
For one positive pair $(w, c_{\text{pos}})$ with noise words $c_{\text{neg}_1},
\ldots, c_{\text{neg}_k}$, the loss is minimized when the positive pair looks like
neighbors and every noise pair looks like non-neighbors,

$$
L_{\mathrm{CE}} \;=\; -\left[\, \log \sigma(\mathbf{c}_{\text{pos}} \cdot \mathbf{w}) \;+\; \sum_{i=1}^{k} \log \sigma(-\mathbf{c}_{\text{neg}_i} \cdot \mathbf{w}) \,\right] .
$$

Minimizing this by stochastic gradient descent pulls $\mathbf{w}$ toward the
context embeddings of words it truly occurs with and pushes it away from the noise
words. The noise words themselves are drawn not from the plain unigram
distribution $P(w)$ but from the weighted $P_\alpha(w) \propto \mathrm{count}(w)^{0.75}$
— the same $\alpha = 0.75$ that tamed PPMI, here giving rare words a slightly higher
chance of being sampled as noise.[^jm-sgns] The full procedure:

```algorithm
caption: $\textsc{Skip-Gram-NS}$ — learn embeddings by classifying real vs. noise context
input: corpus, window $L$, negatives per positive $k$, dimension $d$
initialize $\mathbf{w}_i, \mathbf{c}_i \in \mathbb{R}^d$ randomly for each word $i$
for each target word $w$ in the corpus do
  for each context word $c_{\text{pos}}$ within $\pm L$ of $w$ do
    sample $k$ noise words $c_{\text{neg}_1}, \ldots, c_{\text{neg}_k} \sim P_\alpha$
    $L_{\mathrm{CE}} \gets -\log \sigma(\mathbf{c}_{\text{pos}} \cdot \mathbf{w}) - \sum_{i=1}^{k} \log \sigma(-\mathbf{c}_{\text{neg}_i} \cdot \mathbf{w})$
    $\mathbf{c}_{\text{pos}} \gets \mathbf{c}_{\text{pos}} - \eta\,[\sigma(\mathbf{c}_{\text{pos}} \cdot \mathbf{w}) - 1]\,\mathbf{w}$
    for each noise word $c_{\text{neg}_i}$ do
      $\mathbf{c}_{\text{neg}_i} \gets \mathbf{c}_{\text{neg}_i} - \eta\,\sigma(\mathbf{c}_{\text{neg}_i} \cdot \mathbf{w})\,\mathbf{w}$
    $\mathbf{w} \gets \mathbf{w} - \eta\left([\sigma(\mathbf{c}_{\text{pos}} \cdot \mathbf{w}) - 1]\,\mathbf{c}_{\text{pos}} + \sum_{i=1}^{k}\sigma(\mathbf{c}_{\text{neg}_i} \cdot \mathbf{w})\,\mathbf{c}_{\text{neg}_i}\right)$
return $\mathbf{w}_i + \mathbf{c}_i$ for each word $i$
```

The learner stores two matrices, the target matrix $\mathbf{W}$ and the context
matrix $\mathbf{C}$, each holding one $d$-dimensional vector per vocabulary word.
When training ends, the embedding of word $i$ is usually taken as $\mathbf{w}_i +
\mathbf{c}_i$ (or just $\mathbf{w}_i$, discarding $\mathbf{C}$).[^jm-sgns] The
predictions are never used; the weights _are_ the embeddings.

### The gradient, derived

The three update lines above come from differentiating the loss. Write $\sigma_{\text{pos}}
= \sigma(\mathbf{c}_{\text{pos}} \cdot \mathbf{w})$ for the model's current probability
that the true pair is a neighbor. Since $\frac{d}{dx}\log\sigma(x) = 1 - \sigma(x)$ and
$\frac{d}{dx}\log\sigma(-x) = -\sigma(x)$, differentiating $L_{\mathrm{CE}} = -\log
\sigma(\mathbf{c}_{\text{pos}} \cdot \mathbf{w}) - \sum_i \log \sigma(-\mathbf{c}_{\text{neg}_i}
\cdot \mathbf{w})$ by the chain rule gives[^jm-sgns]

$$
\frac{\partial L_{\mathrm{CE}}}{\partial \mathbf{c}_{\text{pos}}} = [\sigma_{\text{pos}} - 1]\,\mathbf{w},
\qquad
\frac{\partial L_{\mathrm{CE}}}{\partial \mathbf{c}_{\text{neg}_i}} = \sigma(\mathbf{c}_{\text{neg}_i} \cdot \mathbf{w})\,\mathbf{w},
\qquad
\frac{\partial L_{\mathrm{CE}}}{\partial \mathbf{w}} = [\sigma_{\text{pos}} - 1]\,\mathbf{c}_{\text{pos}} + \sum_{i=1}^{k}\sigma(\mathbf{c}_{\text{neg}_i} \cdot \mathbf{w})\,\mathbf{c}_{\text{neg}_i}.
$$

Read the signs. For the positive pair, $\sigma_{\text{pos}} - 1$ is negative (the
probability is below $1$), so $-\eta[\sigma_{\text{pos}} - 1]\mathbf{w}$ _adds_ a
multiple of $\mathbf{w}$ to $\mathbf{c}_{\text{pos}}$: the two vectors move toward each
other, raising their future dot product. For a noise pair, $\sigma(\cdot) > 0$ pushes
$\mathbf{c}_{\text{neg}_i}$ _away_ from $\mathbf{w}$. Attraction to real neighbors,
repulsion from noise — exactly the two goals the loss encodes.

### Worked example: one SGNS update step

Take target _apricot_ with two-dimensional embeddings, one positive
context _jam_ and one noise word _tolstoy_, learning rate $\eta = 0.1$:

$$
\mathbf{w}_{\text{apricot}} = [0.5,\, 0.2], \quad
\mathbf{c}_{\text{jam}} = [0.4,\, 0.3], \quad
\mathbf{c}_{\text{tolstoy}} = [-0.2,\, 0.6].
$$

**Dot products.** $\mathbf{c}_{\text{jam}} \cdot \mathbf{w} = 0.4(0.5) + 0.3(0.2) =
0.26$, and $\mathbf{c}_{\text{tolstoy}} \cdot \mathbf{w} = -0.2(0.5) + 0.6(0.2) = 0.02$.

**Sigmoids.** $\sigma(0.26) = 1/(1 + e^{-0.26}) = 0.565$ and $\sigma(0.02) = 0.505$.
The model currently gives _jam_ a neighbor probability of only $0.565$ (it should be
near $1$) and _tolstoy_ a probability of $0.505$ (it should be near $0$). Both errors
produce useful gradients.

**Updates.** For the positive context, $[\sigma_{\text{pos}} - 1] = -0.435$, so

$$
\mathbf{c}_{\text{jam}} \gets [0.4, 0.3] - 0.1(-0.435)[0.5, 0.2] = [0.4, 0.3] + [0.0218, 0.0087] = [0.422,\, 0.309].
$$

For the noise word, $\sigma(0.02) = 0.505$, so

$$
\mathbf{c}_{\text{tolstoy}} \gets [-0.2, 0.6] - 0.1(0.505)[0.5, 0.2] = [-0.2, 0.6] - [0.0253, 0.0101] = [-0.225,\, 0.590].
$$

And the target embedding, which receives both updates at once,

$$
\mathbf{w} \gets [0.5, 0.2] - 0.1\big((-0.435)[0.4,0.3] + (0.505)[-0.2,0.6]\big) = [0.5, 0.2] - 0.1[-0.275, 0.173] = [0.527,\, 0.183].
$$

Check the effect on the dot products that matter. The new $\mathbf{c}_{\text{jam}} \cdot
\mathbf{w}_{\text{new}} = 0.422(0.527) + 0.309(0.183) = 0.279$, up from $0.26$ — _apricot_
and _jam_ grew more similar. The new $\mathbf{c}_{\text{tolstoy}} \cdot \mathbf{w}_{\text{new}}
= -0.225(0.527) + 0.590(0.183) = -0.011$, down from $0.02$ — _apricot_ and _tolstoy_ grew
less similar. One step of the pretext task moved the geometry exactly as intended; a
corpus of billions of such steps produces the final space.

$$
% caption: One SGNS gradient step in the plane. The update pulls the target
% $apricot$ toward its true context $jam$ (dot product rises $0.26 \to 0.28$) and
% pushes it away from the noise word $tolstoy$ (dot product falls $0.02 \to -0.01$).
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-2.6,0) -- (2.6,0);
  \draw[->, black] (0,-0.6) -- (0,3.2);
  % jam (positive) up-right, scaled x5
  \draw[->, acc, thick] (0,0) -- (2.1,1.55);
  \node[acc, anchor=south west] at (2.0,1.5) {jam (pos)};
  % apricot before
  \draw[->, black, thick] (0,0) -- (2.5,1.0);
  \node[black, anchor=north west] at (2.2,0.85) {apricot};
  % apricot after (dashed, moved toward jam)
  \draw[->, black, thick, dashed] (0,0) -- (2.55,0.92);
  % tolstoy (noise) up-left
  \draw[->, red, thick] (0,0) -- (-1.0,3.0);
  \node[red, anchor=south east] at (-0.9,2.9) {tolstoy (noise)};
  \draw[->, acc] (2.3,1.3) to[bend right=25] (2.5,1.02);
  \node[acc, anchor=west, font=\scriptsize] at (2.6,1.25) {pull};
\end{tikzpicture}
$$

## The analogy parallelogram

Embeddings trained this way turn out to encode _relations_, not just similarities,
in their geometry. The **parallelogram model** of analogy, proposed for cognition
long before word2vec, solves _a is to b as $a^\ast$ is to what?_ by vector arithmetic:
compute the offset $\mathbf{b} - \mathbf{a}$ that carries $a$ to $b$, add it to
$a^\ast$, and return the nearest word.[^jm-analogy]

$$
% caption: The parallelogram model for the analogy $apple:tree :: grape:?$. The
% offset from $apple$ to $tree$ is added to $grape$; the nearest word to the
% resulting corner is $vine$, completing the parallelogram.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % four corners of the parallelogram
  \coordinate (apple) at (0,0.6);
  \coordinate (tree)  at (2.6,2.1);
  \coordinate (grape) at (1.1,-1.7);
  \coordinate (vine)  at (3.7,-0.2);
  % parallelogram fill (light tint) + outline
  \fill[acc!8] (apple) -- (tree) -- (vine) -- (grape) -- cycle;
  \draw[black, dashed] (apple) -- (tree);
  \draw[black, dashed] (grape) -- (vine);
  \draw[black, dashed] (apple) -- (grape);
  % the two offset arrows (the relation)
  \draw[->, acc, thick] (apple) -- (tree);
  \draw[->, acc, thick] (grape) -- (vine);
  % dots + labels, offset so they never touch the nodes
  \fill[black] (apple) circle (1.6pt); \node[anchor=south east] at (apple) {apple};
  \fill[black] (tree)  circle (1.6pt); \node[anchor=south west] at (tree)  {tree};
  \fill[black] (grape) circle (1.6pt); \node[anchor=north east] at (grape) {grape};
  \fill[black] (vine)  circle (1.6pt); \node[anchor=north west] at (vine)  {vine};
\end{tikzpicture}
$$

On real word2vec and GloVe spaces this produces the famous result: the expression
$\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}$
lands near $\mathbf{v}_{\text{queen}}$, and $\mathbf{v}_{\text{Paris}} -
\mathbf{v}_{\text{France}} + \mathbf{v}_{\text{Italy}}$ lands near
$\mathbf{v}_{\text{Rome}}$.[^jm-analogy] The embedding space has learned
consistent directions for relations like MALE–FEMALE and
CAPITAL-CITY-OF, recovered from raw co-occurrence with no relational supervision.

Two axes show the structure of the _king_ analogy. Project the vectors onto a gender direction
(male to female) and a royalty direction (commoner to royal), and the four words fall
at the corners of a rectangle: _man_ and _woman_ sit low on royalty, _king_ and _queen_
high, with _man_/_king_ on the male side and _woman_/_queen_ on the female side. The
offset $\mathbf{v}_{\text{woman}} - \mathbf{v}_{\text{man}}$ is a step along the gender
axis; adding it to _king_ slides along that same axis and lands on _queen_. That is
exactly $\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}$,
and it works because the space assigned the MALE-FEMALE relation one consistent
direction shared by both the royal and the common pair — the two horizontal edges of
the parallelogram are parallel and equal.

$$
% caption: The king - man + woman = queen analogy as a parallelogram on two learned
% directions: a gender axis (male to female) and a royalty axis (commoner to royal).
% The gender offset (woman minus man) is the same vector that carries king to queen,
% so adding it to king lands on queen. The MALE-FEMALE relation is one shared
% direction, which is why the two horizontal edges are parallel and equal.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (0,0) -- (6.2,0) node[right, font=\scriptsize] {gender};
  \draw[->, black] (0,0) -- (0,4.3) node[above, font=\scriptsize] {royalty};
  \node[font=\scriptsize, anchor=north, text=black] at (0.9,-0.1) {male};
  \node[font=\scriptsize, anchor=north, text=black] at (4.7,-0.1) {female};
  \node[font=\scriptsize, anchor=east, text=black, rotate=90] at (-0.12,1.1) {commoner};
  \node[font=\scriptsize, anchor=east, text=black, rotate=90] at (-0.12,3.3) {royal};
  % four corners
  \coordinate (man)   at (0.9,1.1);
  \coordinate (woman) at (4.7,1.1);
  \coordinate (king)  at (0.9,3.3);
  \coordinate (queen) at (4.7,3.3);
  % parallelogram fill + light edges
  \fill[acc!8] (man) -- (woman) -- (queen) -- (king) -- cycle;
  \draw[black, dashed] (man) -- (king);
  \draw[black, dashed] (woman) -- (queen);
  % the two equal gender offsets (the relation)
  \draw[->, acc, thick] (man) -- (woman);
  \draw[->, acc, thick] (king) -- (queen);
  \node[acc, font=\scriptsize, anchor=north] at (2.8,1.02) {woman - man};
  \node[acc, font=\scriptsize, anchor=north] at (2.8,2.98) {same of\/fset (king to queen)};
  % dots + labels placed clear of the nodes
  \fill[black] (man)   circle (1.7pt); \node[anchor=north east] at (man)   {man};
  \fill[black] (woman) circle (1.7pt); \node[anchor=north west] at (woman) {woman};
  \fill[black] (king)  circle (1.7pt); \node[anchor=south east] at (king)  {king};
  \fill[acc]   (queen) circle (1.9pt); \node[acc, anchor=south west] at (queen) {queen};
\end{tikzpicture}
$$

Formally, for the problem $a : b :: a^\ast : b^\ast$ the parallelogram answer is

$$
\hat{\mathbf{b}}^\ast \;=\; \argmin_{\mathbf{x}} \; \mathrm{distance}\big(\mathbf{x},\; \mathbf{a}^\ast - \mathbf{a} + \mathbf{b}\big) .
$$

The geometry is real but fragile. The nearest vector is often one of the three
input words (so those are excluded by hand), and the method works cleanly only for
frequent words and a few relation types, like capitals and inflections; for many
relations it fails, and it is too simple to model human analogy in general.[^jm-analogy]

## The embedding papers and what came after

Jurafsky & Martin present skip-gram with negative sampling as _a_ method; the primary
sources fix the details. Four public papers
frame the static-embedding era, and one theorem ties them together.

**word2vec** (Mikolov et al., 2013).[^mikolov] The original paper introduced two
architectures, not one: the **CBOW** model predicts a target word from the average
of its surrounding context embeddings, and the **skip-gram** model predicts each
context word from the target — the direction this lesson develops. A companion
paper the same year added the negative-sampling objective and the $0.75$-power noise
distribution used here, and reported the analogy result that made the method famous:
on a test set of _a : b :: c : d_ relations, the vector $\mathbf{v}_b - \mathbf{v}_a +
\mathbf{v}_c$ recovered $d$ for roughly $60\%$ of syntactic and semantic analogies,
far above prior methods. The change that made it fast at billion-word scale
was the shift from a full softmax over $|V|$ to the $k$-negative-sample
sigmoid loss, cutting the per-step cost from $O(|V|)$ to $O(k)$.

**GloVe** (Pennington, Socher, and Manning, 2014).[^glove] Where word2vec learns
from local windows one at a time, GloVe fits vectors to the _global_ co-occurrence
counts directly. Its objective is a weighted least-squares regression on log-counts,
$\sum_{ij} f(X_{ij})\,(\mathbf{w}_i \cdot \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j -
\log X_{ij})^2$, where $X_{ij}$ is the number of times word $j$ appears in the context
of word $i$ and $f$ is a weighting that damps very frequent pairs. The design goal
was to make _ratios_ of co-occurrence probabilities linear in the embedding space, so
that analogy directions fall out by construction. On the standard analogy benchmark
GloVe matched or beat skip-gram at comparable cost.

**The equivalence** (Levy and Goldberg, 2014).[^levygoldberg] The count-based and
prediction-based traditions turned out to be the same object viewed twice. Levy and
Goldberg proved that skip-gram with negative sampling, at its optimum, factorizes a
word-context matrix whose entries are the **shifted PPMI**, $\mathrm{PMI}(w, c) -
\log k$, where $k$ is the number of negative samples. The neural pretext task and the
explicit PPMI matrix of this lesson are, up to the shift, computing the same
association scores — one by SGD, one by counting. This retroactively explained why
the $\alpha = 0.75$ correction helps both, and why simple count models tuned carefully can
match word2vec.

**fastText** (Bojanowski et al., 2017).[^fasttext] Static word2vec assigns nothing
to a word it never saw in training — an out-of-vocabulary word has no vector.
fastText fixes this by embedding **character n-grams** and summing them, so the
vector for _apricots_ is built from the sub-pieces of _apricot_ plus a plural suffix.
Morphologically rich languages, where a stem appears in dozens of inflected forms,
gained the most, and any unseen word can be assigned a vector from its spelling.

All four share the same limitation: one vector per word type,
regardless of sentence. **ELMo** (Peters et al., 2018) removed it by reading
each word through a bidirectional LSTM language model and using its _internal states_
as the word's representation, so _bank_ near _river_ and _bank_ near _money_ receive
different vectors.[^elmo] That is the contextual turn — the same distributional idea,
but computed per occurrence rather than per type — and it is where the next lessons
lead.

## Bias and the road to contextual embeddings

Embeddings inherit whatever is in the text, and text carries human bias.[^jm-bias]
The same arithmetic that yields _king − man + woman ≈ queen_ also yields
_man − computer programmer + woman ≈ homemaker_, and _father : doctor :: mother :
nurse_. Worse, embeddings **amplify** bias: gendered associations come out stronger
in the vector space than in the source statistics. When such embeddings feed a
hiring or search system they can cause **allocational harm**, unfairly distributing
real resources, and **representational harm**, demeaning social groups.[^jm-bias]
Debiasing transformations reduce these effects but do not remove them — an open
problem, and a reason to treat embeddings as artifacts of their training data, not
neutral facts about language.

The deepest limitation is architectural. A **static** embedding assigns each word
_one_ vector, so _bank_ (river) and _bank_ (money) are forced to share a single
compromise point, and every sense a word ever has is averaged into one location.
Meaning, though, depends on context. The next step is a representation that gives a
word a _different_ vector in every sentence it appears in — a **contextual
embedding**, computed on the fly by a
[neural language model](/natural-language-processing/semantics/neural-language-models)
and, at scale, by the
[transformer](/natural-language-processing/transformers/transformers-and-attention).
Static vectors give each word type one fixed point; contextual vectors recompute
the point for each sentence. The ideas built here — the move from word-as-atom to
word-as-point, and the
self-supervised dot-product-and-sigmoid learning — reappear throughout the
[deep-learning](/deep-learning/foundations/what-is-deep-learning) course.

[^jm-w2v]: **Jurafsky & Martin**, §6.8 — Word2vec: short dense embeddings versus long sparse vectors, why dense vectors help generalization, and the self-supervised classifier-whose-weights-are-embeddings framing.
[^jm-sgns]: **Jurafsky & Martin**, §6.8.1–6.8.2 — The Classifier / Learning Skip-Gram Embeddings: sigmoid-of-dot-product probability, positive and negatively sampled context pairs, the cross-entropy objective (Eq. 6.34), the SGD updates (Eqs. 6.38–6.40), and the $P_\alpha$ noise distribution.
[^jm-analogy]: **Jurafsky & Martin**, §6.10 — Semantic Properties of Embeddings: the parallelogram model of analogy, the $king - man + woman \approx queen$ result, and its caveats (input-word exclusion, restriction to frequent words and few relation types).
[^jm-glove]: **Jurafsky & Martin**, §6.8.3; §6.9 — Other Static Embeddings: GloVe as a global-count model based on co-occurrence probability ratios, and the result that skip-gram implicitly factors a shifted PPMI matrix.
[^jm-bias]: **Jurafsky & Martin**, §6.11 — Bias and Embeddings: gendered and stereotyped analogies reproduced and amplified by embeddings, allocational and representational harms, and the open problem of debiasing.
[^mikolov]: **Mikolov, Chen, Corrado, and Dean (2013)**, _Efficient Estimation of Word Representations in Vector Space_, ICLR Workshop; and **Mikolov, Sutskever, Chen, Corrado, and Dean (2013)**, _Distributed Representations of Words and Phrases and their Compositionality_, NeurIPS — the CBOW and skip-gram architectures, the negative-sampling objective, the $0.75$-power noise distribution, and the analogy-completion results on word-vector arithmetic.
[^glove]: **Pennington, Socher, and Manning (2014)**, _GloVe: Global Vectors for Word Representation_, EMNLP — a weighted least-squares model fitting embeddings to global log co-occurrence counts, designed so co-occurrence probability ratios are linear in the vector space.
[^levygoldberg]: **Levy and Goldberg (2014)**, _Neural Word Embedding as Implicit Matrix Factorization_, NeurIPS — the proof that skip-gram with negative sampling implicitly factorizes a word-context matrix of shifted PMI values, $\mathrm{PMI}(w,c) - \log k$.
[^fasttext]: **Bojanowski, Grave, Joulin, and Mikolov (2017)**, _Enriching Word Vectors with Subword Information_, TACL — the fastText model, representing a word as the sum of its character-n-gram embeddings so that out-of-vocabulary and morphologically inflected words receive vectors.
[^elmo]: **Peters, Neumann, Iyyer, Gardner, Clark, Lee, and Zettlemoyer (2018)**, _Deep Contextualized Word Representations_, NAACL — ELMo, deriving a word's vector from the internal states of a bidirectional LSTM language model so that each occurrence of a word gets a context-dependent representation.