Static Word Embeddings: word2vec and After

This builds on Vector Semantics and Embeddings, which represented each word as a long, sparse vector of context counts, weighted by tf-idf or PPMI. Those vectors work, but they carry one dimension per vocabulary word and are almost all zeros. Here we compress them: instead of counting the contexts a word appears in, we train a small classifier to predict them, and keep the weights it learns as the word's dense vector. Same distributional idea, a far smaller and more useful representation.

Dense embeddings and word2vec

The vectors so far are sparse and long: dimension $|V|$, mostly zeros. The alternative is a dense vector — short (50 to 1000 dimensions), with real-valued entries that can be negative and no per-dimension interpretation. These are embeddings.¹ Dense vectors work better on essentially every NLP task. A classifier over 300 dense dimensions has far fewer weights to fit than one over 50,000 sparse dimensions, so it generalizes better; and a dense space can capture synonymy that a sparse one misses, where car and automobile occupy distinct, unrelated dimensions.¹

Word2vec is the method that made dense embeddings routine.¹ Rather than computing anything about counts directly, it trains a binary classifier on a pretext task — is context word $c$ likely to occur near target word $w$? — then discards the predictions and keeps the learned weights as the embeddings. Because the training data is just running text (a word and its actual neighbors are positive examples, needing no human labels), this is self-supervision — the same idea that later scales up to language-model pretraining.

Skip-gram with negative sampling

The specific algorithm is skip-gram with negative sampling (SGNS). Take a target word $w$ and a window of $\pm L$ words around it as its context. Each (target, true-context) pair is a positive example. To teach the classifier what non-neighbors look like, we manufacture negative examples: for each positive pair, sample $k$ noise words $c_{\text{neg}}$ at random from the vocabulary, each paired with the same target.²

The classifier decides by embedding similarity. It keeps two vectors per word — a target embedding $\mathbf{w}$ and a context embedding $\mathbf{c}$ — and models the probability that $c$ is a real context of $w$ as the sigmoid of their dot product:²

The dot product is a raw similarity; the sigmoid squashes it into a probability. For one positive pair $(w,c_{\text{pos}})$ with noise words $c_{{\text{neg}}_1},\dots ,c_{{\text{neg}}_k}$, the loss is minimized when the positive pair looks like neighbors and every noise pair looks like non-neighbors,

Minimizing this by stochastic gradient descent pulls $\mathbf{w}$ toward the context embeddings of words it truly occurs with and pushes it away from the noise words. The noise words themselves are drawn not from the plain unigram distribution $P(w)$ but from the weighted $P_{\alpha }(w)\propto \mathrm{count}(w{)}^{0.75}$ — the same $\alpha =0.75$ that tamed PPMI, here giving rare words a slightly higher chance of being sampled as noise.² The full procedure:

The learner stores two matrices, the target matrix $\mathbf{W}$ and the context matrix $\mathbf{C}$, each holding one $d$-dimensional vector per vocabulary word. When training ends, the embedding of word $i$ is usually taken as ${\mathbf{w}}_i+{\mathbf{c}}_i$ (or just ${\mathbf{w}}_i$, discarding $\mathbf{C}$).² The predictions are never used; the weights are the embeddings.

The gradient, derived

The three update lines above come from differentiating the loss. Write ${\sigma }_{\text{pos}}=\sigma ({\mathbf{c}}_{\text{pos}}\cdot \mathbf{w})$ for the model's current probability that the true pair is a neighbor. Since $\frac{d}{dx}\log \sigma (x)=1-\sigma (x)$ and $\frac{d}{dx}\log \sigma (-x)=-\sigma (x)$, differentiating $L_{\mathrm{CE}}=-\log \sigma ({\mathbf{c}}_{\text{pos}}\cdot \mathbf{w})-{\sum }_i\log \sigma (-{\mathbf{c}}_{{\text{neg}}_i}\cdot \mathbf{w})$ by the chain rule gives²

Read the signs. For the positive pair, ${\sigma }_{\text{pos}}-1$ is negative (the probability is below $1$), so $-\eta [{\sigma }_{\text{pos}}-1]\mathbf{w}$ adds a multiple of $\mathbf{w}$ to ${\mathbf{c}}_{\text{pos}}$: the two vectors move toward each other, raising their future dot product. For a noise pair, $\sigma (\cdot )>0$ pushes ${\mathbf{c}}_{{\text{neg}}_i}$ away from $\mathbf{w}$. Attraction to real neighbors, repulsion from noise — exactly the two goals the loss encodes.

Worked example: one SGNS update step

Take target apricot with two-dimensional embeddings, one positive context jam and one noise word tolstoy, learning rate $\eta =0.1$:

Dot products. ${\mathbf{c}}_{\text{jam}}\cdot \mathbf{w}=0.4(0.5)+0.3(0.2)=0.26$, and ${\mathbf{c}}_{\text{tolstoy}}\cdot \mathbf{w}=-0.2(0.5)+0.6(0.2)=0.02$.

Sigmoids. $\sigma (0.26)=1/(1+e^{-0.26})=0.565$ and $\sigma (0.02)=0.505$. The model currently gives jam a neighbor probability of only $0.565$ (it should be near $1$) and tolstoy a probability of $0.505$ (it should be near $0$). Both errors produce useful gradients.

Updates. For the positive context, $[{\sigma }_{\text{pos}}-1]=-0.435$, so

For the noise word, $\sigma (0.02)=0.505$, so

And the target embedding, which receives both updates at once,

Check the effect on the dot products that matter. The new ${\mathbf{c}}_{\text{jam}}\cdot {\mathbf{w}}_{\text{new}}=0.422(0.527)+0.309(0.183)=0.279$, up from $0.26$ — apricot and jam grew more similar. The new ${\mathbf{c}}_{\text{tolstoy}}\cdot {\mathbf{w}}_{\text{new}}=-0.225(0.527)+0.590(0.183)=-0.011$, down from $0.02$ — apricot and tolstoy grew less similar. One step of the pretext task moved the geometry exactly as intended; a corpus of billions of such steps produces the final space.

The analogy parallelogram

Embeddings trained this way turn out to encode relations, not just similarities, in their geometry. The parallelogram model of analogy, proposed for cognition long before word2vec, solves a is to b as $a^{\ast }$ is to what? by vector arithmetic: compute the offset $\mathbf{b}-\mathbf{a}$ that carries $a$ to $b$, add it to $a^{\ast }$, and return the nearest word.³

On real word2vec and GloVe spaces this produces the famous result: the expression ${\mathbf{v}}_{\text{king}}-{\mathbf{v}}_{\text{man}}+{\mathbf{v}}_{\text{woman}}$ lands near ${\mathbf{v}}_{\text{queen}}$, and ${\mathbf{v}}_{\text{Paris}}-{\mathbf{v}}_{\text{France}}+{\mathbf{v}}_{\text{Italy}}$ lands near ${\mathbf{v}}_{\text{Rome}}$.³ The embedding space has learned consistent directions for relations like MALE–FEMALE and CAPITAL-CITY-OF, recovered from raw co-occurrence with no relational supervision.

Two axes show the structure of the king analogy. Project the vectors onto a gender direction (male to female) and a royalty direction (commoner to royal), and the four words fall at the corners of a rectangle: man and woman sit low on royalty, king and queen high, with man/king on the male side and woman/queen on the female side. The offset ${\mathbf{v}}_{\text{woman}}-{\mathbf{v}}_{\text{man}}$ is a step along the gender axis; adding it to king slides along that same axis and lands on queen. That is exactly ${\mathbf{v}}_{\text{king}}-{\mathbf{v}}_{\text{man}}+{\mathbf{v}}_{\text{woman}}$, and it works because the space assigned the MALE-FEMALE relation one consistent direction shared by both the royal and the common pair — the two horizontal edges of the parallelogram are parallel and equal.

Formally, for the problem $a:b::a^{\ast }:b^{\ast }$ the parallelogram answer is

The geometry is real but fragile. The nearest vector is often one of the three input words (so those are excluded by hand), and the method works cleanly only for frequent words and a few relation types, like capitals and inflections; for many relations it fails, and it is too simple to model human analogy in general.³

The embedding papers and what came after

Jurafsky & Martin present skip-gram with negative sampling as a method; the primary sources fix the details. Four public papers frame the static-embedding era, and one theorem ties them together.

word2vec (Mikolov et al., 2013).⁴ The original paper introduced two architectures, not one: the CBOW model predicts a target word from the average of its surrounding context embeddings, and the skip-gram model predicts each context word from the target — the direction this lesson develops. A companion paper the same year added the negative-sampling objective and the $0.75$-power noise distribution used here, and reported the analogy result that made the method famous: on a test set of a : b :: c : d relations, the vector ${\mathbf{v}}_b-{\mathbf{v}}_a+{\mathbf{v}}_c$ recovered $d$ for roughly $60\%$ of syntactic and semantic analogies, far above prior methods. The change that made it fast at billion-word scale was the shift from a full softmax over $|V|$ to the $k$-negative-sample sigmoid loss, cutting the per-step cost from $O(|V|)$ to $O(k)$.

GloVe (Pennington, Socher, and Manning, 2014).⁵ Where word2vec learns from local windows one at a time, GloVe fits vectors to the global co-occurrence counts directly. Its objective is a weighted least-squares regression on log-counts, ${\sum }_{ij}f(X_{ij})({\mathbf{w}}_i\cdot {\tilde{\mathbf{w}}}_j+b_i+{\tilde{b}}_j-\log X_{ij}{)}^2$, where $X_{ij}$ is the number of times word $j$ appears in the context of word $i$ and $f$ is a weighting that damps very frequent pairs. The design goal was to make ratios of co-occurrence probabilities linear in the embedding space, so that analogy directions fall out by construction. On the standard analogy benchmark GloVe matched or beat skip-gram at comparable cost.

The equivalence (Levy and Goldberg, 2014).⁶ The count-based and prediction-based traditions turned out to be the same object viewed twice. Levy and Goldberg proved that skip-gram with negative sampling, at its optimum, factorizes a word-context matrix whose entries are the shifted PPMI, $\mathrm{PMI}(w,c)-\log k$, where $k$ is the number of negative samples. The neural pretext task and the explicit PPMI matrix of this lesson are, up to the shift, computing the same association scores — one by SGD, one by counting. This retroactively explained why the $\alpha =0.75$ correction helps both, and why simple count models tuned carefully can match word2vec.

fastText (Bojanowski et al., 2017).⁷ Static word2vec assigns nothing to a word it never saw in training — an out-of-vocabulary word has no vector. fastText fixes this by embedding character n-grams and summing them, so the vector for apricots is built from the sub-pieces of apricot plus a plural suffix. Morphologically rich languages, where a stem appears in dozens of inflected forms, gained the most, and any unseen word can be assigned a vector from its spelling.

All four share the same limitation: one vector per word type, regardless of sentence. ELMo (Peters et al., 2018) removed it by reading each word through a bidirectional LSTM language model and using its internal states as the word's representation, so bank near river and bank near money receive different vectors.⁸ That is the contextual turn — the same distributional idea, but computed per occurrence rather than per type — and it is where the next lessons lead.

Bias and the road to contextual embeddings

Embeddings inherit whatever is in the text, and text carries human bias.⁹ The same arithmetic that yields king − man + woman ≈ queen also yields man − computer programmer + woman ≈ homemaker, and father : doctor :: mother : nurse. Worse, embeddings amplify bias: gendered associations come out stronger in the vector space than in the source statistics. When such embeddings feed a hiring or search system they can cause allocational harm, unfairly distributing real resources, and representational harm, demeaning social groups.⁹ Debiasing transformations reduce these effects but do not remove them — an open problem, and a reason to treat embeddings as artifacts of their training data, not neutral facts about language.

The deepest limitation is architectural. A static embedding assigns each word one vector, so bank (river) and bank (money) are forced to share a single compromise point, and every sense a word ever has is averaged into one location. Meaning, though, depends on context. The next step is a representation that gives a word a different vector in every sentence it appears in — a contextual embedding, computed on the fly by a neural language model and, at scale, by the transformer. Static vectors give each word type one fixed point; contextual vectors recompute the point for each sentence. The ideas built here — the move from word-as-atom to word-as-point, and the self-supervised dot-product-and-sigmoid learning — reappear throughout the deep-learning course.