---
title: Vector Semantics and Embeddings
module: Semantics
moduleNumber: 3
lessonNumber: 1
order: 301
summary: >
  Vector semantics represents a word's meaning as a point in space, derived from
  the company the word keeps. This first part builds the count-based side: the
  distributional hypothesis, co-occurrence matrices in their term-document and
  word-word forms, cosine as the similarity measure, and the two weightings —
  tf-idf and PPMI — that fix what raw counts get wrong. The result is a sparse,
  interpretable vector for every word, and the setup for the dense embeddings of
  the next lesson.
topics: [Semantics]
sources:
  - book: Jurafsky
    ref: "Ch. 6 — Vector Semantics and Embeddings; §6.2 Vector Semantics; §6.3 Words and Vectors"
  - book: Jurafsky
    ref: "§6.4 Cosine; §6.5 TF-IDF; §6.6 Pointwise Mutual Information"
---

The [n-gram model](/natural-language-processing/foundations/n-gram-language-models)
and [naive Bayes](/natural-language-processing/classification/naive-bayes-and-sentiment)
share a limitation: both treat a word as an atom with no internal structure. _Ongchoi_,
_water spinach_, and _kangkong_ are three distinct symbols with nothing in common,
even though they name the same vegetable. Under that view the only thing a model
knows about a word is whether it is identical to another word. Every notion of
_similar meaning_ — that _delicious_ and _tasty_ are close, that _cat_ and _dog_
are closer than _cat_ and _theorem_ — is invisible.

**Vector semantics** repairs this. It represents each word as a point in a
high-dimensional space, a **vector**, positioned so that words with similar
meanings sit near each other.[^jm-vecsem] A classifier trained on _delicious_
reviews can then generalize to a review that says _tasty_, because the two words
land in the same neighborhood. Meaning is no longer a discrete label but a position
in a geometric space.

## The distributional hypothesis

Where does the geometry come from? The answer, from 1950s linguistics, is that
meaning can be read off from _use_. Suppose you have never met the word _ongchoi_
but you see it in these sentences:

> Ongchoi is delicious sauteed with garlic.
> Ongchoi is superb over rice.
> ...ongchoi leaves with salty sauces...

and you have separately seen _spinach sauteed with garlic over rice_, _chard
stems and leaves are delicious_, and _collard greens and other salty leafy
greens_. The company _ongchoi_ keeps — _garlic_, _rice_, _delicious_, _salty_,
_leaves_ — is the company those leafy greens keep. You can conclude that _ongchoi_
is a leafy green vegetable, having never seen it on a plate.[^jm-vecsem] This is
the **distributional hypothesis**: words that occur in similar contexts tend to
have similar meanings, so a word's distribution over contexts is a usable proxy
for its meaning.

> **Definition (Distributional hypothesis).** The meaning of a word is
> characterized by the distribution of contexts in which it occurs. Two words with
> similar distributions — similar neighboring words — have similar meanings.
> J.R. Firth's slogan: _you shall know a word by the company it keeps_.

The plan follows directly. Count, for each word, the contexts it appears in;
store those counts as a vector; and let vector geometry stand in for meaning.
Everything in this lesson builds on this idea.

## Words and vectors: co-occurrence matrices

The raw material is a **co-occurrence matrix**, a table of how often words appear
together. Two versions matter, differing only in what a "context" is.

### The term-document matrix

In a **term-document matrix** each row is a word in the vocabulary and each column
is a document; cell $(t, d)$ holds the number of times word $t$ occurs in document
$d$. Four words across four Shakespeare plays give a $4 \times 4$ slice:[^jm-tdmat]

| | _As You Like It_ | _Twelfth Night_ | _Julius Caesar_ | _Henry V_ |
| --- | --- | --- | --- | --- |
| **battle** | 1 | 0 | 7 | 13 |
| **good** | 114 | 80 | 62 | 89 |
| **fool** | 36 | 58 | 1 | 4 |
| **wit** | 20 | 15 | 2 | 3 |

Read the matrix two ways. Each _column_ is a **document vector**: _As You Like It_
is $[1, 114, 36, 20]$, a point in a space whose dimensions are the four words. This
is the **vector space model** of information retrieval — documents with similar
word counts are similar documents, and a query is just another vector to compare
against.[^jm-tdmat] Each _row_ is a **word vector**: _battle_ is $[1, 0, 7, 13]$,
a point in a space whose dimensions are the four plays. Similar words occur in
similar documents, so the row for _fool_ $[36, 58, 1, 4]$ and the row for _wit_
$[20, 15, 2, 3]$ — both high in the comedies, low in the histories — come out close.

$$
% caption: Two dimensions of the Shakespeare term-document matrix, plotting each
% play by its counts of $fool$ and $battle$: the two comedies (high $fool$, low
% $battle$) cluster apart from the histories.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % axes
  \draw[->, black] (0,0) -- (6.6,0) node[right] {fool};
  \draw[->, black] (0,0) -- (0,4.3) node[above] {battle};
  \foreach \x/\lab in {1/10,2/20,3/30,4/40,5/50,6/60}
    \draw[black] (\x,0.06) -- (\x,-0.06) node[below, font=\scriptsize] {\lab};
  \foreach \y/\lab in {1/5,2/10,3/15}
    \draw[black] (0.06,\y) -- (-0.06,\y) node[left, font=\scriptsize] {\lab};
  % Henry V [4,13] -> x=0.4, y=2.6
  \draw[->, acc, thick] (0,0) -- (0.4,2.6);
  \node[acc, anchor=south west] at (0.42,2.55) {Henry V (4,13)};
  % Julius Caesar [1,7] -> x=0.1, y=1.4
  \draw[->, acc, thick] (0,0) -- (0.1,1.4);
  \node[acc, anchor=south west] at (0.14,1.35) {Julius Caesar (1,7)};
  % As You Like It [36,1] -> x=3.6, y=0.2
  \draw[->, acc, thick] (0,0) -- (3.6,0.2);
  \node[acc, anchor=south east] at (3.6,0.32) {As You Like It (36,1)};
  % Twelfth Night [58,0] -> x=5.8, y=0
  \draw[->, acc, thick] (0,0) -- (5.8,0.05);
  \node[acc, anchor=north] at (5.4,-0.55) {Twelfth Night (58,0)};
\end{tikzpicture}
$$

### The word-word matrix

For word meaning specifically, a finer context works better than a whole document.
The **word-word matrix** (also **term-term** or **term-context** matrix) has both
rows and columns labeled by words; cell $(w, c)$ counts how often context word $c$
occurred near target word $w$ — typically within a window of $\pm 4$ words across a
training corpus.[^jm-wwmat] The matrix is $|V| \times |V|$. A hand-picked slice
from Wikipedia counts:

| | computer | data | result | pie | sugar |
| --- | --- | --- | --- | --- | --- |
| **cherry** | 2 | 8 | 9 | 442 | 25 |
| **strawberry** | 0 | 0 | 1 | 60 | 19 |
| **digital** | 1670 | 1683 | 85 | 5 | 4 |
| **information** | 3325 | 3982 | 378 | 5 | 13 |

The rows already carry meaning. _cherry_ and _strawberry_ both live near _pie_ and
_sugar_, so their vectors point the same way; _digital_ and _information_ both live
near _computer_ and _data_, so they cluster elsewhere. In a real corpus $|V|$ runs
from 10,000 to 50,000, and almost every cell is zero — these are **sparse**,
**long** vectors.[^jm-wwmat]

## Cosine: measuring similarity

Two words are similar when their vectors point the same direction. The standard
measure is the **cosine** of the angle between them, built from the **dot product**
(inner product),

$$
\mathbf{v} \cdot \mathbf{w} \;=\; \sum_{i=1}^{N} v_i w_i .
$$

The dot product is large when the two vectors have big values in the same
dimensions, and zero for **orthogonal** vectors with no shared dimensions. But used
raw it is a poor similarity metric: it favors _long_ vectors. Frequent words
co-occur with many things, so their vectors have large entries and large dot
products with everything, similarity or not.[^jm-cosine] The fix is to divide out
the lengths, where the length of a vector is $\lVert \mathbf{v} \rVert =
\sqrt{\sum_i v_i^2}$. Dividing the dot product by both lengths gives exactly the
cosine of the angle, since $\mathbf{a} \cdot \mathbf{b} = \lVert \mathbf{a} \rVert
\lVert \mathbf{b} \rVert \cos\theta$:

$$
\cos(\mathbf{v}, \mathbf{w}) \;=\; \frac{\mathbf{v} \cdot \mathbf{w}}{\lVert \mathbf{v} \rVert \, \lVert \mathbf{w} \rVert} \;=\; \frac{\sum_{i=1}^{N} v_i w_i}{\sqrt{\sum_{i=1}^{N} v_i^2}\,\sqrt{\sum_{i=1}^{N} w_i^2}} .
$$

For the non-negative count vectors here the cosine runs from $0$ (orthogonal,
share nothing) to $1$ (identical direction). Two vectors are more similar when the
angle between them is _smaller_ and the cosine is _larger_.

$$
% caption: Cosine similarity in the plane spanned by the contexts $pie$ and
% $computer$. The angle from $information$ to $digital$ is small (high cosine),
% while the angle to $cherry$ is wide (low cosine): a small angle means a large cosine.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (0,0) -- (6.6,0) node[right] {computer};
  \draw[->, black] (0,0) -- (0,4.1) node[above] {pie};
  % cherry: mostly pie, little computer  -> steep
  \draw[->, red, thick] (0,0) -- (0.6,3.6);
  \node[red, anchor=south] at (0.6,3.62) {cherry};
  % digital: mostly computer, little pie -> shallow
  \draw[->, acc, thick] (0,0) -- (5.2,0.7);
  \node[acc, anchor=west] at (5.22,0.72) {digital};
  % information: like digital -> shallow, slightly higher
  \draw[->, acc, thick] (0,0) -- (5.9,1.05);
  \node[acc, anchor=west] at (5.92,1.08) {information};
  % small angle arc between digital and information
  \draw[black] (2.6,0.35) arc (7.7:10.1:2.65);
  \node[black, anchor=north, font=\scriptsize] at (3.55,0.12) {small angle};
  % wide angle arc between information and cherry
  \draw[black] (1.1,0.2) arc (10:80:1.15);
  \node[black, font=\scriptsize] at (1.75,1.35) {wide angle};
\end{tikzpicture}
$$

On the actual Wikipedia counts, $\cos(\mathbf{v}_{\text{digital}},
\mathbf{v}_{\text{information}}) = 0.996$ while $\cos(\mathbf{v}_{\text{cherry}},
\mathbf{v}_{\text{information}}) = 0.017$: the measure puts _information_ far
closer to _digital_ than to _cherry_, as it should.[^jm-cosine]

## Weighting: why raw counts mislead

Raw frequency is skewed and undiscriminating. The most frequent context words —
_the_, _it_, _they_ — sit near everything and so tell you almost nothing about
which words are alike. Yet it is the words that appear in only a few, specific
contexts — _pie_, say — that are informative about _cherry_. Raw
counts get this backwards: they reward ubiquity. Two weighting schemes fix it, one
per matrix type.[^jm-tfidf]

### TF-IDF, for term-document matrices

**TF-IDF** balances two opposing intuitions, and is the product of two terms.

The **term frequency** rewards a word for appearing often in a document, but with
diminishing returns — a word seen 100 times is not 100 times more relevant than a
word seen once — so we squash the count with a log:

$$
\mathrm{tf}_{t,d} \;=\; \log_{10}\!\big(\mathrm{count}(t, d) + 1\big) .
$$

The **inverse document frequency** penalizes a word for appearing in _many_
documents. A word confined to a few documents discriminates them from the rest; a
word in every document (like _good_ across all 37 Shakespeare plays) discriminates
nothing. With $N$ documents in the collection and $\mathrm{df}_t$ the number
containing term $t$,

$$
\mathrm{idf}_t \;=\; \log_{10}\!\left(\frac{N}{\mathrm{df}_t}\right) .
$$

A word in all $N$ documents gets $\mathrm{idf} = \log_{10}(1) = 0$; a word in one
gets the maximum. The **tf-idf** weight is their product,

$$
w_{t,d} \;=\; \mathrm{tf}_{t,d} \times \mathrm{idf}_t .
$$

Under tf-idf the dimension for _good_ collapses to $0$ everywhere — it appears in
every play, so its idf is zero — and the near-ubiquitous _fool_ is sharply
downweighted, leaving the discriminative words to carry the meaning.[^jm-tfidf]

#### Worked example: tf-idf on the Shakespeare matrix

Take the collection of $N = 37$ Shakespeare plays and weight the four-word slice
above. First the idf column, from each word's document frequency. _good_ appears in
all $37$ plays, so $\mathrm{df}_{\text{good}} = 37$ and $\mathrm{idf}_{\text{good}} =
\log_{10}(37/37) = 0$. _fool_ appears in $36$, giving $\mathrm{idf}_{\text{fool}} =
\log_{10}(37/36) = 0.012$. _wit_ appears in $34$, giving $\log_{10}(37/34) = 0.037$,
and _battle_ in $21$, giving $\log_{10}(37/21) = 0.246$ — the rarest of the four, so
the most discriminative.

| term | df | $\mathrm{idf} = \log_{10}(37/\mathrm{df})$ |
| --- | --- | --- |
| **good** | 37 | $0.000$ |
| **fool** | 36 | $0.012$ |
| **wit** | 34 | $0.037$ |
| **battle** | 21 | $0.246$ |

Now the tf-idf cell for _wit_ in _As You Like It_. The raw count is $20$, so the
squashed term frequency is $\mathrm{tf} = \log_{10}(20 + 1) = \log_{10}(21) = 1.322$,
and the weight is $w = \mathrm{tf} \times \mathrm{idf} = 1.322 \times 0.037 = 0.049$.
The same procedure over the whole slice replaces every count with a weight:

| | _As You Like It_ | _Twelfth Night_ | _Julius Caesar_ | _Henry V_ |
| --- | --- | --- | --- | --- |
| **battle** | $0.074$ | $0$ | $0.220$ | $0.280$ |
| **good** | $0$ | $0$ | $0$ | $0$ |
| **fool** | $0.019$ | $0.021$ | $0.004$ | $0.008$ |
| **wit** | $0.049$ | $0.044$ | $0.018$ | $0.022$ |

Two things happened. The _good_ row is now zero everywhere — a word in every
document carries no information about which document you are in, and its idf of $0$
enforces exactly that. And _battle_, once a near-invisible count of $1$ in _As You
Like It_, is now the row with the largest weights, because its rarity across the
collection makes each occurrence informative. The weighting inverted the raw ranking:
frequency alone had _good_ dominating, and tf-idf shifts the discriminative power to
_battle_.[^jm-tfidf]

$$
% caption: tf-idf as a product of two curves against a word's document frequency.
% Term frequency (per document) is roughly flat; inverse document frequency falls
% as $df$ grows, hitting $0$ at $df = N$. Their product peaks for words frequent in
% a document but rare across the collection, and vanishes for ubiquitous words.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (6.6,0) node[right, black, font=\scriptsize] {df (documents containing t)};
  \draw[->, black] (0,0) -- (0,3.4) node[above, black, font=\scriptsize] {weight};
  % idf curve: log(N/df), decreasing, hits 0 at df=N (x=6)
  \draw[acc, thick, domain=0.6:6.0, samples=60, variable=\x]
    plot ({\x}, {2.9*ln(6.0/\x)/ln(10)});
  \node[acc, anchor=west] at (5.1,2.35) {idf};
  % tf: flat-ish
  \draw[red, thick] (0.3,1.6) -- (6.0,1.6);
  \node[red, anchor=west] at (5.1,1.75) {tf};
  % product (tf x idf), peaks mid, drops to 0 at df=N
  \draw[black, very thick, domain=0.6:6.0, samples=60, variable=\x]
    plot ({\x}, {1.6*2.9*ln(6.0/\x)/ln(10)/2.9});
  \node[black, anchor=west] at (4.3,0.55) {tf-idf};
  \node[font=\scriptsize, anchor=north] at (6.0,-0.05) {N};
  \draw[black, dashed] (6.0,0) -- (6.0,0.05);
\end{tikzpicture}
$$

### PPMI, for word-word matrices

When the dimensions are context words rather than documents, the natural question
is different: does word $w$ co-occur with context $c$ _more than we would expect by
chance_? **Pointwise mutual information** measures exactly that — the log ratio of
the joint probability to the product of the marginals:[^jm-ppmi]

$$
\mathrm{PMI}(w, c) \;=\; \log_2 \frac{P(w, c)}{P(w)\,P(c)} .
$$

The numerator is how often $w$ and $c$ actually occur together; the denominator is
how often they would if they were independent. A ratio above $1$ (positive PMI)
means association; below $1$ (negative PMI) means the pair co-occurs less than
chance. But
negative PMI is unreliable — asserting that two words co-occur _less_ than chance
demands an enormous corpus to establish — so in practice we clip it to zero,
giving **positive PMI**:

$$
\mathrm{PPMI}(w, c) \;=\; \max\!\left(\log_2 \frac{P(w, c)}{P(w)\,P(c)},\; 0\right) .
$$

Given a co-occurrence matrix with counts $f_{ij}$, the probabilities are the
maximum-likelihood estimates: $p_{ij} = f_{ij} / \sum_{ij} f_{ij}$, with $p_{i\ast}$
and $p_{\ast{}j}$ the row and column marginals. PPMI has one known flaw — it is biased
toward rare events, giving very infrequent words inflated scores. The common fix
raises the context probability to a power $\alpha = 0.75$, which inflates $P(c)$
for rare contexts and so damps their PMI.[^jm-ppmi] The same $\alpha = 0.75$
correction reappears inside word2vec.

#### Worked example: PPMI from co-occurrence counts

Carry the whole computation through on a five-context slice of Wikipedia counts,
with the row and column marginals attached. Pretend, for arithmetic's sake, that
these are the only words and contexts in the corpus, so the grand total is
$\sum_{ij} f_{ij} = 11716$.[^jm-ppmi]

| | computer | data | result | pie | sugar | **count(w)** |
| --- | --- | --- | --- | --- | --- | --- |
| **cherry** | 2 | 8 | 9 | 442 | 25 | 486 |
| **strawberry** | 0 | 0 | 1 | 60 | 19 | 80 |
| **digital** | 1670 | 1683 | 85 | 5 | 4 | 3447 |
| **information** | 3325 | 3982 | 378 | 5 | 13 | 7703 |
| **count(c)** | 4997 | 5673 | 473 | 512 | 61 | **11716** |

Compute $\mathrm{PPMI}(\textit{information}, \textit{data})$. The joint probability is
the cell over the grand total, $p_{ij} = 3982/11716 = 0.3399$. The two marginals are
$p_{i\ast} = P(\textit{information}) = 7703/11716 = 0.6575$ and $p_{*j} = P(\textit{data})
= 5673/11716 = 0.4842$. Then

$$
\mathrm{PPMI}(\textit{information}, \textit{data}) \;=\; \max\!\left(\log_2 \frac{0.3399}{0.6575 \times 0.4842},\; 0\right) \;=\; \max(0.094,\, 0) \;=\; 0.094 .
$$

A small positive value: _information_ and _data_ co-occur slightly more than chance.
Now a pair that co-occurs less than chance, _cherry_ and _computer_: the joint is $2/11716 =
0.00017$, the marginals are $486/11716 = 0.0415$ and $4997/11716 = 0.4265$, so
$\log_2\!\big(0.00017 / (0.0415 \times 0.4265)\big) = \log_2(0.0096) = -6.7$. Negative,
so PPMI clips it to $0$ rather than assert a co-occurrence _deficit_ that cannot be
reliably measured. Applying this to every cell turns the raw counts into
the PPMI matrix:

| | computer | data | result | pie | sugar |
| --- | --- | --- | --- | --- | --- |
| **cherry** | $0$ | $0$ | $0$ | $4.38$ | $3.30$ |
| **strawberry** | $0$ | $0$ | $0$ | $4.10$ | $5.51$ |
| **digital** | $0.18$ | $0.01$ | $0$ | $0$ | $0$ |
| **information** | $0.02$ | $0.09$ | $0.28$ | $0$ | $0$ |

The block structure the raw counts only hinted at is now explicit: _cherry_ and
_strawberry_ carry all their weight on _pie_ and _sugar_, _digital_ and _information_
on _computer_ and _data_, and the cross terms are zeroed out. The two fruit rows are
close, the two tech rows are close, and the two groups are orthogonal — similarity
is now encoded in the geometry, through association rather than raw frequency.[^jm-ppmi]

## From counts to embeddings

The vectors built so far already work: cosine over a PPMI-weighted word-word
matrix recovers real similarity, and every dimension is an interpretable context
word. But they have two practical defects. They are **long** — one dimension per
vocabulary word, so tens of thousands wide — and **sparse**, almost all zeros.
The next lesson keeps the distributional idea but compresses it, learning a
short, dense vector per word whose dimensions mean nothing individually yet
capture similarity better than any count matrix. That is the move from _counting_
contexts to _predicting_ them.

This continues in [Static Word Embeddings: word2vec and After](/natural-language-processing/semantics/static-word-embeddings).

[^jm-vecsem]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), §6.2 — Vector Semantics: representing words as points in space so that similar words are nearby, motivated by the distributional idea (Firth, Harris, Joos) that a word's meaning is fixed by its distribution over contexts.
[^jm-tdmat]: **Jurafsky & Martin**, §6.3.1 — Vectors and Documents: the term-document matrix, its columns as document vectors in the vector space model of information retrieval, and its rows as word vectors over documents.
[^jm-wwmat]: **Jurafsky & Martin**, §6.3.3 — Words as Vectors, Word Dimensions: the $|V| \times |V|$ word-word (term-context) co-occurrence matrix built from a $\pm 4$-word window, and why such vectors are long and sparse.
[^jm-cosine]: **Jurafsky & Martin**, §6.4 — Cosine for Measuring Similarity: the dot product as similarity, its bias toward long (frequent) vectors, and the length-normalized cosine, with the $digital$/$information$/$cherry$ worked example.
[^jm-tfidf]: **Jurafsky & Martin**, §6.5 — TF-IDF: Weighing Terms in the Vector: log-squashed term frequency times inverse document frequency, collapsing ubiquitous words like $good$ to weight zero.
[^jm-ppmi]: **Jurafsky & Martin**, §6.6 — Pointwise Mutual Information: PMI as the log ratio of joint to independent probability, its unreliability for negative values, positive PMI, and the $\alpha = 0.75$ correction for rare-context bias.