---
title: "Multimodal Contrastive Learning"
module: Large Models & Agents
moduleNumber: 10
lessonNumber: 10
order: 1010
summary: >
  A multimodal model places images, text, and audio in one representation space,
  so a picture and its caption land close together. This first part builds the
  contrastive route: the shared embedding space and its residual modality gap, the
  Vision Transformer image encoder (patch embedding, CLS token, position embeddings,
  with shapes), the symmetric InfoNCE loss that trains the CLIP dual encoder from a
  batch similarity matrix (with a worked numeric step), and zero-shot classification
  as a softmax over class-prompt embeddings. Fusion and vision-language models
  continue in part two.
topics: [Large Models & Agents]
sources:
  - book: Chollet
    ref: "Ch. 11 — combining modalities; multimodal architectures"
---

[Large language models](/deep-learning/large-models-and-agents/large-language-models)
read one modality: text. A **multimodal model** maps
inputs from several modalities, images, text, audio, into a single representation
where comparison across modalities is a dot product, so that a photograph and the
sentence describing it become neighbors. Everything in this lesson follows from
one design problem: putting a $224 \times 224$ image and a string of tokens into
the _same_ space, and then letting them interact.

> **Definition (Multimodal model).** A model with encoders
> $f_{\text{img}}, f_{\text{txt}}, \dots$ for two or more modalities and a rule
> for combining their outputs. A **contrastive** model maps each modality to a
> shared vector space $\mathbb{R}^d$ and compares by similarity; a **fusion**
> model interleaves the modalities inside a network so one can attend to another.

## The shared embedding space and the modality gap

The cleanest multimodal design gives each modality its own encoder and forces
their outputs into one space. An image $x$ and a text $y$ become unit vectors
$f_{\text{img}}(x), f_{\text{txt}}(y) \in \mathbb{R}^d$, and their semantic
agreement is the cosine similarity $f_{\text{img}}(x)^{T} f_{\text{txt}}(y)$.

> **Definition (Shared embedding space).** A vector space $\mathbb{R}^d$ together
> with encoders $f_{\text{img}}, f_{\text{txt}}$ whose images are
> $\ell_2$-normalized, so that for any image $x$ and text $y$ the cosine
> similarity $s(x, y) = f_{\text{img}}(x)^{T} f_{\text{txt}}(y) \in [-1, 1]$
> measures how well the caption matches the picture.

The alignment is imperfect. Even after contrastive training the
two modalities occupy _separate cones_: image embeddings cluster on one side of
the sphere and text embeddings on another, with a measurable angular separation
between the two clouds. This is the **modality gap**, a consequence of separate
encoders and random initialization that contrastive training narrows but never
fully closes.[^liang-gap]

> **Definition (Modality gap).** The systematic offset between the image and text
> clouds in a shared space: the two sets of embeddings concentrate in distinct
> regions of the unit sphere, so the mean image embedding and the mean text
> embedding differ by a nonzero vector even for matched pairs. The contrastive
> loss aligns _relative_ geometry (matched pairs closer than mismatched) without
> erasing this absolute offset.

$$
% caption: A shared sphere holds image (blue) and text (green) embeddings. Matched
% pairs are pulled together, yet the two clouds keep an angular modality gap.
\begin{tikzpicture}[>=stealth, font=\footnotesize, x=1cm, y=1cm]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % the sphere
  \draw[black, thick] (0,0) circle (2.2);
  \draw[black, thick] (0,0) ellipse (2.2 and 0.7);
  % image cloud (upper-left cone)
  \fill[acc] (-1.35,1.30) circle (1.8pt);
  \fill[acc] (-1.05,1.55) circle (1.8pt);
  \fill[acc] (-1.55,1.05) circle (1.8pt);
  \fill[acc] (-0.85,1.15) circle (1.8pt);
  \node[acc, anchor=south east] at (-1.0,1.7) {images};
  % text cloud (upper-right cone)
  \fill[green] (1.35,1.30) circle (1.8pt);
  \fill[green] (1.05,1.55) circle (1.8pt);
  \fill[green] (1.55,1.05) circle (1.8pt);
  \fill[green] (0.85,1.15) circle (1.8pt);
  \node[green, anchor=south west] at (1.0,1.7) {text};
  % a matched pair pulled together (dashed link)
  \draw[red, thick, dashed] (-0.85,1.15) -- (0.85,1.15);
  \node[red, anchor=north] at (0,1.05) {matched pair};
  % gap arrow between cloud centers
  \draw[<->, black, thick] (-1.2,0.35) -- (1.2,0.35);
  \node[black, anchor=north] at (0,0.3) {gap};
\end{tikzpicture}
$$

## The image encoder: a Vision Transformer

Before two modalities can share a space, the image needs an encoder that produces
a single vector the way a text Transformer produces one from a sentence. The
**Vision Transformer (ViT)** does this by cutting the image into fixed patches,
treating each patch as a token, and running the ordinary Transformer encoder over
that sequence.[^dosovitskiy-vit] The only new step is turning a grid of pixels
into a sequence of embeddings.

> **Definition (Patch embedding).** Split an image
> $x \in \mathbb{R}^{H \times W \times C}$ into a grid of non-overlapping
> $P \times P$ patches. There are $N_p = HW / P^2$ of them, each a vector of
> $P^2 C$ pixels once flattened. A single learned linear map
> $E \in \mathbb{R}^{(P^2 C) \times d}$ projects every flattened patch to a
> $d$-dimensional token, so the image becomes a sequence
> $X_{\text{patch}} \in \mathbb{R}^{N_p \times d}$.

Take the standard configuration: a $224 \times 224$ RGB image, patch size
$P = 16$. The grid is $14 \times 14$, so $N_p = 196$ patches, each flattened to
$16^2 \cdot 3 = 768$ pixel values, then projected to $d = 768$. The linear
projection $E$ is literally a $\mathrm{Conv2d}$ with kernel and stride both $16$:
one convolution stride per patch, output channels $= d$. The shapes flow as

$$
\underbrace{224 \times 224 \times 3}_{\text{image}}
\;\longrightarrow\;
\underbrace{14 \times 14 \times 768}_{\text{conv, stride }16}
\;\longrightarrow\;
\underbrace{196 \times 768}_{\text{flatten to tokens}}.
$$

Two additions finish the input. A learned **[CLS] token**
$x_{\text{cls}} \in \mathbb{R}^{d}$ is prepended to the sequence; after the
encoder its output row is the image's single summary vector, the analogue of a
sentence embedding. And because attention is permutation-invariant, a learned
**position embedding** $E_{\text{pos}} \in \mathbb{R}^{(N_p + 1) \times d}$ is
added so the model can tell a top-left patch from a bottom-right one. The encoder
input is

$$
Z_0 = \big[\, x_{\text{cls}} \, ; \, X_{\text{patch}} \,\big] + E_{\text{pos}}
\;\in\; \mathbb{R}^{(N_p + 1) \times d}
= \mathbb{R}^{197 \times 768}.
$$

A stack of $L$ standard [Transformer](/deep-learning/architectures/the-transformer-architecture)
encoder blocks (multi-head self-attention, then an MLP, each with a residual and
LayerNorm) maps $Z_0$ to $Z_L$ of the same shape. The image representation is the
[CLS] output row $z_{\text{cls}} = (Z_L)_0 \in \mathbb{R}^d$, which a final linear
head projects to the shared space. Nothing about attention changed; only the
tokenizer did.

$$
% caption: The Vision Transformer. A $224{\times}224$ image is cut into 196 patches
% of $16{\times}16$, each linearly embedded to a 768-vector; a CLS token and position
% embeddings are added, and a Transformer encoder produces the image vector.
\begin{tikzpicture}[font=\footnotesize,>=stealth,
  cell/.style={draw=acc, thick, minimum size=4.6mm, inner sep=0pt, fill=acc!10},
  tokn/.style={draw=acc, thick, minimum width=6mm, minimum height=6mm, inner sep=0pt, fill=acc!14, font=\scriptsize},
  cls/.style={draw=green, thick, minimum width=6mm, minimum height=6mm, inner sep=0pt, fill=green!16, font=\scriptsize},
  enc/.style={draw=black, thick, minimum width=10mm, minimum height=34mm, align=center, fill=black!7, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % patch grid
  \foreach \c in {0,1,2}{ \foreach \r in {0,1,2}{
    \node[cell] at (\c*0.5, \r*0.5) {};
  }}
  \node[black, anchor=south] at (0.5,1.75) {\texttt{image}};
  \node[black, anchor=north, font=\footnotesize] at (0.5,-0.35) {\texttt{224x224x3}};
  \node[black, anchor=north, font=\footnotesize] at (0.5,-0.85) {\texttt{patch 16}};
  % arrow to token row
  \draw[->, acc, thick] (1.35,0.5) -- (2.35,0.5);
  \node[acc, anchor=south, font=\footnotesize] at (1.85,0.55) {\texttt{embed}};
  % token sequence (CLS + patches)
  \node[cls] (t0) at (2.9,0.5) {c};
  \node[tokn] (t1) at (3.5,0.5) {1};
  \node[tokn] (t2) at (4.1,0.5) {2};
  \node[tokn] (t3) at (4.7,0.5) {3};
  \node[black, font=\footnotesize] at (5.25,0.5) {\texttt{...}};
  \node[tokn] (t4) at (5.85,0.5) {196};
  \node[black, anchor=south, font=\footnotesize] at (4.3,1.05) {\texttt{197 tokens x 768}};
  \node[green, anchor=north, font=\footnotesize] at (2.9,0.05) {\texttt{CLS}};
  % position embeddings added
  \node[black, anchor=north, font=\footnotesize] at (4.3,-0.35) {\texttt{+ position embeddings}};
  % encoder
  \draw[->, acc, thick] (6.35,0.5) -- (7.05,0.5);
  \node[enc] (enc) at (7.75,0.5) {Trans-\\former\\encoder\\(L x)};
  % output CLS vector
  \draw[->, green, thick] (8.35,0.5) -- (9.15,0.5);
  \node[cls, minimum width=7mm, minimum height=7mm] (out) at (9.6,0.5) {z};
  \node[green, anchor=south, font=\footnotesize] at (9.6,1.05) {\texttt{image vector}};
  \node[green, anchor=north, font=\footnotesize] at (9.6,0.0) {\texttt{768}};
\end{tikzpicture}
$$

The patch count sets the compute: self-attention is $O(N_p^2)$, so halving the
patch size ($P = 16 \to 8$) quadruples the token count ($196 \to 784$) and raises
attention cost roughly $16\times$. That is the accuracy-versus-cost dial a ViT
exposes, and it is why CLIP ships several patch sizes.

## Contrastive vision-language pretraining

The dominant way to learn a shared space is **contrastive language-image
pretraining (CLIP)**.[^radford-clip] Two encoders, a Vision Transformer for the
image and a Transformer for the text, map a large batch of paired
(image, caption) examples into $\mathbb{R}^d$, and a contrastive loss pulls each
image toward its own caption and pushes it away from every other caption in the
batch.

> **Definition (CLIP).** A pair of encoders trained so that, over a batch of $N$
> image-text pairs $\braces{(x_i, y_i)}_{i=1}^{N}$, the matched embeddings
> $\parens{f_{\text{img}}(x_i), f_{\text{txt}}(y_i)}$ have high similarity and all
> $N^2 - N$ mismatched cross-pairs have low similarity. The image encoder is a
> [Vision Transformer](/deep-learning/architectures/the-transformer-architecture)
> and the text encoder is a causal Transformer; both project to a common $d$.

### The batch similarity matrix

Embed the batch and $\ell_2$-normalize: let $u_i = f_{\text{img}}(x_i)$ and
$v_j = f_{\text{txt}}(y_j)$, each on the unit sphere. The pairwise cosine
similarities form an $N \times N$ matrix $S$, scaled by a learned temperature
$\tau$,

$$
S_{ij} = \frac{u_i^{T} v_j}{\tau}, \qquad \norm{u_i} = \norm{v_j} = 1.
$$

The correct matches lie on the diagonal: $S_{ii}$ should be large, every
off-diagonal $S_{ij}$ ($i \neq j$) should be small. The loss is a classification
over the matrix that makes the diagonal win in both directions.

$$
% caption: The CLIP batch similarity matrix. Row $i$ is image $i$ scored against
% every caption; the loss makes the diagonal (the true pair) the row and column max.
\begin{tikzpicture}[>=stealth, font=\scriptsize, x=0.74cm, y=0.74cm]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \def\n{5}
  % off-diagonal light cells, diagonal accent cells
  \foreach \r in {1,...,5}{ \foreach \c in {1,...,5}{
    \pgfmathparse{\r==\c ? 1 : 0}
    \ifnum\pgfmathresult=1
      \fill[acc!18] (\c-1,\n-\r) rectangle (\c,\n-\r+1);
    \else
      \fill[black] (\c-1,\n-\r) rectangle (\c,\n-\r+1);
    \fi
  }}
  \draw[black, thick] (0,0) grid (\n,\n);
  % diagonal outline to mark positives
  \foreach \k in {1,...,5}{
    \draw[acc, thick] (\k-1,\n-\k) rectangle (\k,\n-\k+1);
  }
  % row labels (images) on the left
  \foreach \r/\lab in {1/I_1,2/I_2,3/I_3,4/I_4,5/I_5}{
    \node[acc, anchor=east] at (-0.15,\n-\r+0.5) {$\lab$};
  }
  % column labels (texts) on top
  \foreach \c/\lab in {1/T_1,2/T_2,3/T_3,4/T_4,5/T_5}{
    \node[green, anchor=south] at (\c-0.5,\n+0.1) {$\lab$};
  }
  \node[acc, anchor=east] at (-0.15,\n+0.45) {images};
  \node[green, anchor=south] at (\n+1.4,\n+0.1) {captions};
  \node[acc, anchor=west] at (\n+0.2,\n-1+0.5) {true pair};
  \draw[acc, thick, ->] (\n+0.15,\n-1+0.5) -- (\n-0.05,\n-1+0.5);
\end{tikzpicture}
$$

### Deriving the symmetric InfoNCE loss

Read each row of $S$ as logits over the $N$ captions for one fixed image. The
probability the model assigns to caption $j$ being the match for image $i$ is a
softmax over the row,

$$
p_{i \to j}^{\text{img}}
= \frac{\exp\!\parens{u_i^{T} v_j / \tau}}{\sum_{k=1}^{N} \exp\!\parens{u_i^{T} v_k / \tau}}.
$$

The true caption for image $i$ is $j = i$, so the image-to-text loss is the
cross-entropy that the row softmax places all its mass on the diagonal,

$$
\mathcal{L}_{\text{img} \to \text{txt}}
= -\frac{1}{N} \sum_{i=1}^{N} \log p_{i \to i}^{\text{img}}
= -\frac{1}{N} \sum_{i=1}^{N}
\log \frac{\exp\!\parens{u_i^{T} v_i / \tau}}{\sum_{k=1}^{N} \exp\!\parens{u_i^{T} v_k / \tau}}.
$$

This loss is **InfoNCE**: one positive ($v_i$) against $N - 1$ in-batch
negatives, with $\tau$ the temperature.[^oord-infonce] The similarity matrix is
not symmetric in its _use_: reading by columns instead gives the text-to-image
direction, classifying each caption among the $N$ images,

$$
\mathcal{L}_{\text{txt} \to \text{img}}
= -\frac{1}{N} \sum_{j=1}^{N}
\log \frac{\exp\!\parens{u_j^{T} v_j / \tau}}{\sum_{k=1}^{N} \exp\!\parens{u_k^{T} v_j / \tau}}.
$$

CLIP averages the two so neither modality is privileged. The full objective is
the **symmetric contrastive loss**.

> **Definition (Symmetric InfoNCE).** Over a batch of $N$ pairs with similarity
> matrix $S_{ij} = u_i^{T} v_j / \tau$, the CLIP loss is the mean of the
> row-wise and column-wise cross-entropies against the diagonal labels,
> $$
> \mathcal{L}_{\text{CLIP}}
> = \tfrac{1}{2}\parens{\mathcal{L}_{\text{img} \to \text{txt}} + \mathcal{L}_{\text{txt} \to \text{img}}}.
> $$
> Minimizing it maximizes diagonal similarity while suppressing every
> off-diagonal entry; the temperature $\tau$ sets how sharply.

The temperature is not a fixed hyperparameter. CLIP learns $\log(1/\tau)$ and
clips it to avoid collapse: small $\tau$ sharpens the softmax (hard separation,
unstable gradients), large $\tau$ flattens it (soft separation, weak signal). The
gradient gives the intuition. Writing $\sigma$ for the row softmax, the
derivative of $\mathcal{L}_{\text{img} \to \text{txt}}$ with respect to the
logit $S_{ij}$ is

$$
\frac{\partial \mathcal{L}_{\text{img} \to \text{txt}}}{\partial S_{ij}}
= \frac{1}{N}\parens{\sigma_{ij} - \mathbb{1}[i = j]},
$$

so each step pushes the diagonal logit up (target $1$) and every off-diagonal
logit down (target $0$), with force proportional to the current softmax mass on
the wrong caption: large negatives are corrected hardest. The full dual-encoder
training step is one algorithm block.

$$
% caption: The CLIP dual encoder. A ViT embeds the image, a Transformer embeds the
% caption; both project to a shared space where the symmetric loss is computed.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, black, minimum width=20mm, minimum height=9mm, align=center, font=\scriptsize},
  enc/.style={draw=acc, thick, minimum width=22mm, minimum height=11mm, align=center, font=\scriptsize, fill=acc!12},
  tenc/.style={draw=green, thick, minimum width=22mm, minimum height=11mm, align=center, font=\scriptsize, fill=green!12}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % image branch (top)
  \node[box] (img) at (0,1.6) {image};
  \node[enc] (ie) at (3.0,1.6) {image encoder\\(ViT)};
  \node[acc, draw=acc, thick, minimum width=9mm, minimum height=9mm, fill=acc!15] (u) at (6.0,1.6) {$u_i$};
  \draw[->, acc, thick] (img) -- (ie);
  \draw[->, acc, thick] (ie) -- (u);
  % text branch (bottom)
  \node[box] (txt) at (0,-1.6) {caption};
  \node[tenc] (te) at (3.0,-1.6) {text encoder};
  \node[green, draw=green, thick, minimum width=9mm, minimum height=9mm, fill=green!15] (v) at (6.0,-1.6) {$v_i$};
  \draw[->, green, thick] (txt) -- (te);
  \draw[->, green, thick] (te) -- (v);
  % shared space + loss
  \node[box, draw=black, thick, minimum width=16mm, minimum height=14mm] (loss) at (8.7,0) {shared\\space\\loss};
  \draw[->, acc, thick] (u) -- (loss);
  \draw[->, green, thick] (v) -- (loss);
  \node[black, anchor=west, font=\scriptsize] at (9.6,0) {$u^{T} v$};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{ClipLoss}(\braces{x_i}, \braces{y_i}, \tau)$ — symmetric contrastive loss over a batch
$U \gets [\,f_{\text{img}}(x_1), \dots, f_{\text{img}}(x_N)\,]$ // image embeddings, rows
$V \gets [\,f_{\text{txt}}(y_1), \dots, f_{\text{txt}}(y_N)\,]$ // text embeddings, rows
$\ell_2$-normalize every row of $U$ and $V$
$S \gets U V^{T} / \tau$ // $N \times N$ similarity logits
$\text{labels} \gets (1, 2, \dots, N)$ // true match is the diagonal
$\mathcal{L}_i \gets \textsc{CrossEntropy}(\text{rows of } S, \text{labels})$ // image to text
$\mathcal{L}_t \gets \textsc{CrossEntropy}(\text{columns of } S, \text{labels})$ // text to image
return $\tfrac{1}{2}(\mathcal{L}_i + \mathcal{L}_t)$
```

CLIP trains on $400$M image-text pairs scraped from the web; the batch is the
source of negatives, so very large batches (CLIP used $32{,}768$) sharpen the
signal because each positive competes against tens of thousands of distractors.

### A worked step

For example, take $N = 3$ pairs, $\tau = 0.1$, and suppose the three
normalized image vectors already sit near their captions so the cosine
similarities $u_i^{T} v_j$ come out as

$$
U V^{T} =
\begin{pmatrix}
0.8 & 0.1 & 0.2 \\
0.0 & 0.7 & 0.1 \\
0.3 & 0.2 & 0.9
\end{pmatrix},
\qquad
S = \frac{U V^{T}}{\tau} =
\begin{pmatrix}
8 & 1 & 2 \\
0 & 7 & 1 \\
3 & 2 & 9
\end{pmatrix}.
$$

Row $1$ is the logit vector $(8, 1, 2)$ for image $1$ against the three captions.
Its softmax puts

$$
p_{1 \to 1} = \frac{e^{8}}{e^{8} + e^{1} + e^{2}}
= \frac{2981}{2981 + 2.7 + 7.4} \approx 0.9966,
$$

so the image-to-text loss on this row is $-\log 0.9966 \approx 0.0034$. Divide
$\tau$ by ten (sharper) and the same row gives $p_{1\to 1}\approx 1 - 10^{-30}$,
essentially zero loss but a knife-edge gradient; multiply $\tau$ by ten (flatter)
and $p_{1 \to 1}$ falls toward $\tfrac13$, so the loss rises and the encoders get a
larger, softer push. That single dial trades gradient sharpness against gradient
signal, which is why CLIP learns it rather than fixing it.

The tensor shapes for one training step, with embedding dimension $d$ and batch
$N$, are

$$
\underbrace{U}_{N \times d}, \;\;
\underbrace{V}_{N \times d}
\;\xrightarrow{\;U V^{T}/\tau\;}\;
\underbrace{S}_{N \times N}
\;\xrightarrow{\text{row/col softmax + CE}}\;
\underbrace{\mathcal{L}_{\text{CLIP}}}_{\text{scalar}},
$$

so the only $N \times N$ object is the similarity matrix; everything else is a
thin $N \times d$ activation. The memory cost of the negatives is the $N^2$
matrix, cheap next to the two encoders, which is what makes tens-of-thousands
batches affordable.

## Zero-shot transfer

A CLIP model never sees class labels, only captions, yet it classifies images
without any task-specific training. The method is to turn each class name into a
caption and pick the caption the image embedding is closest to.

> **Definition (Zero-shot classification).** Given classes
> $c_1, \dots, c_K$, build a text prompt per class, e.g.
> "a photo of a $\braces{c_k}$", embed each with the text encoder to get
> $v_1, \dots, v_K$, embed the test image to $u$, and predict
> $\hat c = \arg\max_k u^{T} v_k$. No gradient step is taken; classification is
> nearest-caption retrieval in the shared space.

The prediction is a plain softmax classifier whose weight matrix is the stack of
text embeddings. With $V_c = [\,v_1, \dots, v_K\,] \in \mathbb{R}^{K \times d}$ the
class-caption embeddings and $u$ the image embedding, the class posterior is

$$
p(c_k \mid x)
= \frac{\exp\!\parens{u^{T} v_k / \tau}}{\sum_{j=1}^{K} \exp\!\parens{u^{T} v_j / \tau}},
\qquad \hat c = \arg\max_k \; u^{T} v_k,
$$

the same row-softmax as training, only now the columns are $K$ class prompts
instead of $N$ batch captions. No parameters change; the "classifier weights"
$V_c$ are computed once by running the text encoder over the $K$ prompts.

The choice of prompt template matters, because the text encoder was trained on
natural captions, not bare nouns. "a photo of a dog" sits where dog photos sit;
the lone token "dog" does not. **Prompt ensembling** averages the (normalized)
embeddings of many templates per class, $\bar v_k = \tfrac1M \sum_{m} v_k^{(m)}$
over $M$ templates such as "a photo of a $\braces{c_k}$", "a blurry photo of a
$\braces{c_k}$", "a $\braces{c_k}$ in a video game", then renormalizes. Averaging
in embedding space cancels template-specific noise while keeping the shared class
direction, and it recovers a few points of accuracy for free.

$$
% caption: Zero-shot transfer. The image embeds once; each class name becomes a
% caption, and the prediction is the caption with maximum similarity.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, black, minimum width=30mm, minimum height=7mm, align=center, font=\scriptsize},
  hit/.style={draw=green, thick, minimum width=30mm, minimum height=7mm, align=center, font=\scriptsize, fill=green!12}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % image and its embedding
  \node[box, draw=acc, thick, fill=acc!12] (img) at (0,0.6) {test image};
  \node[acc, draw=acc, thick, minimum width=9mm, minimum height=9mm, fill=acc!15] (u) at (3.2,0.6) {$u$};
  \draw[->, acc, thick] (img) -- (u);
  % prompt templates
  \node[box] (p1) at (8.2,2.4) {"a photo of a cat"};
  \node[hit] (p2) at (8.2,0.6) {"a photo of a dog"};
  \node[box] (p3) at (8.2,-1.2) {"a photo of a car"};
  % similarity scores
  \draw[->, black, thick] (u) -- (p1) node[midway, above, font=\footnotesize] {\texttt{0.21}};
  \draw[->, green, thick] (u) -- (p2) node[midway, above, font=\footnotesize, text=green] {\texttt{0.74}};
  \draw[->, black, thick] (u) -- (p3) node[midway, below, font=\footnotesize] {\texttt{0.18}};
  \node[green, anchor=west, font=\scriptsize] at (10.0,0.6) {argmax};
\end{tikzpicture}
$$

CLIP transfers zero-shot to ImageNet at accuracy competitive with a fully
supervised ResNet-50, and is far more robust to distribution shift, because the
caption supervision is broader than a fixed label set.[^radford-clip]

### ALIGN: the noisy-scale variant

CLIP curated its pairs. **ALIGN** drops the curation: it trains the same
dual-encoder contrastive objective on over a billion image-alt-text pairs taken
raw from the web, noise and all, showing that scale of data substitutes for
quality of data.[^jia-align] The architecture differs in detail (an EfficientNet
image tower, a BERT text tower) but the loss is the same symmetric InfoNCE.

| Model | Image encoder | Text encoder | Pairs | Curation |
| --- | --- | --- | --- | --- |
| CLIP | ViT / ResNet | Transformer | $400$M | filtered |
| ALIGN | EfficientNet | BERT | $1.8$B | raw alt-text |

Together, the two show that the contrastive recipe tolerates enormous label
noise, provided the batch is large enough that the signal on the diagonal
outvotes the noise off it.


## Sharpening and scaling the contrastive recipe

CLIP fixed the recipe; the work since refined the loss and the data without touching the dual-encoder idea. Two changes are now standard.

**SigLIP replaces the softmax with a sigmoid.** CLIP's symmetric InfoNCE needs the full $N \times N$ similarity matrix, because each row's softmax normalizes over every caption in the batch, which couples the whole batch and makes very large batches a memory and communication problem. **SigLIP** swaps the softmax cross-entropy for an independent **sigmoid** loss on each pair: every image-text pair is a binary "match or not" decision, positive on the diagonal and negative off it, with no row-wise normalization.[^zhai-siglip] Because the pairs decouple, the loss shards cleanly across devices and trains well at both small and enormous batch sizes, and it tends to be more robust than the softmax at the batch sizes most teams can actually afford.

$$
% caption: CLIP versus SigLIP loss. CLIP normalizes each row over the whole batch (a
% coupled softmax over N captions); SigLIP scores each pair independently with a
% sigmoid, positive on the diagonal and negative off it, so pairs decouple across the batch.
\begin{tikzpicture}[>=stealth, font=\scriptsize, x=0.62cm, y=0.62cm]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \def\n{4}
  % --- CLIP: row-normalized softmax ---
  \begin{scope}
    \foreach \r in {1,...,4}{ \foreach \c in {1,...,4}{
      \pgfmathparse{\r==\c ? 1 : 0}
      \ifnum\pgfmathresult=1 \fill[acc!18] (\c-1,\n-\r) rectangle (\c,\n-\r+1);
      \else \fill[black] (\c-1,\n-\r) rectangle (\c,\n-\r+1); \fi
    }}
    \draw[black] (0,0) grid (\n,\n);
    \draw[acc, thick] (0,\n-1) rectangle (\n,\n);
    \node[acc, anchor=south, font=\footnotesize] at (\n/2,\n+0.2) {CLIP: softmax per row};
  \end{scope}
  % --- SigLIP: independent sigmoid per cell ---
  \begin{scope}[xshift=5.2cm]
    \foreach \r in {1,...,4}{ \foreach \c in {1,...,4}{
      \pgfmathparse{\r==\c ? 1 : 0}
      \ifnum\pgfmathresult=1 \fill[green!20] (\c-1,\n-\r) rectangle (\c,\n-\r+1);
      \else \fill[black] (\c-1,\n-\r) rectangle (\c,\n-\r+1); \fi
    }}
    \draw[black] (0,0) grid (\n,\n);
    \node[green, anchor=south, font=\footnotesize] at (\n/2,\n+0.2) {SigLIP: sigmoid per cell};
  \end{scope}
\end{tikzpicture}
$$

**Contrastive encoders became the vision front-end of language models.** The most consequential downstream use of CLIP is not zero-shot classification but the fact that a frozen CLIP vision tower is the standard image encoder feeding a language model — the bridge that part two builds. The contrastive objective produces exactly the property a language model needs from vision: patch features that already live in a space aligned to text, so a thin projector can map them into token space. The refinements above (a better loss, cleaner or larger data) propagate straight through: a stronger contrastive encoder makes a stronger vision-language model, which is why the contrastive route and the fusion route are two halves of one pipeline rather than competing designs.

In short, the dual-encoder contrastive model is the durable core of multimodal learning. Its loss got cheaper to scale (SigLIP), its data got larger (ALIGN and successors), and its output became the input to the fusion models of part two. The picture-and-caption-as-neighbors idea outlasted every specific architecture built on top of it.

## Takeaways

- A **multimodal model** maps images, text, and audio into one space (contrastive)
  or interleaves them inside a network (fusion); the residual **modality gap**
  means matched pairs are pulled together without the two clouds merging.
- The **Vision Transformer** turns an image into tokens: cut it into $P \times P$
  patches ($196$ for a $224^2$ image at $P=16$), linearly embed each to a
  $d$-vector, prepend a **[CLS] token**, add learned **position embeddings**, and
  run a standard Transformer encoder; the CLS output is the image vector.
- **CLIP** trains two encoders so that, over a batch, the $N \times N$ similarity
  matrix $S_{ij} = u_i^{T} v_j / \tau$ has its true pairs on the diagonal; the
  **symmetric InfoNCE** loss is the mean of the row and column cross-entropies, one
  positive against $N-1$ in-batch negatives, with a learned temperature $\tau$.
- **Zero-shot transfer** classifies by embedding "a photo of a $\braces{class}$"
  prompts and taking $\arg\max_k u^{T} v_k$; no gradient step, just nearest-caption
  retrieval. **ALIGN** is the same recipe at raw web scale.
- **Since CLIP:** **SigLIP** swaps the coupled softmax for an independent sigmoid
  loss that shards across devices and trains at any batch size, and the contrastive
  vision tower is what feeds the vision-language models of part two.

[^liang-gap]: **Liang et al.**, _Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning_, NeurIPS 2022 — shows image and text embeddings occupy separate cones on the sphere, an offset set at initialization and only narrowed by contrastive training.
[^dosovitskiy-vit]: **Dosovitskiy et al.**, _An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale_ (ViT), ICLR 2021 — patch embedding, a prepended CLS token, learned position embeddings, and a standard Transformer encoder over the patch sequence.
[^radford-clip]: **Radford et al.**, _Learning Transferable Visual Models From Natural Language Supervision_ (CLIP), ICML 2021 — the dual-encoder contrastive recipe on $400$M pairs, learned temperature, and zero-shot transfer competitive with supervised baselines.
[^oord-infonce]: **van den Oord, Li & Vinyals**, _Representation Learning with Contrastive Predictive Coding_, 2018 — introduces the InfoNCE loss, one positive against $N-1$ negatives, the contrastive objective CLIP applies across modalities.
[^jia-align]: **Jia et al.**, _Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision_ (ALIGN), ICML 2021 — trains the same contrastive objective on over a billion uncurated alt-text pairs, trading data quality for data scale.
[^zhai-siglip]: **Zhai et al.**, _Sigmoid Loss for Language Image Pre-Training_ (SigLIP), ICCV 2023 — replaces CLIP's batch-coupled softmax with an independent per-pair sigmoid loss that decouples across the batch and trains robustly at any batch size.