Multimodal Contrastive Learning

Large language models read one modality: text. A multimodal model maps inputs from several modalities, images, text, audio, into a single representation where comparison across modalities is a dot product, so that a photograph and the sentence describing it become neighbors. Everything in this lesson follows from one design problem: putting a $224\times 224$ image and a string of tokens into the same space, and then letting them interact.

The shared embedding space and the modality gap

The cleanest multimodal design gives each modality its own encoder and forces their outputs into one space. An image $x$ and a text $y$ become unit vectors $f_{\text{img}}(x),f_{\text{txt}}(y)\in {\mathbb{R}}^d$, and their semantic agreement is the cosine similarity $f_{\text{img}}(x{)}^T{f}_{\text{txt}}(y)$.

The alignment is imperfect. Even after contrastive training the two modalities occupy separate cones: image embeddings cluster on one side of the sphere and text embeddings on another, with a measurable angular separation between the two clouds. This is the modality gap, a consequence of separate encoders and random initialization that contrastive training narrows but never fully closes.¹

The image encoder: a Vision Transformer

Before two modalities can share a space, the image needs an encoder that produces a single vector the way a text Transformer produces one from a sentence. The Vision Transformer (ViT) does this by cutting the image into fixed patches, treating each patch as a token, and running the ordinary Transformer encoder over that sequence.² The only new step is turning a grid of pixels into a sequence of embeddings.

Take the standard configuration: a $224\times 224$ RGB image, patch size $P=16$. The grid is $14\times 14$, so $N_p=196$ patches, each flattened to $16^2\cdot 3=768$ pixel values, then projected to $d=768$. The linear projection $E$ is literally a $Conv2d$ with kernel and stride both $16$: one convolution stride per patch, output channels $=d$. The shapes flow as

Two additions finish the input. A learned CLS token$x_{\text{cls}}\in {\mathbb{R}}^d$ is prepended to the sequence; after the encoder its output row is the image's single summary vector, the analogue of a sentence embedding. And because attention is permutation-invariant, a learned position embedding $E_{\text{pos}}\in {\mathbb{R}}^{(N_p+1)\times d}$ is added so the model can tell a top-left patch from a bottom-right one. The encoder input is

A stack of $L$ standard Transformer encoder blocks (multi-head self-attention, then an MLP, each with a residual and LayerNorm) maps $Z_0$ to $Z_L$ of the same shape. The image representation is the CLS output row $z_{\text{cls}}=(Z_L{)}_0\in {\mathbb{R}}^d$, which a final linear head projects to the shared space. Nothing about attention changed; only the tokenizer did.

The patch count sets the compute: self-attention is $O(N_p^2)$, so halving the patch size ($P=16\to 8$) quadruples the token count ($196\to 784$) and raises attention cost roughly $16\times$. That is the accuracy-versus-cost dial a ViT exposes, and it is why CLIP ships several patch sizes.

Contrastive vision-language pretraining

The dominant way to learn a shared space is contrastive language-image pretraining (CLIP).³ Two encoders, a Vision Transformer for the image and a Transformer for the text, map a large batch of paired (image, caption) examples into ${\mathbb{R}}^d$, and a contrastive loss pulls each image toward its own caption and pushes it away from every other caption in the batch.

The batch similarity matrix

Embed the batch and ${\ell }_2$-normalize: let $u_i=f_{\text{img}}(x_i)$ and $v_j=f_{\text{txt}}(y_j)$, each on the unit sphere. The pairwise cosine similarities form an $N\times N$ matrix $S$, scaled by a learned temperature $\tau$,

The correct matches lie on the diagonal: $S_{ii}$ should be large, every off-diagonal $S_{ij}$ ($i\ne j$) should be small. The loss is a classification over the matrix that makes the diagonal win in both directions.

Deriving the symmetric InfoNCE loss

Read each row of $S$ as logits over the $N$ captions for one fixed image. The probability the model assigns to caption $j$ being the match for image $i$ is a softmax over the row,

The true caption for image $i$ is $j=i$, so the image-to-text loss is the cross-entropy that the row softmax places all its mass on the diagonal,

This loss is InfoNCE: one positive ($v_i$) against $N-1$ in-batch negatives, with $\tau$ the temperature.⁴ The similarity matrix is not symmetric in its use: reading by columns instead gives the text-to-image direction, classifying each caption among the $N$ images,

CLIP averages the two so neither modality is privileged. The full objective is the symmetric contrastive loss.

The temperature is not a fixed hyperparameter. CLIP learns $\log (1/\tau )$ and clips it to avoid collapse: small $\tau$ sharpens the softmax (hard separation, unstable gradients), large $\tau$ flattens it (soft separation, weak signal). The gradient gives the intuition. Writing $\sigma$ for the row softmax, the derivative of ${\mathcal{L}}_{\text{img}\to \text{txt}}$ with respect to the logit $S_{ij}$ is

so each step pushes the diagonal logit up (target $1$) and every off-diagonal logit down (target $0$), with force proportional to the current softmax mass on the wrong caption: large negatives are corrected hardest. The full dual-encoder training step is one algorithm block.

CLIP trains on $400$M image-text pairs scraped from the web; the batch is the source of negatives, so very large batches (CLIP used $32,768$) sharpen the signal because each positive competes against tens of thousands of distractors.

A worked step

For example, take $N=3$ pairs, $\tau =0.1$, and suppose the three normalized image vectors already sit near their captions so the cosine similarities $u_i^T{v}_j$ come out as

Row $1$ is the logit vector $(8,1,2)$ for image $1$ against the three captions. Its softmax puts

so the image-to-text loss on this row is $-\log 0.9966\approx 0.0034$. Divide $\tau$ by ten (sharper) and the same row gives $p_{1\to 1}\approx 1-10^{-30}$, essentially zero loss but a knife-edge gradient; multiply $\tau$ by ten (flatter) and $p_{1\to 1}$ falls toward $\frac{1}{3}$, so the loss rises and the encoders get a larger, softer push. That single dial trades gradient sharpness against gradient signal, which is why CLIP learns it rather than fixing it.

The tensor shapes for one training step, with embedding dimension $d$ and batch $N$, are

so the only $N\times N$ object is the similarity matrix; everything else is a thin $N\times d$ activation. The memory cost of the negatives is the $N^2$ matrix, cheap next to the two encoders, which is what makes tens-of-thousands batches affordable.

Zero-shot transfer

A CLIP model never sees class labels, only captions, yet it classifies images without any task-specific training. The method is to turn each class name into a caption and pick the caption the image embedding is closest to.

The prediction is a plain softmax classifier whose weight matrix is the stack of text embeddings. With $V_c=[v_1,\dots ,v_K]\in {\mathbb{R}}^{K\times d}$ the class-caption embeddings and $u$ the image embedding, the class posterior is

the same row-softmax as training, only now the columns are $K$ class prompts instead of $N$ batch captions. No parameters change; the classifier weights$V_c$ are computed once by running the text encoder over the $K$ prompts.

The choice of prompt template matters, because the text encoder was trained on natural captions, not bare nouns. a photo of a dog sits where dog photos sit; the lone token dog does not. Prompt ensembling averages the (normalized) embeddings of many templates per class, ${\bar{v}}_k=\frac{1}{M}{\sum }_m{v}_k^{(m)}$ over $M$ templates such as a photo of a $\left\{c_k\right\}$, a blurry photo of a $\left\{c_k\right\}$, a $\left\{c_k\right\}$ in a video game, then renormalizes. Averaging in embedding space cancels template-specific noise while keeping the shared class direction, and it recovers a few points of accuracy for free.

CLIP transfers zero-shot to ImageNet at accuracy competitive with a fully supervised ResNet-50, and is far more robust to distribution shift, because the caption supervision is broader than a fixed label set.³

ALIGN: the noisy-scale variant

CLIP curated its pairs. ALIGN drops the curation: it trains the same dual-encoder contrastive objective on over a billion image-alt-text pairs taken raw from the web, noise and all, showing that scale of data substitutes for quality of data.⁵ The architecture differs in detail (an EfficientNet image tower, a BERT text tower) but the loss is the same symmetric InfoNCE.

Together, the two show that the contrastive recipe tolerates enormous label noise, provided the batch is large enough that the signal on the diagonal outvotes the noise off it.

Sharpening and scaling the contrastive recipe

CLIP fixed the recipe; the work since refined the loss and the data without touching the dual-encoder idea. Two changes are now standard.

SigLIP replaces the softmax with a sigmoid. CLIP's symmetric InfoNCE needs the full $N\times N$ similarity matrix, because each row's softmax normalizes over every caption in the batch, which couples the whole batch and makes very large batches a memory and communication problem. SigLIP swaps the softmax cross-entropy for an independent sigmoid loss on each pair: every image-text pair is a binary match or not decision, positive on the diagonal and negative off it, with no row-wise normalization.⁶ Because the pairs decouple, the loss shards cleanly across devices and trains well at both small and enormous batch sizes, and it tends to be more robust than the softmax at the batch sizes most teams can actually afford.

Contrastive encoders became the vision front-end of language models. The most consequential downstream use of CLIP is not zero-shot classification but the fact that a frozen CLIP vision tower is the standard image encoder feeding a language model — the bridge that part two builds. The contrastive objective produces exactly the property a language model needs from vision: patch features that already live in a space aligned to text, so a thin projector can map them into token space. The refinements above (a better loss, cleaner or larger data) propagate straight through: a stronger contrastive encoder makes a stronger vision-language model, which is why the contrastive route and the fusion route are two halves of one pipeline rather than competing designs.

In short, the dual-encoder contrastive model is the durable core of multimodal learning. Its loss got cheaper to scale (SigLIP), its data got larger (ALIGN and successors), and its output became the input to the fusion models of part two. The picture-and-caption-as-neighbors idea outlasted every specific architecture built on top of it.

Takeaways

• A multimodal model maps images, text, and audio into one space (contrastive) or interleaves them inside a network (fusion); the residual modality gap means matched pairs are pulled together without the two clouds merging.
• The Vision Transformer turns an image into tokens: cut it into $P\times P$ patches ($196$ for a $224^2$ image at $P=16$), linearly embed each to a $d$-vector, prepend a CLS token, add learned position embeddings, and run a standard Transformer encoder; the CLS output is the image vector.
• CLIP trains two encoders so that, over a batch, the $N\times N$ similarity matrix $S_{ij}=u_i^T{v}_j/\tau$ has its true pairs on the diagonal; the symmetric InfoNCE loss is the mean of the row and column cross-entropies, one positive against $N-1$ in-batch negatives, with a learned temperature $\tau$.
• Zero-shot transfer classifies by embedding a photo of a $\left\{class\right\}$ prompts and taking $\arg {\max }_k{u}^T{v}_k$; no gradient step, just nearest-caption retrieval. ALIGN is the same recipe at raw web scale.
• Since CLIP: SigLIP swaps the coupled softmax for an independent sigmoid loss that shards across devices and trains at any batch size, and the contrastive vision tower is what feeds the vision-language models of part two.