--- title: "Fusion and Vision-Language Models" module: Large Models & Agents moduleNumber: 10 lessonNumber: 11 order: 1011 summary: > A contrastive model compares modalities but never lets one read another. This second part builds the fusion route: early, late, and cross-attention fusion, then the three designs that connect a frozen vision encoder to a frozen language model — Flamingo's zero-initialized gated cross-attention, BLIP-2's Q-Former, and LLaVA's linear projector. We work the token-budget arithmetic that separates them, name the object-hallucination and fine-detail failure modes, cover the contrastive-then- instruction-tune recipe and its retrieval/captioning/VQA benchmarks, and close on natively multimodal models. topics: [Large Models & Agents] sources: - book: Chollet ref: "Ch. 11 — combining modalities; multimodal architectures" --- This builds on [Multimodal Contrastive Learning](/deep-learning/large-models-and-agents/multimodal-models), which put images and text in one space with a dual encoder and a symmetric contrastive loss, giving a picture and its caption a high dot product. A contrastive model can _compare_ modalities but cannot let one _read_ another: nothing in CLIP lets a text token pull information from a specific image region. For captioning and visual question answering the modalities have to fuse inside the network. This lesson builds that fusion, then uses it to connect a frozen language model to vision. ## Fusion strategies Contrastive models compare modalities; they do not let one modality _read_ another. For tasks that need interaction, captioning, visual question answering, the modalities must **fuse** inside the network. Where the fusion happens defines the strategy. > **Definition (Early vs late fusion).** **Early fusion** concatenates raw or > shallowly-encoded features from each modality and processes the joint > representation with one network from the start. **Late fusion** runs each > modality through its own deep encoder and combines only the final outputs (by > concatenation, sum, or a contrastive score). Early fusion models fine cross-modal > interaction; late fusion is modular and cheap but cannot mix modalities until the > end.[^chollet-multimodal] The middle ground, and the one that powers modern vision-language models, is **cross-attention**: keep separate encoders but let one modality's queries attend to the other modality's keys and values. > **Definition (Cross-attention fusion).** Given a query stream > $Q = X_a W_Q$ from modality $a$ and key/value streams > $K = X_b W_K$, $V = X_b W_V$ from modality $b$, the fused representation is > $$ > \mathrm{CrossAttn}(X_a, X_b) > = \mathrm{softmax}\!\parens{\frac{Q K^{T}}{\sqrt{d_k}}} V, > $$ > identical in form to [self-attention](/deep-learning/architectures/the-transformer-architecture) > but with queries and keys drawn from different modalities. It lets text tokens > pull information from image patches (or the reverse) at every layer. $$ % caption: Cross-attention fusion. Text tokens form queries that attend over image % patch keys and values, so each text position pulls in the relevant visual content. \begin{tikzpicture}[>=stealth, font=\footnotesize, patch/.style={draw=acc, thick, minimum size=7mm, inner sep=0pt, fill=acc!12, font=\scriptsize}, tok/.style={draw=green, thick, minimum width=13mm, minimum height=7mm, inner sep=1pt, fill=green!12, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % image patch grid (keys / values) \node[acc, anchor=south] at (1.05,2.5) {image patches (K, V)}; \foreach \c in {0,1,2}{ \foreach \r in {0,1}{ \node[patch] at (\c*0.75, \r*0.75) {}; }} % the patches as a key block boundary \draw[acc, thick] (-0.45,-0.45) rectangle (2.55,1.95); % text tokens (queries) \node[green, anchor=south] at (6.6,2.5) {\texttt{text tokens (Q)}}; \node[tok] (q1) at (6.6,1.5) {what}; \node[tok] (q2) at (6.6,0.6) {color}; \node[tok] (q3) at (6.6,-0.3) {is}; % cross-attention edges from one query into the patch block \draw[->, black, thick] (3.0,0.95) -- (q2.west); \draw[->, black, thick] (2.0,1.5) to[bend left=12] (q2.north west); \draw[->, black, thick] (1.0,0.0) to[bend right=14] (q3.west); \node[black, anchor=south, font=\footnotesize] at (4.5,1.75) {\texttt{cross-attention}}; \end{tikzpicture} $$ | Strategy | Where modalities meet | Cross-modal interaction | Cost | | --- | --- | --- | --- | | Late (CLIP) | only at the final similarity | none inside the net | cheap, modular | | Early | concatenate at the input | full, from layer $1$ | expensive | | Cross-attention | dedicated attention layers | rich, controllable | moderate | ## Vision-language models: feeding images into an LLM The dominant design does not train a multimodal model from scratch; it _connects_ a strong frozen vision encoder to a strong frozen LLM with a small trainable bridge, so the language model gains visual input without losing its language ability. Three designs span the space. ### Flamingo: perceiver resampler and gated cross-attention **Flamingo** keeps a frozen vision encoder and a frozen LLM and inserts two trained pieces.[^alayrac-flamingo] A **perceiver resampler** maps a variable number of image features to a fixed small set of latent tokens, and **gated cross-attention** layers interleaved into the frozen LLM let text tokens attend to those visual latents. > **Definition (Perceiver resampler).** A small attention module with a fixed set > of $m$ learned latent queries $L \in \mathbb{R}^{m \times d}$ that cross-attend > to the (possibly many) image features $X$, producing exactly $m$ output tokens > regardless of input resolution or frame count, > $L' = \mathrm{CrossAttn}(L, X)$. It bounds the number of visual tokens the LLM > must process. The cross-attention is **gated** with a $\tanh$ gate initialized at zero, so at the start of training the visual stream contributes nothing and the model is exactly the original LLM; the gate opens as training proceeds. This is what lets a frozen language model absorb a new modality without destabilizing. $$ % caption: Flamingo gated cross-attention. A $\tanh$ gate starts at zero, so the % block begins as the identity and the frozen LLM is unchanged until the gate opens. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, black, minimum width=22mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % main residual stream (text) bottom to top \node[box, fill=black!8] (in) at (0,0) {LLM hidden}; \node[box, draw=acc, thick, fill=acc!12] (xa) at (0,1.6) {cross-attn (visual)}; \node[circle, draw=green, thick, minimum size=7mm, fill=green!12] (g) at (0,3.0) {gate}; \node[box, fill=black!8] (out) at (0,4.4) {to next layer}; \draw[->, black, thick] (in) -- (xa); \draw[->, acc, thick] (xa) -- (g); \draw[->, green, thick] (g) -- (out); % residual skip around the gate \draw[->, black, thick] (in.east) -- ++(2.4,0) |- (out.east); \node[black, anchor=west, font=\scriptsize] at (2.5,2.2) {residual skip}; % gate label \node[green, anchor=west, font=\scriptsize] at (0.5,3.0) {$\tanh(g)$, init $0$}; % visual latents feeding cross-attn \node[box, draw=acc, thick, fill=acc!15] (lat) at (-3.4,1.6) {\texttt{visual latents}}; \draw[->, acc, thick] (lat) -- (xa); \end{tikzpicture} $$ ### BLIP-2: the Q-Former **BLIP-2** bridges a frozen image encoder and a frozen LLM with a lightweight **querying Transformer (Q-Former)**.[^li-blip2] A small set of learned query tokens extracts a fixed number of visual features through cross-attention, and a linear layer projects them into the LLM's input space as "soft visual tokens". The Q-Former is the only trained module, and it is trained in two stages, inheriting the vision-language objectives of the earlier **BLIP**.[^li-blip] > **Definition (Q-Former).** A small Transformer holding $k$ learnable query > embeddings that cross-attend to a frozen image encoder's patch features and > self-attend among themselves, outputting $k$ vectors that summarize the image. > A linear projection maps these into the frozen LLM's embedding space, where they > are prepended to the text tokens. ### LLaVA: a linear projector **LLaVA** is the minimal design.[^liu-llava] A frozen [ViT](/deep-learning/architectures/the-transformer-architecture) (from CLIP) produces patch features; a single trainable **linear (or two-layer MLP) projector** maps each patch feature into the frozen LLM's token-embedding space; the projected patches are prepended to the text and the LLM does the rest. There is no resampler and no Q-Former: the projector is the entire bridge, and it makes **visual instruction tuning** cheap enough to run on modest hardware. $$ % caption: A vision-language model: a frozen vision encoder produces patch % features, a trained projector maps them to token space, and a frozen LLM reads % them alongside text. \begin{tikzpicture}[>=stealth, font=\footnotesize, frozen/.style={draw=black, thick, minimum width=22mm, minimum height=11mm, align=center, font=\scriptsize, fill=black!8}, proj/.style={draw=acc, thick, minimum width=20mm, minimum height=11mm, align=center, font=\scriptsize, fill=acc!12}, io/.style={draw, black, minimum width=18mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % image path \node[io] (img) at (0,1.2) {image}; \node[frozen] (ve) at (2.8,1.2) {\texttt{vision encoder}\\(frozen)}; \node[proj] (pr) at (6.0,1.2) {\texttt{projector}\\(trained)}; \draw[->, black, thick] (img) -- (ve); \draw[->, acc, thick] (ve) -- (pr); % text path \node[io] (txt) at (6.0,-1.4) {\texttt{text tokens}}; % LLM \node[frozen, minimum height=16mm, minimum width=24mm] (llm) at (9.4,0) {LLM\\(frozen)}; \draw[->, acc, thick] (pr.east) -- ++(0.6,0) |- (llm.170); \draw[->, green, thick] (txt.east) -- ++(0.6,0) |- (llm.190); % output \node[io, draw=green, thick, fill=green!12] (out) at (12.6,0) {answer}; \draw[->, green, thick] (llm) -- (out); % labels for what is trained \node[acc, anchor=north, font=\footnotesize] at (6.0,0.5) {\texttt{soft visual tokens}}; \end{tikzpicture} $$ The three differ mainly in the bridge between vision and language, and in how many visual tokens reach the LLM. | Model | How vision meets language | Bridge module | What is trained | Frozen | | --- | --- | --- | --- | --- | | CLIP | shared space, no fusion | none (dual encoder) | both encoders | nothing | | Flamingo | gated cross-attention in the LLM | perceiver resampler $+$ gates | resampler, gates | vision, LLM | | BLIP-2 | soft tokens from a Q-Former | Q-Former $+$ linear | Q-Former, projection | vision, LLM | | LLaVA | soft tokens from a projector | linear / MLP projector | projector (then LLM) | vision (LLM partly) | The token budget is the practical difference. LLaVA's projector emits one soft token per patch, so a $336 \times 336$ image at patch $14$ gives $576$ visual tokens prepended to the prompt; the LLM pays $O((576 + T)^2)$ attention for a text length $T$. Flamingo's resampler and BLIP-2's Q-Former instead fix the count at $m \approx 32$-$64$ latents regardless of resolution, trading some spatial detail for a constant, small prefix. That is the reason to prefer a resampler at high resolution or over video: it caps the sequence the LLM must read. Two failure modes recur across these designs, and both trace back to the bridge. **Object hallucination**, the model confidently naming things not in the image, worsens when too few visual tokens or a weak projector let the language prior override the visual evidence; the LLM falls back on what captions usually say. And **fine-detail blindness**, missing small text or counts, follows directly from patch size and token budget: a resampler that compresses $576$ patches to $32$ latents discards exactly the high-frequency detail an OCR or counting question needs. Raising input resolution or the visual-token count mitigates both, at quadratic attention cost. ## Training recipe and evaluation The modern pipeline has two phases, and they map onto the two halves of this lesson: align the spaces, then teach the model to follow instructions. > **Definition (Two-stage multimodal training).** **Stage 1, contrastive or > feature-alignment pretraining:** train the bridge (or both encoders) on > large-scale image-text pairs so visual features land in a space the language > model can read. **Stage 2, instruction tuning:** fine-tune on curated > (image, instruction, response) triples so the model answers questions, follows > directions, and describes images in the assistant format. LLaVA's stage 2 is **visual instruction tuning**: it bootstraps the triples by prompting a text-only LLM to generate questions and answers about images from their captions and bounding boxes, then trains on the synthetic conversations. This is instruction tuning lifted from [language alignment](/deep-learning/large-models-and-agents/large-language-models) into the visual setting. Three benchmark families score the result, and each isolates a different ability. > **Definition (Multimodal evaluation tasks).** > - **Retrieval:** rank captions for an image (or images for a caption) by > similarity; report Recall@$K$. This scores the _alignment_ a contrastive model > learns. > - **Captioning:** generate a description of an image; score against references > with CIDEr or BLEU. This scores _generation grounded in vision_. > - **Visual question answering (VQA):** answer a natural-language question about > an image; report accuracy. This scores _reasoning over fused modalities_. | Task | Input | Output | Metric | Tests | | --- | --- | --- | --- | --- | | Image-text retrieval | image or caption | ranked list | Recall@$K$ | shared-space alignment | | Captioning | image | sentence | CIDEr, BLEU | grounded generation | | VQA | image $+$ question | answer | accuracy | cross-modal reasoning | Retrieval rewards the contrastive models (CLIP, ALIGN) because their whole training objective _is_ retrieval; captioning and VQA reward the fusion models (Flamingo, BLIP-2, LLaVA) because answering requires the language model to read the image, not merely score it. ## Natively multimodal models Flamingo, BLIP-2, and LLaVA all _bolt_ vision onto a language model trained on text: a frozen LLM gains sight through a trained bridge. The frontier since has moved toward models that are multimodal from the start, and two ideas reframe the bridge designs above. **Any-to-any token models.** If every modality can be turned into discrete tokens — text by a tokenizer, images by a vision quantizer, audio by a codec — then a single decoder-only Transformer can read and write all of them in one sequence, with no separate encoders or projectors.[^team-chameleon] The bridge disappears because there is nothing to bridge: an image is just more tokens in the same stream. This is the multimodal endpoint of the same "turn it into tokens" move that recurred in the speech and language lessons, and it lets one model both answer questions about an image and generate one, in a single autoregressive pass. **Native mixed-modal pretraining.** Rather than freeze a text LLM and attach vision, a natively multimodal model interleaves image and text data throughout pretraining, so the two modalities are represented jointly from the first step and the "modality gap" of part one never opens as widely. The token-budget arithmetic of this lesson still governs the cost — images are expensive because they are many tokens, and high resolution or video multiplies the count — so the resampler-versus-projector trade (a small fixed prefix versus one token per patch) survives even when there is no frozen LLM to protect. In short, the bridge modules — resampler, Q-Former, projector — were a transitional design for the era when a strong LLM existed and a strong vision encoder existed and the cheapest move was to connect them. As multimodal pretraining matures, the trend is to dissolve the bridge into a single token stream, but the constraints this lesson isolated (how many visual tokens the model must read, and whether the language prior overrides weak visual evidence) are properties of the token budget, not of any particular bridge, so they persist. ## Takeaways - **Fusion** ranges from late (CLIP, similarity only) through early (concatenate at the input) to **cross-attention**, where one modality's queries read another's keys and values at every layer — the middle ground that powers modern vision-language models. - **Vision-language models** connect a frozen vision encoder to a frozen LLM with a small trained bridge: **Flamingo** (perceiver resampler $+$ zero-initialized gated cross-attention that starts as the identity), **BLIP-2** (Q-Former soft tokens), **LLaVA** (a linear projector, the minimal design). - The **token budget** is the practical difference: LLaVA emits one soft token per patch ($576$ for a $336^2$ image), so attention costs $O((576+T)^2)$, while a resampler or Q-Former fixes the count at $\approx 32$–$64$ latents, trading spatial detail for a small constant prefix — the reason to prefer a resampler at high resolution or over video. - Two failure modes trace to the bridge: **object hallucination** (too few visual tokens let the language prior override the image) and **fine-detail blindness** (compression discards the high-frequency detail OCR and counting need); both ease with more resolution or tokens, at quadratic attention cost. - The **recipe** is contrastive or feature-alignment pretraining, then **instruction tuning** on (image, instruction, response) triples; LLaVA bootstraps those triples from a text-only LLM. **Evaluation** splits by ability: retrieval (Recall@$K$) scores alignment, captioning (CIDEr) scores grounded generation, VQA (accuracy) scores cross-modal reasoning. - **Frontier direction:** natively multimodal models turn every modality into tokens and dissolve the bridge into one decoder stream (any-to-any), but the token-budget constraints this lesson isolated persist regardless of bridge. [^alayrac-flamingo]: **Alayrac et al.**, _Flamingo: a Visual Language Model for Few-Shot Learning_, NeurIPS 2022 — a frozen vision encoder and LLM bridged by a perceiver resampler and zero-initialized gated cross-attention layers. [^li-blip]: **Li et al.**, _BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation_, ICML 2022 — the captioning/filtering bootstrap and the image-text contrastive, matching, and generation objectives BLIP-2 builds on. [^li-blip2]: **Li et al.**, _BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models_, ICML 2023 — the Q-Former: learned query tokens that extract a fixed set of visual features for a frozen LLM. [^liu-llava]: **Liu et al.**, _Visual Instruction Tuning_ (LLaVA), NeurIPS 2023 — a single linear/MLP projector from a frozen CLIP vision encoder into a frozen LLM, trained on instruction triples bootstrapped from a text-only model. [^team-chameleon]: **Chameleon Team (Meta)**, _Chameleon: Mixed-Modal Early-Fusion Foundation Models_, 2024 — a single decoder-only Transformer over interleaved image and text tokens, reading and generating both modalities without separate encoders or projectors. [^chollet-multimodal]: **Chollet**, _Deep Learning with Python_, Ch. 11 — combining modalities and multimodal architectures: how text and image branches are encoded and fused for joint tasks.