Fusion and Vision-Language Models

This builds on Multimodal Contrastive Learning, which put images and text in one space with a dual encoder and a symmetric contrastive loss, giving a picture and its caption a high dot product. A contrastive model can compare modalities but cannot let one read another: nothing in CLIP lets a text token pull information from a specific image region. For captioning and visual question answering the modalities have to fuse inside the network. This lesson builds that fusion, then uses it to connect a frozen language model to vision.

Fusion strategies

Contrastive models compare modalities; they do not let one modality read another. For tasks that need interaction, captioning, visual question answering, the modalities must fuse inside the network. Where the fusion happens defines the strategy.

The middle ground, and the one that powers modern vision-language models, is cross-attention: keep separate encoders but let one modality's queries attend to the other modality's keys and values.

Vision-language models: feeding images into an LLM

The dominant design does not train a multimodal model from scratch; it connects a strong frozen vision encoder to a strong frozen LLM with a small trainable bridge, so the language model gains visual input without losing its language ability. Three designs span the space.

Flamingo: perceiver resampler and gated cross-attention

Flamingo keeps a frozen vision encoder and a frozen LLM and inserts two trained pieces.² A perceiver resampler maps a variable number of image features to a fixed small set of latent tokens, and gated cross-attention layers interleaved into the frozen LLM let text tokens attend to those visual latents.

The cross-attention is gated with a $\tanh$ gate initialized at zero, so at the start of training the visual stream contributes nothing and the model is exactly the original LLM; the gate opens as training proceeds. This is what lets a frozen language model absorb a new modality without destabilizing.

BLIP-2: the Q-Former

BLIP-2 bridges a frozen image encoder and a frozen LLM with a lightweight querying Transformer (Q-Former).³ A small set of learned query tokens extracts a fixed number of visual features through cross-attention, and a linear layer projects them into the LLM's input space as soft visual tokens. The Q-Former is the only trained module, and it is trained in two stages, inheriting the vision-language objectives of the earlier BLIP.⁴

LLaVA: a linear projector

LLaVA is the minimal design.⁵ A frozen ViT (from CLIP) produces patch features; a single trainable linear (or two-layer MLP) projector maps each patch feature into the frozen LLM's token-embedding space; the projected patches are prepended to the text and the LLM does the rest. There is no resampler and no Q-Former: the projector is the entire bridge, and it makes visual instruction tuning cheap enough to run on modest hardware.

The three differ mainly in the bridge between vision and language, and in how many visual tokens reach the LLM.

The token budget is the practical difference. LLaVA's projector emits one soft token per patch, so a $336\times 336$ image at patch $14$ gives $576$ visual tokens prepended to the prompt; the LLM pays $O((576+T{)}^2)$ attention for a text length $T$. Flamingo's resampler and BLIP-2's Q-Former instead fix the count at $m\approx 32$-$64$ latents regardless of resolution, trading some spatial detail for a constant, small prefix. That is the reason to prefer a resampler at high resolution or over video: it caps the sequence the LLM must read.

Two failure modes recur across these designs, and both trace back to the bridge. Object hallucination, the model confidently naming things not in the image, worsens when too few visual tokens or a weak projector let the language prior override the visual evidence; the LLM falls back on what captions usually say. And fine-detail blindness, missing small text or counts, follows directly from patch size and token budget: a resampler that compresses $576$ patches to $32$ latents discards exactly the high-frequency detail an OCR or counting question needs. Raising input resolution or the visual-token count mitigates both, at quadratic attention cost.

Training recipe and evaluation

The modern pipeline has two phases, and they map onto the two halves of this lesson: align the spaces, then teach the model to follow instructions.

LLaVA's stage 2 is visual instruction tuning: it bootstraps the triples by prompting a text-only LLM to generate questions and answers about images from their captions and bounding boxes, then trains on the synthetic conversations. This is instruction tuning lifted from language alignment into the visual setting.

Three benchmark families score the result, and each isolates a different ability.

Retrieval rewards the contrastive models (CLIP, ALIGN) because their whole training objective is retrieval; captioning and VQA reward the fusion models (Flamingo, BLIP-2, LLaVA) because answering requires the language model to read the image, not merely score it.

Natively multimodal models

Flamingo, BLIP-2, and LLaVA all bolt vision onto a language model trained on text: a frozen LLM gains sight through a trained bridge. The frontier since has moved toward models that are multimodal from the start, and two ideas reframe the bridge designs above.

Any-to-any token models. If every modality can be turned into discrete tokens — text by a tokenizer, images by a vision quantizer, audio by a codec — then a single decoder-only Transformer can read and write all of them in one sequence, with no separate encoders or projectors.⁶ The bridge disappears because there is nothing to bridge: an image is just more tokens in the same stream. This is the multimodal endpoint of the same turn it into tokens move that recurred in the speech and language lessons, and it lets one model both answer questions about an image and generate one, in a single autoregressive pass.

Native mixed-modal pretraining. Rather than freeze a text LLM and attach vision, a natively multimodal model interleaves image and text data throughout pretraining, so the two modalities are represented jointly from the first step and the modality gap of part one never opens as widely. The token-budget arithmetic of this lesson still governs the cost — images are expensive because they are many tokens, and high resolution or video multiplies the count — so the resampler-versus-projector trade (a small fixed prefix versus one token per patch) survives even when there is no frozen LLM to protect.

In short, the bridge modules — resampler, Q-Former, projector — were a transitional design for the era when a strong LLM existed and a strong vision encoder existed and the cheapest move was to connect them. As multimodal pretraining matures, the trend is to dissolve the bridge into a single token stream, but the constraints this lesson isolated (how many visual tokens the model must read, and whether the language prior overrides weak visual evidence) are properties of the token budget, not of any particular bridge, so they persist.

Takeaways

• Fusion ranges from late (CLIP, similarity only) through early (concatenate at the input) to cross-attention, where one modality's queries read another's keys and values at every layer — the middle ground that powers modern vision-language models.
• Vision-language models connect a frozen vision encoder to a frozen LLM with a small trained bridge: Flamingo (perceiver resampler $+$ zero-initialized gated cross-attention that starts as the identity), BLIP-2 (Q-Former soft tokens), LLaVA (a linear projector, the minimal design).
• The token budget is the practical difference: LLaVA emits one soft token per patch ($576$ for a $336^2$ image), so attention costs $O((576+T{)}^2)$, while a resampler or Q-Former fixes the count at $\approx 32$–$64$ latents, trading spatial detail for a small constant prefix — the reason to prefer a resampler at high resolution or over video.
• Two failure modes trace to the bridge: object hallucination (too few visual tokens let the language prior override the image) and fine-detail blindness (compression discards the high-frequency detail OCR and counting need); both ease with more resolution or tokens, at quadratic attention cost.
• The recipe is contrastive or feature-alignment pretraining, then instruction tuning on (image, instruction, response) triples; LLaVA bootstraps those triples from a text-only LLM. Evaluation splits by ability: retrieval (Recall@$K$) scores alignment, captioning (CIDEr) scores grounded generation, VQA (accuracy) scores cross-modal reasoning.
• Frontier direction: natively multimodal models turn every modality into tokens and dissolve the bridge into one decoder stream (any-to-any), but the token-budget constraints this lesson isolated persist regardless of bridge.