---
title: "Denoising Sequence-to-Sequence Pretraining: BART"
module: Large Models & Agents
moduleNumber: 10
lessonNumber: 3
order: 1003
summary: >
  BERT corrupts and reconstructs; GPT predicts the next token. Sequence-to-sequence
  pretraining unifies both by training a full encoder–decoder as a denoising
  autoencoder: corrupt the text with a noise function, then reconstruct the original
  through a bidirectional encoder and an autoregressive decoder. This first part
  derives the denoising objective, catalogs BART's five noise functions (with a
  worked Poisson-infilling budget), proves BART specializes to both BERT and GPT,
  and traces a dimension-annotated forward pass through its encoder--decoder. T5,
  PEGASUS, fine-tuning, and decoding continue in part two.
topics: [Large Models & Agents]
sources:
  - book: Chollet
    ref: "Ch. 11 — Deep Learning for Text; sequence-to-sequence models"
  - book: Goodfellow
    ref: "Ch. 14 — Autoencoders; denoising autoencoders"
---

[The Transformer](/deep-learning/architectures/the-transformer-architecture)
splits into three families by which stack they keep and how they mask. Two of
them dominate pretraining: the **encoder-only** model (BERT) corrupts input
tokens and reconstructs them with full bidirectional context, and the
**decoder-only** model (GPT) predicts the next token causally. The first reads
but cannot generate; the second generates but reads only its own left context.
Sequence-to-sequence pretraining keeps both halves and trains the whole
encoder–decoder as a [denoising autoencoder](/deep-learning/generative-models/autoencoders):
a bidirectional encoder reads a corrupted document, and an autoregressive decoder
reconstructs the clean original.[^chollet-seq2seq]

> **Definition (Denoising sequence-to-sequence pretraining).** Given a noise
> function $g$ that maps a clean text $x$ to a corrupted version
> $\tilde x = g(x)$, train an encoder–decoder model $p_\theta$ to reconstruct
> $x$ from $\tilde x$ by maximizing the conditional log-likelihood
> $$
> \mathcal{L}(\theta) = \sum_{(x,\tilde x)} \log p_\theta\parens{x \mid \tilde x}
> = \sum_{(x,\tilde x)} \sum_{t=1}^{\abs{x}} \log p_\theta\parens{x_t \mid x_{<t}, \tilde x}.
> $$
> The encoder sees all of $\tilde x$ bidirectionally; the decoder produces $x$
> one token at a time, conditioned on the encoder output and its own prefix.

## Denoising as the unifying objective

The pretraining objective is the conditional likelihood of the clean sequence
given the corrupted one. With $\theta$ the model parameters and a corpus of
clean documents $\mathcal{D}$, the loss is the negative log-likelihood

$$
\mathcal{L}(\theta)
= -\,\mathbb{E}_{x \sim \mathcal{D}}\;
   \mathbb{E}_{\tilde x \sim g(\cdot \mid x)}
   \brackets{\, \log p_\theta\parens{x \mid \tilde x} \,},
\qquad
\log p_\theta\parens{x \mid \tilde x}
= \sum_{t=1}^{\abs{x}} \log p_\theta\parens{x_t \mid x_{<t}, \tilde x}.
$$

This lifts the [denoising autoencoder](/deep-learning/generative-models/autoencoders)
objective from continuous vectors to discrete sequences. There the model
maps a corrupted vector $\tilde x = x + \varepsilon$ back to the clean $x$ and
learns the data manifold; here it maps a corrupted token sequence back to the
clean sequence and learns the structure of language.[^gf-dae]

> **Definition (Noise function).** A stochastic map $g$ from a clean sequence
> $x$ to a corrupted sequence $\tilde x$, drawn from a corruption distribution
> $C(\tilde x \mid x)$. Setting $g$ to the identity ($\tilde x = x$) reduces the
> objective to ordinary language modeling; choosing a corruption that destroys
> structure the model must rebuild is what makes the pretext task informative.

The corruption is what forbids the model from copying its input verbatim. A pure
copy would make $p_\theta(x \mid x)$ trivially maximal and teach nothing, exactly
as the identity defeats an autoencoder. A good $g$ is one whose undoing requires
modeling syntax, coreference, world knowledge, and discourse, and whose pretrained
weights transfer to the downstream task.

$$
% caption: Denoising seq2seq: a noise function $g$ corrupts $x$ to $\tilde x$; the bidirectional
% encoder reads $\tilde x$, and the autoregressive decoder reconstructs the clean $x$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=20mm, minimum height=9mm, align=center},
  stk/.style={draw, minimum width=26mm, minimum height=11mm, align=center, thick}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  \node[box] (x) at (0,0) {\texttt{clean}\\\texttt{text x}};
  \node[box, draw=red, text=red, fill=red!15] (xt) at (3.2,0) {\texttt{corrupt}\\\texttt{x-tilde}};
  \node[stk, draw=acc, text=acc, fill=acc!12] (enc) at (6.6,0) {\texttt{encoder}\\\texttt{(bidirectional)}};
  \node[stk, draw=acc, text=acc, fill=acc!12] (dec) at (10.0,0) {\texttt{decoder}\\\texttt{(autoregressive)}};
  \node[box, draw=green, text=green, fill=green!15] (xh) at (13.2,0) {\texttt{rebuilt}\\\texttt{text x}};
  \draw[->, red, thick] (x) -- (xt) node[midway, above, text=red] {\texttt{noise g}};
  \draw[->, black, thick] (xt) -- (enc);
  \draw[->, acc, thick] (enc) -- (dec) node[midway, above, yshift=7mm, text=acc] {\texttt{context}};
  \draw[->, green, thick] (dec) -- (xh);
  \draw[->, green, thick, dashed] (xh.south) .. controls (13.2,-2.0) and (0,-2.0) .. (x.south)
    node[midway, below, text=green] {\texttt{target is the clean x}};
\end{tikzpicture}
$$

## BART: noising the whole document

**BART** (Bidirectional and Auto-Regressive Transformers) is the canonical
instance.[^bart] It keeps the standard Transformer encoder–decoder unchanged and
leaves only one design choice: the noise function $g$. Unlike BERT, which
corrupts only at the token level (a fixed fraction of positions become `[MASK]`),
BART permits corruptions that change the **length** and **order** of the sequence,
because the decoder reconstructs the original from scratch rather than predicting
in place.

| Noise function | What it does | What the model must learn |
| --- | --- | --- |
| Token masking | replace random tokens with `[MASK]` | infer a token from bidirectional context (BERT-style) |
| Token deletion | delete random tokens, no marker left | _which positions_ are missing, then fill them |
| Text infilling | replace a span of length $\ell \sim \text{Poisson}(\lambda)$ with one `[MASK]` | predict span content _and_ its length ($\ell$ unknown) |
| Sentence permutation | shuffle the order of sentences | restore document-level coherence and discourse order |
| Document rotation | rotate so a random token starts the document | find the true start of the document |

> **Definition (Text infilling).** A noise function that samples a number of
> spans whose lengths are drawn from $\text{Poisson}(\lambda)$ (with $\lambda = 3$
> in BART) and replaces each entire span with a _single_ `[MASK]` token. A
> sampled length of $\ell = 0$ inserts a `[MASK]` without deleting anything. The
> model must predict both _how many_ tokens each mask hides and _what_ they are.

Text infilling is the most important of the five. Token masking tells the model
exactly how many tokens are missing (one mask per gap); infilling hides that
count, so the decoder must learn span length as well as span content. This single
change accounts for most of BART's gains on generation tasks.[^bart]

The span length is $\ell \sim \text{Poisson}(\lambda = 3)$, whose probability mass
$P(\ell = k) = e^{-3}\, 3^k / k!$ gives $P(0) \approx 0.050$, $P(1) \approx 0.149$,
$P(2) \approx 0.224$, $P(3) \approx 0.224$, $P(4) \approx 0.168$, and a right tail
$P(\ell \ge 7) \approx 0.034$. About $5\%$ of draws are $\ell = 0$, so roughly one
in twenty masks is a pure insertion the model must learn to delete. To corrupt a
$30\%$ token budget in a document of $n = 512$ tokens, BART samples spans until the
deleted count reaches $0.30 \times 512 \approx 154$ tokens; at a mean span of $3$
that is about $154/3 \approx 51$ masks. The encoder input therefore shrinks from
$512$ to roughly $512 - 154 + 51 = 409$ positions (each span of mean length $3$
collapses to one `[MASK]`), while the decoder still emits all $512$. A decoder-only
masked-LM cannot represent this length mismatch: BERT keeps the sequence length
fixed at $512$ and predicts in place.

> **Takeaway.** Poisson($3$) infilling is a length-destroying corruption. The
> encoder sees fewer positions than the target has, so the decoder cannot align
> masks to outputs one-to-one; it must _count_ as well as _fill_. A fixed-slot
> BERT mask never requires this.

$$
% caption: Text infilling. A multi-token span collapses to one MASK in the encoder input; the
% decoder must recover both the span's length and its tokens, not just fill a fixed slot.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tok/.style={draw, minimum width=12mm, minimum height=7mm, align=center, fill=black!8},
  msk/.style={draw=red, text=red, thick, minimum width=12mm, minimum height=7mm, align=center, fill=red!15},
  rec/.style={draw=green, text=green, thick, minimum width=12mm, minimum height=7mm, align=center, fill=green!15}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % clean source (top)
  \node[tok] (a1) at (0,3.0) {the};
  \node[tok] (a2) at (1.4,3.0) {cat};
  \node[tok] (a3) at (2.8,3.0) {sat};
  \node[tok] (a4) at (4.2,3.0) {on};
  \node[tok] (a5) at (5.6,3.0) {the};
  \node[tok] (a6) at (7.0,3.0) {mat};
  \node[anchor=east, font=\footnotesize] at (-0.8,3.0) {\texttt{clean x}};
  % bracket marking the corrupted span "sat on the"
  \draw[red, thick] (2.1,2.55) -- (2.1,2.35) -- (6.3,2.35) -- (6.3,2.55);
  \node[red, anchor=north, font=\footnotesize] at (4.2,2.3) {\texttt{span "sat on the"}};
  % corrupted encoder input (middle) -- the 3-token span becomes ONE mask
  \node[tok] (b1) at (1.0,0.9) {the};
  \node[tok] (b2) at (2.6,0.9) {cat};
  \node[msk] (b3) at (4.2,0.9) {MASK};
  \node[tok] (b4) at (5.8,0.9) {mat};
  \node[anchor=east, font=\footnotesize] at (0.0,0.9) {\texttt{x-tilde}};
  \draw[->, red, thick] (4.2,2.05) -- (4.2,1.3) node[midway, right, text=red, font=\footnotesize] {\texttt{infill}};
  % decoder output (bottom) -- recovers all three tokens
  \node[rec] (c3) at (2.8,-1.2) {sat};
  \node[rec] (c4) at (4.2,-1.2) {on};
  \node[rec] (c5) at (5.6,-1.2) {the};
  \node[anchor=east, font=\footnotesize] at (1.6,-1.2) {\texttt{decoder}};
  \draw[->, green, thick] (b3.south) .. controls (4.2,-0.1) and (4.2,-0.3) .. (4.2,-0.85);
  \node[green, anchor=west, font=\footnotesize] at (6.4,-1.2) {\texttt{3 tokens from 1 mask}};
\end{tikzpicture}
$$

The corruptions compose: BART's strongest configuration combines text infilling
with sentence permutation, corrupting structure at both the token and discourse
levels. Empirically, token deletion and infilling help generation; sentence
permutation and rotation alone help less, but stack usefully with infilling.

$$
% caption: BART corrupts a document with several noise functions at once, then trains the full
% encoder–decoder to output the original; infilling plus sentence permutation is the strongest mix.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  n/.style={draw, minimum width=30mm, minimum height=7mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  \node[n] (doc) at (0,0) {\texttt{clean document}};
  \node[n, draw=red, text=red, fill=red!15] (m1) at (-3.4,-1.7) {\texttt{token mask/delete}};
  \node[n, draw=red, text=red, fill=red!15] (m2) at (0,-1.7) {\texttt{text infilling}};
  \node[n, draw=red, text=red, fill=red!15] (m3) at (3.4,-1.7) {\texttt{permute/rotate}};
  \node[n, draw=acc, text=acc, fill=acc!12, thick] (mod) at (0,-3.6) {\texttt{encoder + decoder}};
  \node[n, draw=green, text=green, fill=green!15] (rec) at (0,-5.3) {\texttt{reconstruct original}};
  \draw[->, red, thick] (doc) -- (m1);
  \draw[->, red, thick] (doc) -- (m2);
  \draw[->, red, thick] (doc) -- (m3);
  \draw[->, black, thick] (m1) -- (mod);
  \draw[->, black, thick] (m2) -- (mod);
  \draw[->, black, thick] (m3) -- (mod);
  \draw[->, green, thick] (mod) -- (rec);
\end{tikzpicture}
$$

## BART architecture and its two specializations

BART is architecturally a vanilla Transformer with one cosmetic change: it uses a
GELU activation in place of ReLU. The structure is the standard pairing of a
bidirectional encoder with an autoregressive decoder, and that pairing is what lets
BART subsume both BERT and GPT as degenerate cases.[^bart]

> **Theorem (BART specializes to BERT and GPT).** The BART encoder run alone, with
> token masking as the only noise and a per-position classification head, is BERT;
> the BART decoder run alone, with an empty encoder input ($\tilde x = \varnothing$),
> is a causal (GPT-style) language model.

> **Proof.** Token masking sets $\tilde x$ to $x$ with a fraction of positions
> replaced by `[MASK]`; reading the encoder's final state at each masked position
> through a softmax over the vocabulary gives $p(x_i \mid \tilde x)$ with full
> bidirectional context, the BERT masked-LM objective. Conversely, fix the
> encoder input to the empty sequence so cross-attention is constant; the decoder
> then factors $p_\theta(x) = \prod_t p_\theta(x_t \mid x_{<t})$ over its own
> causal self-attention alone, which is the decoder-only language model of GPT.
> BART contains both as boundary cases of one $g$ and one architecture. $\qed$

$$
% caption: BART joins BERT and GPT. A bidirectional encoder reads the corrupted input; an
% autoregressive decoder cross-attends to it and reconstructs the original left to right.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  sub/.style={draw, minimum width=30mm, minimum height=6mm, align=center},
  io/.style={font=\scriptsize, text=black}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % ENCODER
  \draw[black, dashed] (-1.9,1.2) rectangle (1.9,4.0);
  \node[io] (ein) at (0,0.2) {\texttt{corrupted x-tilde}};
  \node[sub, draw=acc, text=acc, thick, fill=acc!12] (e1) at (0,1.7) {\texttt{self-attention}};
  \node[sub, draw=acc, text=acc, thick, fill=acc!12] (e2) at (0,3.3) {\texttt{feed-forward}};
  \node[io] at (0,4.5) {\texttt{bidirectional encoder}};
  \draw[->, thick] (ein) -- (e1);
  \draw[->, thick] (e1) -- (e2);
  % DECODER
  \begin{scope}[xshift=7.2cm]
  \draw[black, dashed] (-1.9,1.2) rectangle (1.9,5.6);
  \node[io] (din) at (0,0.2) {\texttt{target (shifted)}};
  \node[sub, draw=red, text=red, thick, fill=red!15] (d1) at (0,1.7) {\texttt{masked self-attn}};
  \node[sub, draw=green, text=green, thick, fill=green!15] (d2) at (0,3.3) {\texttt{cross-attention}};
  \node[sub, draw=acc, text=acc, thick, fill=acc!12] (d3) at (0,4.9) {\texttt{feed-forward}};
  \node[sub, draw=acc, text=acc] (out) at (0,6.4) {\texttt{linear + softmax}};
  \node[io] (yo) at (0,7.4) {\texttt{rebuilt x}};
  \draw[->, thick] (din) -- (d1);
  \draw[->, thick] (d1) -- (d2);
  \draw[->, thick] (d2) -- (d3);
  \draw[->, thick] (d3) -- (out);
  \draw[->, thick] (out) -- (yo);
  \end{scope}
  % cross-attention routing
  \draw[->, green, thick] (1.9,3.3) .. controls (3.6,3.3) and (4.0,3.3) .. (5.3,3.3)
    node[midway, above, text=green, font=\footnotesize] {\texttt{keys, values}};
\end{tikzpicture}
$$

| Slice of BART | Attention | Objective | Equivalent to |
| --- | --- | --- | --- |
| Encoder only | bidirectional | masked token prediction | BERT |
| Decoder only | causal | next-token prediction | GPT |
| Encoder $+$ decoder | encoder full, decoder causal | denoising seq2seq | BART |

The three families differ only in the attention mask each token sees. BERT reads
every position of the input in both directions but has no decoder to generate.
GPT sees only its own left context and generates in place. BART splits the two:
its encoder is fully bidirectional over the corrupted source, and its decoder is
causal over the target while cross-attending to the whole encoder output.

$$
% caption: The three attention regimes. Filled cells are visible; a row is a query, a column a key.
% BERT: full bidirectional. GPT: lower-triangular causal. BART decoder: causal self-attn plus full
% cross-attention to the encoder.
\begin{tikzpicture}[>=stealth, font=\scriptsize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % helper: a 4x4 grid at (ox,oy); pass a macro would be cleaner, but inline for clarity
  % --- BERT: full bidirectional ---
  \begin{scope}[shift={(0,0)}]
    \foreach \r in {0,1,2,3}{ \foreach \c in {0,1,2,3}{
      \fill[acc!30] (\c*0.42,-\r*0.42) rectangle ++(0.42,-0.42);
    }}
    \draw[black] (0,0) grid[step=0.42] (1.68,-1.68);
    \node[anchor=south, font=\footnotesize] at (0.84,0.15) {\texttt{BERT}};
    \node[anchor=north, text=black] at (0.84,-1.9) {\texttt{bidirectional}};
  \end{scope}
  % --- GPT: lower-triangular causal ---
  \begin{scope}[shift={(3.3,0)}]
    \foreach \r in {0,1,2,3}{ \foreach \c in {0,1,2,3}{
      \ifnum\c>\r\else \fill[red!30] (\c*0.42,-\r*0.42) rectangle ++(0.42,-0.42);\fi
    }}
    \draw[black] (0,0) grid[step=0.42] (1.68,-1.68);
    \node[anchor=south, font=\footnotesize] at (0.84,0.15) {\texttt{GPT}};
    \node[anchor=north, text=black] at (0.84,-1.9) {\texttt{causal}};
  \end{scope}
  % --- BART decoder: causal self + full cross ---
  \begin{scope}[shift={(6.6,0)}]
    \foreach \r in {0,1,2,3}{ \foreach \c in {0,1,2,3}{
      \ifnum\c>\r\else \fill[red!30] (\c*0.42,-\r*0.42) rectangle ++(0.42,-0.42);\fi
    }}
    \draw[black] (0,0) grid[step=0.42] (1.68,-1.68);
    \node[anchor=south, font=\footnotesize] at (0.84,0.15) {\texttt{BART self}};
    \node[anchor=north, text=black] at (0.84,-1.9) {\texttt{causal}};
  \end{scope}
  \begin{scope}[shift={(9.9,0)}]
    \foreach \r in {0,1,2,3}{ \foreach \c in {0,1,2,3}{
      \fill[green!30] (\c*0.42,-\r*0.42) rectangle ++(0.42,-0.42);
    }}
    \draw[black] (0,0) grid[step=0.42] (1.68,-1.68);
    \node[anchor=south, font=\footnotesize] at (0.84,0.15) {\texttt{BART cross}};
    \node[anchor=north, text=black] at (0.84,-1.9) {\texttt{full to encoder}};
  \end{scope}
\end{tikzpicture}
$$

BART-large is a 12-layer encoder over a 12-layer decoder with model width
$d_\text{model} = 1024$, feed-forward width $d_\text{ff} = 4096$, and $16$
attention heads, for roughly $406$M parameters; BART-base halves the depth to $6+6$
with $d_\text{model} = 768$ and about $140$M parameters. The vocabulary is the
GPT-2 byte-level BPE of $|V| = 50{,}265$ subwords, so the model shares one embedding
matrix $E \in \mathbb{R}^{|V| \times d_\text{model}}$ between the encoder input, the
decoder input, and the output projection.

### Dimension-annotated forward pass

Trace one denoising step through the stack with a source of $S$ corrupted tokens
and a target of $T$ clean tokens, batch size $B$. The encoder runs once; the
decoder runs once per position at inference and once over the whole shifted target
at training (teacher forcing).

$$
\begin{aligned}
\text{tok ids } \tilde x &\in \{0,\dots,|V|-1\}^{B \times S}
  &&\text{corrupted source, integer tokens} \\
H^{(0)}_\text{enc} = E[\tilde x] + P_{1:S}
  &\in \mathbb{R}^{B \times S \times d_\text{model}}
  &&\text{embed} + \text{learned position } P \\
H^{(L)}_\text{enc} = \text{Encoder}\parens{H^{(0)}_\text{enc}}
  &\in \mathbb{R}^{B \times S \times d_\text{model}}
  &&\text{12 bidirectional blocks} \\
H^{(0)}_\text{dec} = E[y_{\text{shift}}] + P_{1:T}
  &\in \mathbb{R}^{B \times T \times d_\text{model}}
  &&\text{shifted target, causal input} \\
H^{(L)}_\text{dec} = \text{Decoder}\parens{H^{(0)}_\text{dec},\, H^{(L)}_\text{enc}}
  &\in \mathbb{R}^{B \times T \times d_\text{model}}
  &&\text{self-attn} + \text{cross-attn} \\
Z = H^{(L)}_\text{dec}\, E^\top
  &\in \mathbb{R}^{B \times T \times |V|}
  &&\text{tied output projection (logits)} \\
p_\theta(\cdot \mid y_{<t}, \tilde x) = \text{softmax}(Z_{:,t,:})
  &\in \Delta^{|V|-1}
  &&\text{per-position vocabulary distribution}
\end{aligned}
$$

Cross-attention is where the two stacks meet. In each decoder block the queries come
from the target stream, $Q = H_\text{dec} W_Q \in \mathbb{R}^{B \times T \times d_k}$,
while the keys and values come from the _frozen_ encoder output,
$K = H^{(L)}_\text{enc} W_K$ and $V = H^{(L)}_\text{enc} W_V$, both
$\in \mathbb{R}^{B \times S \times d_k}$ per head with $d_k = d_\text{model}/h = 64$.
The attention scores form a $T \times S$ matrix per head, so every target position
reads the entire source. The encoder is run once and its $H^{(L)}_\text{enc}$ is
cached for all $T$ decoding steps; only the decoder's self-attention key-value cache
grows with each generated token.

$$
% caption: Dimension-annotated denoising forward pass. The encoder maps a $B\times S$ corrupted
% source to $B\times S\times d$; the decoder cross-attends to it and projects to $B\times T\times|V|$ logits.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  blk/.style={draw, minimum width=34mm, minimum height=8mm, align=center},
  sh/.style={font=\scriptsize, text=black}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % encoder column (shape labels to the LEFT to keep the cross-attn lane clear)
  \node[blk] (es) at (0,0) {\texttt{corrupted source}};
  \node[sh, anchor=east] at (-2.1,0) {\texttt{B x S}};
  \node[blk, draw=acc, text=acc, fill=acc!12] (ee) at (0,1.6) {\texttt{embed + position}};
  \node[sh, anchor=east] at (-2.1,1.6) {\texttt{B x S x d}};
  \node[blk, draw=acc, text=acc, fill=acc!12, thick] (enc) at (0,3.2) {\texttt{encoder x12}};
  \node[sh, anchor=east] at (-2.1,3.2) {\texttt{B x S x d}};
  \draw[->, thick] (es) -- (ee);
  \draw[->, thick] (ee) -- (enc);
  % decoder column (shape labels to the RIGHT)
  \begin{scope}[xshift=8.0cm]
  \node[blk] (ds) at (0,0) {\texttt{shifted target}};
  \node[sh, anchor=west] at (2.1,0) {\texttt{B x T}};
  \node[blk, draw=acc, text=acc, fill=acc!12] (de) at (0,1.6) {\texttt{embed + position}};
  \node[blk, draw=green, text=green, fill=green!12, thick] (dec) at (0,3.2) {\texttt{decoder x12}};
  \node[blk, draw=acc, text=acc, fill=acc!12] (proj) at (0,4.8) {\texttt{tied projection}};
  \node[sh, anchor=west] at (2.1,4.8) {\texttt{B x T x V}};
  \draw[->, thick] (ds) -- (de);
  \draw[->, thick] (de) -- (dec);
  \draw[->, thick] (dec) -- (proj);
  \end{scope}
  % cross-attention link: encoder output feeds decoder self+cross block
  \draw[->, green, thick] (enc.east) -- ++(1.6,0) |- (dec.west)
    node[pos=0.28, above, sh, text=green] {\texttt{K, V}};
\end{tikzpicture}
$$

This continues in [Text-to-Text Transfer and Conditional Generation](/deep-learning/large-models-and-agents/text-to-text-transfer-and-conditional-generation), which specializes the denoising idea to T5's span corruption and PEGASUS's gap sentences, covers fine-tuning and beam-search decoding, and proves why a bidirectional encoder--decoder beats a decoder-only model when the output is conditioned on a full input.

## Takeaways

- **Denoising seq2seq pretraining** trains a full encoder–decoder to reconstruct
  a clean text $x$ from a corrupted $\tilde x = g(x)$, maximizing
  $\log p_\theta(x \mid \tilde x)$. It is the [denoising autoencoder](/deep-learning/generative-models/autoencoders)
  objective lifted to discrete sequences, and it bridges encoder-only (BERT) and
  decoder-only (GPT) pretraining.
- The **noise function** $g$ is the only design choice: a corruption that destroys
  structure the model must rebuild makes the pretext task informative; the
  identity $g$ collapses the objective back to ordinary language modeling.
- **BART** keeps a vanilla Transformer and varies only the noise: token masking,
  token deletion, **text infilling** (a whole span collapses to one `[MASK]`,
  hiding its length), sentence permutation, and document rotation. Poisson($3$)
  infilling forces the decoder to _count_ as well as fill; infilling plus
  permutation is the strongest mix.
- BART **specializes**: its encoder with token masking is BERT; its decoder with an
  empty source is GPT — one architecture and one $g$ contain both as boundary cases.
- The **dimension-annotated forward pass** shows the two stacks meet at
  **cross-attention**: decoder queries read frozen encoder keys/values over a
  $T \times S$ score matrix, and the encoder is run once and cached for all decoding
  steps.

[^bart]: **Lewis et al.**, _BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension_, 2020 — a standard Transformer encoder–decoder pretrained as a denoising autoencoder; text infilling is the most effective noise.
[^bert]: **Devlin et al.**, _BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding_, 2019 — bidirectional encoder pretrained by masked language modeling; the encoder-only family BART subsumes.
[^gf-dae]: **Goodfellow**, _Deep Learning_, §14.5 — denoising autoencoders: reconstruct a clean input from a corrupted one, learning the data manifold; the continuous analogue of denoising seq2seq pretraining.
[^chollet-seq2seq]: **Chollet**, _Deep Learning with Python_, Ch. 11 — sequence-to-sequence models with an encoder and a decoder, and why conditional generation tasks pair a bidirectional reader with an autoregressive writer.