Denoising Sequence-to-Sequence Pretraining: BART

The Transformer splits into three families by which stack they keep and how they mask. Two of them dominate pretraining: the encoder-only model (BERT) corrupts input tokens and reconstructs them with full bidirectional context, and the decoder-only model (GPT) predicts the next token causally. The first reads but cannot generate; the second generates but reads only its own left context. Sequence-to-sequence pretraining keeps both halves and trains the whole encoder–decoder as a denoising autoencoder: a bidirectional encoder reads a corrupted document, and an autoregressive decoder reconstructs the clean original.¹

Denoising as the unifying objective

The pretraining objective is the conditional likelihood of the clean sequence given the corrupted one. With $\theta$ the model parameters and a corpus of clean documents $\mathcal{D}$, the loss is the negative log-likelihood

This lifts the denoising autoencoder objective from continuous vectors to discrete sequences. There the model maps a corrupted vector $\tilde{x}=x+\epsilon$ back to the clean $x$ and learns the data manifold; here it maps a corrupted token sequence back to the clean sequence and learns the structure of language.²

The corruption is what forbids the model from copying its input verbatim. A pure copy would make $p_{\theta }(x\mid x)$ trivially maximal and teach nothing, exactly as the identity defeats an autoencoder. A good $g$ is one whose undoing requires modeling syntax, coreference, world knowledge, and discourse, and whose pretrained weights transfer to the downstream task.

BART: noising the whole document

BART (Bidirectional and Auto-Regressive Transformers) is the canonical instance.³ It keeps the standard Transformer encoder–decoder unchanged and leaves only one design choice: the noise function $g$. Unlike BERT, which corrupts only at the token level (a fixed fraction of positions become [MASK]), BART permits corruptions that change the length and order of the sequence, because the decoder reconstructs the original from scratch rather than predicting in place.

Text infilling is the most important of the five. Token masking tells the model exactly how many tokens are missing (one mask per gap); infilling hides that count, so the decoder must learn span length as well as span content. This single change accounts for most of BART's gains on generation tasks.³

The span length is $\ell \sim \text{Poisson}(\lambda =3)$, whose probability mass $P(\ell =k)=e^{-3}{3}^k/k!$ gives $P(0)\approx 0.050$, $P(1)\approx 0.149$, $P(2)\approx 0.224$, $P(3)\approx 0.224$, $P(4)\approx 0.168$, and a right tail $P(\ell \ge 7)\approx 0.034$. About $5\%$ of draws are $\ell =0$, so roughly one in twenty masks is a pure insertion the model must learn to delete. To corrupt a $30\%$ token budget in a document of $n=512$ tokens, BART samples spans until the deleted count reaches $0.30\times 512\approx 154$ tokens; at a mean span of $3$ that is about $154/3\approx 51$ masks. The encoder input therefore shrinks from $512$ to roughly $512-154+51=409$ positions (each span of mean length $3$ collapses to one [MASK]), while the decoder still emits all $512$. A decoder-only masked-LM cannot represent this length mismatch: BERT keeps the sequence length fixed at $512$ and predicts in place.

The corruptions compose: BART's strongest configuration combines text infilling with sentence permutation, corrupting structure at both the token and discourse levels. Empirically, token deletion and infilling help generation; sentence permutation and rotation alone help less, but stack usefully with infilling.

BART architecture and its two specializations

BART is architecturally a vanilla Transformer with one cosmetic change: it uses a GELU activation in place of ReLU. The structure is the standard pairing of a bidirectional encoder with an autoregressive decoder, and that pairing is what lets BART subsume both BERT and GPT as degenerate cases.³

The three families differ only in the attention mask each token sees. BERT reads every position of the input in both directions but has no decoder to generate. GPT sees only its own left context and generates in place. BART splits the two: its encoder is fully bidirectional over the corrupted source, and its decoder is causal over the target while cross-attending to the whole encoder output.

BART-large is a 12-layer encoder over a 12-layer decoder with model width $d_{\text{model}}=1024$, feed-forward width $d_{\text{ff}}=4096$, and $16$ attention heads, for roughly $406$M parameters; BART-base halves the depth to $6+6$ with $d_{\text{model}}=768$ and about $140$M parameters. The vocabulary is the GPT-2 byte-level BPE of $|V|=50,265$ subwords, so the model shares one embedding matrix $E\in {\mathbb{R}}^{|V|\times d_{\text{model}}}$ between the encoder input, the decoder input, and the output projection.

Dimension-annotated forward pass

Trace one denoising step through the stack with a source of $S$ corrupted tokens and a target of $T$ clean tokens, batch size $B$. The encoder runs once; the decoder runs once per position at inference and once over the whole shifted target at training (teacher forcing).

Cross-attention is where the two stacks meet. In each decoder block the queries come from the target stream, $Q=H_{\text{dec}}{W}_Q\in {\mathbb{R}}^{B\times T\times d_k}$, while the keys and values come from the frozen encoder output, $K=H_{\text{enc}}^{(L)}{W}_K$ and $V=H_{\text{enc}}^{(L)}{W}_V$, both $\in {\mathbb{R}}^{B\times S\times d_k}$ per head with $d_k=d_{\text{model}}/h=64$. The attention scores form a $T\times S$ matrix per head, so every target position reads the entire source. The encoder is run once and its $H_{\text{enc}}^{(L)}$ is cached for all $T$ decoding steps; only the decoder's self-attention key-value cache grows with each generated token.

This continues in Text-to-Text Transfer and Conditional Generation, which specializes the denoising idea to T5's span corruption and PEGASUS's gap sentences, covers fine-tuning and beam-search decoding, and proves why a bidirectional encoder--decoder beats a decoder-only model when the output is conditioned on a full input.

Takeaways

• Denoising seq2seq pretraining trains a full encoder–decoder to reconstruct a clean text $x$ from a corrupted $\tilde{x}=g(x)$, maximizing $\log p_{\theta }(x\mid \tilde{x})$. It is the denoising autoencoder objective lifted to discrete sequences, and it bridges encoder-only (BERT) and decoder-only (GPT) pretraining.
• The noise function $g$ is the only design choice: a corruption that destroys structure the model must rebuild makes the pretext task informative; the identity $g$ collapses the objective back to ordinary language modeling.
• BART keeps a vanilla Transformer and varies only the noise: token masking, token deletion, text infilling (a whole span collapses to one [MASK], hiding its length), sentence permutation, and document rotation. Poisson($3$) infilling forces the decoder to count as well as fill; infilling plus permutation is the strongest mix.
• BART specializes: its encoder with token masking is BERT; its decoder with an empty source is GPT — one architecture and one $g$ contain both as boundary cases.
• The dimension-annotated forward pass shows the two stacks meet at cross-attention: decoder queries read frozen encoder keys/values over a $T\times S$ score matrix, and the encoder is run once and cached for all decoding steps.