Text-to-Text Transfer and Conditional Generation

This builds on Denoising Sequence-to-Sequence Pretraining: BART, which trained a full encoder--decoder as a denoising autoencoder and showed BART subsumes both BERT and GPT. BART reconstructs the whole document from a corrupted copy. This lesson specializes that idea in two directions — a shorter, sentinel- tagged target (T5) and a summarization-matched masking (PEGASUS) — then follows the pretrained model into fine-tuning, decoding, and the structural argument for why the encoder--decoder design persists for conditional generation.

T5: every task is text-to-text

T5 (Text-to-Text Transfer Transformer) pushes the unification one step further.¹ Every NLP problem, including ones that are not naturally generative, is cast as mapping an input string to an output string. Classification emits the class name as text; regression emits the number as a string; translation and summarization are text-to-text already. One encoder–decoder, one cross-entropy loss, one decoding procedure covers them all.

Task	Input string	Target string
Translation	`translate English to German: the house is small`	`das Haus ist klein`
Summarization	`summarize: <article text> ...`	`<one-sentence summary>`
Classification (sentiment)	`sst2 sentence: a charming film`	`positive`
Entailment	`mnli premise: ... hypothesis: ...`	`entailment`
Similarity (regression)	`stsb sentence1: ... sentence2: ...`	`3.8`

T5's pretraining noise is span corruption with sentinel tokens, a variant of text infilling that keeps every mask distinct. Each corrupted span is replaced by a unique sentinel (<X>, <Y>, <Z>, ...), and the target is not the whole document but only the dropped spans, each prefixed by its sentinel and closed by a final sentinel.

T5 span corruption. Two dropped spans become distinct sentinels in the input; the target lists only the missing spans, each tagged by its sentinel, then a closing sentinel.

The target is far shorter than the document, since the decoder never reproduces the kept tokens. Concretely, a $512$ -token input at a $15%$ corruption rate drops $0.15 \times 512 \approx 77$ tokens; at a mean span of $3$ that is about $77/3 \approx 26$ spans, so $26$ sentinels enter the input. The decoder target is those $77$ dropped tokens plus $26$ opening sentinels plus one closing sentinel, about $104$ positions, roughly a fifth of BART's full $512$ -token reconstruction. The saving is quadratic in effect: decoder self-attention costs $O (T^{2})$ , so a $5 \times$ shorter target is about $25 \times$ cheaper per step in the decoder's self-attention, which is why T5 can afford its enormous pretraining grid.¹

Cost per denoising step. BART reconstructs the full

512

-token document; T5 emits only the dropped spans plus sentinels (

\approx 104

tokens), a

\approx 5 \times

shorter decoder target.

This makes pretraining cheaper than BART's full reconstruction, at the cost of a less generative decoder objective: T5 never practices producing long, coherent runs of original text, so it relies more on fine-tuning for generation quality. T5's ablations found that this span-corruption objective, a corruption rate near $15%$ , and a mean span length near $3$ tokens are close to optimal across their grid.¹

PEGASUS and the gap-sentence objective

For one downstream task, abstractive summarization, the pretraining objective can be matched to the task even more tightly. PEGASUS masks not random spans but whole principal sentences, the sentences most representative of the document, and trains the decoder to generate them.² Reconstructing a removed sentence from the rest of the document is a close proxy for summarization: both require condensing the surrounding context into salient text.

Model	Encoder	Decoder	Pretraining corruption	Target
BERT³	bidirectional	none	token masking ( $15%$ )	masked tokens, in place
MASS⁴	bidirectional	autoregressive	mask one contiguous span	the masked span
BART⁵	bidirectional	autoregressive	infilling, deletion, permutation, rotation	the whole document
T5¹	bidirectional	autoregressive	span corruption with sentinels	the dropped spans
PEGASUS²	bidirectional	autoregressive	gap-sentence masking	the removed sentences

MASS sits between BERT and BART: it masks a single contiguous span in the encoder and trains the decoder to generate exactly that span, so the two halves attend to complementary tokens. BART generalizes this to arbitrary noise and full reconstruction; T5 specializes it back to short sentinel-tagged targets; PEGASUS tailors the masking to summarization.

Fine-tuning and decoding

After pretraining, the same encoder–decoder is fine-tuned on a labeled task by feeding the task input to the encoder and training the decoder to emit the target, under the ordinary cross-entropy loss. No new task-specific head is needed: the output is always text from the shared vocabulary.

Algorithm:

\textsc{FineTuneSeq2Seq}(p_\theta, \mathcal{D}, \eta)

— supervised seq2seq fine-tuning

1
initialize $\theta$ from the pretrained checkpoint
2
repeat
3
sample a batch $\braces{(s^{(i)}, y^{(i)})}$ of (source, target) pairs from $\mathcal{D}$
4
for each pair $(s, y)$ do
5
$c \gets \textsc{Encode}_\theta(s)$
bidirectional encoding of the source
6
for each position $t$ do
teacher forcing on the gold prefix
7
$p_t \gets p_\theta(\cdot \mid y_{<t}, c)$
next-token distribution
8
$L \gets -\frac{1}{\abs{\text{batch}}} \sum_i \sum_t \log p_\theta(y^{(i)}_t \mid y^{(i)}_{<t}, c^{(i)})$
9
$\theta \gets \theta - \eta\,\nabla_\theta L$
gradient step
10
until converged
11
return $\theta$

At inference the decoder generates autoregressively, and exact maximization of $p_{θ} (y ∣ s)$ over all sequences is intractable, so we approximate it with beam search: keep the $k$ highest-scoring partial sequences (beams) at each step and extend them. Raw log-probability favors short sequences, since every additional token multiplies in a probability $< 1$ , so a length penalty normalizes the score.

Larger $α$ rewards longer outputs, which matters for translation (where truncation drops content) and is dialed down for summarization (where brevity is the goal). The beam width $k$ trades quality for compute; widths of $4$ to $6$ are typical for both tasks.⁶

Task	Typical beam width $k$	Length penalty $α$	Why
Translation	$4$ – $5$	$0.6$ – $1.0$	avoid truncating content of the source
Summarization	$4$ – $6$	$\leq 1.0$ with a min-length floor	reward complete but concise output
Question answering	$1$ – $4$	small	answers are short; greedy often suffices

Training dynamics and defaults

Seq2seq fine-tuning inherits the pretrained encoder–decoder, so the loss starts far below a from-scratch model and the useful learning rates are small. BART and T5 fine-tune with AdamW at a peak learning rate near $1 \times 1 0^{- 4}$ to $3 \times 1 0^{- 5}$ , a short linear warmup of a few hundred steps, and linear decay to zero, for one to a few epochs on the labeled set. Label smoothing of $ε \approx 0.1$ is standard: it replaces the one-hot target with $(1 - ε)$ on the gold token and $ε / (∣ V ∣ - 1)$ spread over the rest, which caps the target logit gap and improves beam-search calibration. Dropout of $0.1$ on attention and feed-forward layers, plus early stopping on validation ROUGE or BLEU, keep the small fine-tuning sets from overfitting.

The per-token cross-entropy falls quickly because most of the language modeling was already learned in pretraining; what fine-tuning adapts is the mapping from the task's input distribution to its output distribution. A summarization run reaches most of its final ROUGE within the first epoch, and further training mainly sharpens length control and reduces repetition.

Exposure bias is the characteristic failure mode of autoregressive decoding, and it surfaces as three symptoms with standard remedies:

Repetition and loops. The decoder re-enters a high-probability n-gram and cycles (the the the, or a repeated clause). Blocking repeated n-grams during beam search (forbid any 3-gram from recurring) and a mild repetition penalty on already-emitted tokens break the loop.
Length collapse. Raw log-probability favors short outputs, so summaries end after one clause. The length penalty $lp (L)$ and a hard minimum-length floor counter it; too large an $α$ overshoots into padding and hallucinated filler.
Hallucination under distribution shift. When the source is longer or more technical than anything seen in fine-tuning, the decoder falls back on its language-model prior and invents fluent but unsupported content. Bidirectional source encoding mitigates this (every source token is fully in context), but it does not eliminate it; grounding checks and shorter beams help.

Why denoising seq2seq beats a decoder-only LM here

When the output is conditioned on a full input, summarize an article, translate a sentence, answer a question from a passage, a decoder-only language model and an encoder–decoder differ in one structural way: how the input is attended. The encoder–decoder reads the input bidirectionally; a decoder-only model reads it causally, through the same left-to-right mask it uses for generation.

Three consequences follow, and they are why the encoder–decoder design persists for conditional generation even as decoder-only models dominate open-ended generation.

Full-context source encoding. Each source token is encoded with both its left and right neighbors, so a word's representation reflects the whole sentence before any output token is produced.
Decoupled lengths. The encoder consumes a length- $∣ s ∣$ source and the decoder produces a length- $∣ y ∣$ target independently; a summary can be far shorter than its article without the model rebudgeting its causal context.
A denoising prior matched to transduction. Reconstructing a corrupted input during pretraining is itself a transduction task (input sequence to output sequence), so the pretrained weights transfer with little adaptation, while a decoder-only LM must repurpose a next-token objective.⁷

The decoder-only model is optimal for the unconditioned case, where there is no separate input to read bidirectionally and the only task is to continue text. The seq2seq advantage is specific to conditional generation, and it narrows as decoder-only models scale and learn to attend over long prompts. But for a fixed budget on translation, summarization, and grounded QA, the bidirectional encoder remains the stronger inductive bias.

Conditioning the output on the source. The encoder--decoder reads the source bidirectionally (blue); a decoder-only model reads it causally (red), losing right context.

What happened to the encoder--decoder

The theorem above is a statement about hypothesis classes at a fixed budget; the field's trajectory since concerns what happens when the budget is not fixed, and it complicates the comparison in three ways.

T5's successors kept the encoder--decoder. The strongest instruction-following variant of the T5 line, FLAN-T5, kept the encoder--decoder and fine-tuned it on a large mixture of instruction-phrased tasks, and it remains a strong, cheap baseline for exactly the transduction workloads (translation, summarization, closed-book QA) the theorem predicts it should win.⁸ For a team that wants a small model that reads a bounded input and writes a bounded output, the bidirectional encoder is still the right inductive bias.

Decoder-only models closed the gap by scaling the prompt. The theorem says a causal model represents source token $s_{i}$ using only $s_{\leq i}$ . That handicap shrinks as the model gets larger and its context window longer: a big decoder-only model attending over a long, in-context prompt learns to route information rightward well enough that the missing right-context becomes a second-order effect. This is why the frontier is decoder-only despite the encoder's cleaner conditioning — scale substitutes for the architectural prior, and one stack is simpler to train, serve, and pretrain on raw text.

Prefix-LM is the compromise. A prefix language model (used in UL2 and PaLM) runs one decoder-only stack but attends bidirectionally within a designated prefix and causally after it, recovering full-context source encoding without a separate encoder.⁹ It sits exactly between the two families the theorem contrasts: bidirectional over the source, causal over the generation. The cost is that the mask is now position-dependent, and the model must be told where the prefix ends, but it captures most of the encoder--decoder advantage inside a single stack.

In short, the theorem is correct but bounds a per-budget comparison. At the frontier, the simpler decoder-only stack wins on engineering grounds; at small and medium scale, where the budget is fixed and the task is genuine transduction, the encoder--decoder's stronger prior still pays.

Takeaways

T5 casts every task as text-to-text, selecting the task by a natural- language prefix, and pretrains with span corruption using unique sentinel tokens; the target is only the dropped spans, about a fifth the length of BART's full reconstruction, which is a roughly $25 \times$ saving in the decoder's $O (T^{2})$ self-attention.
PEGASUS masks whole principal sentences (gap-sentence generation) to match abstractive summarization; MASS masks a single contiguous span as a midpoint between BERT and BART. The pattern across BERT, MASS, BART, T5, and PEGASUS is one architecture with different corruptions.
Fine-tuning reuses the encoder–decoder with cross-entropy and no new head; decoding uses beam search with a length penalty $lp (L) = ((5 + L) /6)^{α}$ to counter the systematic bias toward short outputs.
Exposure bias — training on gold prefixes but decoding on the model's own — produces repetition/loops, length collapse, and hallucination under distribution shift; n-gram blocking, length floors, and full-context source encoding are the standard remedies.
When the output is conditioned on a full input, the encoder–decoder reads the source bidirectionally and strictly contains the decoder-only model's causal-conditioning family, which is why denoising seq2seq beats a decoder-only LM on translation, summarization, and grounded QA, while decoder-only large language models dominate open-ended generation.
Since then: the encoder--decoder prior still wins per budget on transduction (FLAN-T5), but at the frontier a decoder-only stack closes the gap by scaling context, and prefix-LM (UL2, PaLM) captures most of the encoder's advantage inside one stack with a position-dependent mask.

Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, 2020 — recasts every NLP task as text-to-text and pretrains with span corruption using sentinel tokens; a large systematic ablation of objectives. ↩ ↩² ↩³ ↩⁴
Zhang et al., PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization, 2020 — masks principal (ROUGE-salient) sentences and generates them, a pretraining objective matched to abstractive summarization. ↩ ↩²
Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019 — bidirectional encoder pretrained by masked language modeling. ↩
Song et al., MASS: Masked Sequence to Sequence Pre-training for Language Generation, 2019 — masks a single contiguous span in the encoder and reconstructs exactly that span in the decoder, midway between BERT and full seq2seq. ↩
Lewis et al., BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, 2020 — a standard Transformer encoder–decoder pretrained as a denoising autoencoder; the part-one model this lesson builds on. ↩
Chollet, Deep Learning with Python, Ch. 11 — decoding sequence models autoregressively, including beam search and the length normalization needed to avoid favoring short outputs. ↩
Chollet, Deep Learning with Python, Ch. 11 — sequence-to-sequence models with an encoder and a decoder, and why conditional generation tasks pair a bidirectional reader with an autoregressive writer. ↩
Chung et al., Scaling Instruction-Finetuned Language Models (FLAN-T5), 2022 — instruction-finetunes the T5 encoder--decoder on a large task mixture; a strong, cheap baseline for transduction workloads. ↩
Tay et al., UL2: Unifying Language Learning Paradigms, 2023 — a mixture-of-denoisers objective and a prefix-LM mode that is bidirectional in the prefix and causal after it, blending the encoder--decoder and decoder-only regimes in one stack. ↩