---
title: "Text-to-Text Transfer and Conditional Generation"
module: Large Models & Agents
moduleNumber: 10
lessonNumber: 4
order: 1004
summary: >
  BART reconstructs a corrupted document; T5 pushes the same denoising idea into a
  single interface where every task is a string-to-string map. This second part
  covers T5's span corruption with sentinel tokens (with a worked token budget),
  PEGASUS's summarization-matched gap sentences and the MASS midpoint, supervised
  fine-tuning and beam-search decoding with a length penalty, the exposure-bias
  failure modes of autoregressive decoding, and a theorem showing why a
  bidirectional encoder--decoder strictly dominates a decoder-only model when the
  output is conditioned on a full input.
topics: [Large Models & Agents]
sources:
  - book: Chollet
    ref: "Ch. 11 — Deep Learning for Text; sequence-to-sequence models"
  - book: Goodfellow
    ref: "Ch. 14 — Autoencoders; denoising autoencoders"
---

This builds on [Denoising Sequence-to-Sequence Pretraining: BART](/deep-learning/large-models-and-agents/seq2seq-pretraining-and-bart),
which trained a full encoder--decoder as a denoising autoencoder and showed BART
subsumes both BERT and GPT. BART reconstructs the _whole_ document from a corrupted
copy. This lesson specializes that idea in two directions — a shorter, sentinel-
tagged target (T5) and a summarization-matched masking (PEGASUS) — then follows the
pretrained model into fine-tuning, decoding, and the structural argument for why the
encoder--decoder design persists for conditional generation.

## T5: every task is text-to-text

**T5** (Text-to-Text Transfer Transformer) pushes the unification one step
further.[^t5] Every NLP problem, including ones that are not naturally generative,
is cast as mapping an input _string_ to an output _string_. Classification emits
the class _name_ as text; regression emits the number as a string; translation and
summarization are text-to-text already. One encoder–decoder, one cross-entropy
loss, one decoding procedure covers them all.

> **Definition (Text-to-text framework).** A formulation in which every task is a
> function from a text input to a text output. Task identity is carried by a
> natural-language **prefix** prepended to the input (e.g. `"translate English to
> German:"`), so a single model with shared weights performs every task, selected
> entirely by the prefix string.

| Task | Input string | Target string |
| --- | --- | --- |
| Translation | `translate English to German: the house is small` | `das Haus ist klein` |
| Summarization | `summarize: <article text> ...` | `<one-sentence summary>` |
| Classification (sentiment) | `sst2 sentence: a charming film` | `positive` |
| Entailment | `mnli premise: ... hypothesis: ...` | `entailment` |
| Similarity (regression) | `stsb sentence1: ... sentence2: ...` | `3.8` |

T5's pretraining noise is **span corruption** with **sentinel tokens**, a variant
of text infilling that keeps every mask distinct. Each corrupted span is replaced
by a _unique_ sentinel (`<X>`, `<Y>`, `<Z>`, ...), and the target is not the whole
document but only the dropped spans, each prefixed by its sentinel and closed by a
final sentinel.

> **Definition (Span corruption).** Sample a fraction (about $15\%$) of tokens to
> corrupt, group consecutive corrupted tokens into spans (mean length $\approx 3$),
> and replace each span in the input with a unique sentinel token. The target
> sequence is the concatenation of (sentinel, dropped span) pairs, terminated by a
> final sentinel. The decoder predicts only the missing spans, not the kept tokens.

$$
% caption: T5 span corruption. Two dropped spans become distinct sentinels in the input; the
% target lists only the missing spans, each tagged by its sentinel, then a closing sentinel.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  tok/.style={draw, minimum width=11mm, minimum height=7mm, align=center, fill=black!8},
  sx/.style={draw=acc, text=acc, thick, minimum width=11mm, minimum height=7mm, align=center, fill=acc!15},
  sy/.style={draw=green, text=green, thick, minimum width=11mm, minimum height=7mm, align=center, fill=green!15}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % original
  \node[anchor=east, font=\footnotesize] at (-0.7,3.0) {\texttt{original}};
  \node[tok] at (0,3.0) {thank};
  \node[tok, draw=acc, text=acc, fill=acc!15] at (1.3,3.0) {\texttt{you}};
  \node[tok, draw=acc, text=acc, fill=acc!15] at (2.6,3.0) {for};
  \node[tok] at (3.9,3.0) {\texttt{inviting}};
  \node[tok, draw=green, text=green, fill=green!15] at (5.2,3.0) {me};
  \node[tok] at (6.5,3.0) {to};
  \node[tok] at (7.8,3.0) {\texttt{party}};
  % input with sentinels
  \node[anchor=east, font=\footnotesize] at (-0.7,1.3) {\texttt{input}};
  \node[tok] at (0,1.3) {thank};
  \node[sx] at (1.3,1.3) {[X]};
  \node[tok] at (2.6,1.3) {\texttt{inviting}};
  \node[sy] at (3.9,1.3) {[Y]};
  \node[tok] at (5.2,1.3) {to};
  \node[tok] at (6.5,1.3) {\texttt{party}};
  % target
  \node[anchor=east, font=\footnotesize] at (-0.7,-0.4) {\texttt{target}};
  \node[sx] at (0,-0.4) {[X]};
  \node[tok, draw=acc, text=acc, fill=acc!15] at (1.3,-0.4) {\texttt{you}};
  \node[tok, draw=acc, text=acc, fill=acc!15] at (2.6,-0.4) {for};
  \node[sy] at (3.9,-0.4) {[Y]};
  \node[tok, draw=green, text=green, fill=green!15] at (5.2,-0.4) {me};
  \node[sx, draw=red, text=red, fill=red!15] at (6.5,-0.4) {[Z]};
  \node[red, anchor=west, font=\footnotesize] at (7.3,-0.4) {\texttt{end sentinel}};
  \draw[->, acc, thick] (1.3,2.55) -- (1.3,1.75);
  \draw[->, green, thick] (5.2,2.55) -- (3.9,1.75);
\end{tikzpicture}
$$

The target is far shorter than the document, since the decoder never reproduces
the kept tokens. Concretely, a $512$-token input at a $15\%$ corruption rate drops
$0.15 \times 512 \approx 77$ tokens; at a mean span of $3$ that is about
$77/3 \approx 26$ spans, so $26$ sentinels enter the input. The decoder target is
those $77$ dropped tokens plus $26$ opening sentinels plus one closing sentinel,
about $104$ positions, roughly a fifth of BART's full $512$-token reconstruction.
The saving is quadratic in effect: decoder self-attention costs $O(T^2)$, so a
$5\times$ shorter target is about $25\times$ cheaper per step in the decoder's
self-attention, which is why T5 can afford its enormous pretraining grid.[^t5]

$$
% caption: Cost per denoising step. BART reconstructs the full $512$-token document; T5 emits only the
% dropped spans plus sentinels ($\approx 104$ tokens), a $\approx 5\times$ shorter decoder target.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % BART bar
  \node[anchor=east, font=\footnotesize] at (-0.3,0.8) {\texttt{BART target}};
  \fill[acc!25, draw=acc] (0,0.5) rectangle (8.0,1.1);
  \node[acc, font=\footnotesize] at (4.0,0.8) {\texttt{512 tokens (full document)}};
  % T5 bar
  \node[anchor=east, font=\footnotesize] at (-0.3,-0.5) {\texttt{T5 target}};
  \fill[green!25, draw=green] (0,-0.8) rectangle (1.63,-0.2);
  \node[green, anchor=west, font=\footnotesize] at (1.8,-0.5) {\texttt{104 tokens (dropped spans + sentinels)}};
\end{tikzpicture}
$$

This makes pretraining cheaper than BART's full reconstruction, at the cost of a
less generative decoder objective: T5 never practices producing long, coherent
runs of original text, so it relies more on fine-tuning for generation quality.
T5's ablations found that this span-corruption objective, a corruption rate near
$15\%$, and a mean span length near $3$ tokens are close to optimal across their
grid.[^t5]

## PEGASUS and the gap-sentence objective

For one downstream task, abstractive **summarization**, the pretraining objective
can be matched to the task even more tightly. **PEGASUS** masks not random spans
but whole **principal sentences**, the sentences most representative of the
document, and trains the decoder to generate them.[^pegasus] Reconstructing a
removed sentence from the rest of the document is a close proxy for summarization:
both require condensing the surrounding context into salient text.

> **Definition (Gap-sentence generation).** Select the $m$ sentences whose
> removal leaves the document least self-contained, scored by ROUGE overlap with
> the remaining text, replace each selected sentence with a `[MASK]` token in the
> input, and train the decoder to emit the concatenation of the removed
> sentences. The pretext is a self-supervised analogue of extractive-to-abstractive
> summarization.

| Model | Encoder | Decoder | Pretraining corruption | Target |
| --- | --- | --- | --- | --- |
| BERT[^bert] | bidirectional | none | token masking ($15\%$) | masked tokens, in place |
| MASS[^mass] | bidirectional | autoregressive | mask one contiguous span | the masked span |
| BART[^bart] | bidirectional | autoregressive | infilling, deletion, permutation, rotation | the whole document |
| T5[^t5] | bidirectional | autoregressive | span corruption with sentinels | the dropped spans |
| PEGASUS[^pegasus] | bidirectional | autoregressive | gap-sentence masking | the removed sentences |

MASS sits between BERT and BART: it masks a single contiguous span in the encoder
and trains the decoder to generate exactly that span, so the two halves attend to
complementary tokens. BART generalizes this to arbitrary noise and full
reconstruction; T5 specializes it back to short sentinel-tagged targets; PEGASUS
tailors the masking to summarization.

## Fine-tuning and decoding

After pretraining, the same encoder–decoder is fine-tuned on a labeled task by
feeding the task input to the encoder and training the decoder to emit the target,
under the ordinary cross-entropy loss. No new task-specific head is needed: the
output is always text from the shared vocabulary.

```algorithm
caption: $\textsc{FineTuneSeq2Seq}(p_\theta, \mathcal{D}, \eta)$ — supervised seq2seq fine-tuning
initialize $\theta$ from the pretrained checkpoint
repeat
  sample a batch $\braces{(s^{(i)}, y^{(i)})}$ of (source, target) pairs from $\mathcal{D}$
  for each pair $(s, y)$ do
    $c \gets \textsc{Encode}_\theta(s)$ // bidirectional encoding of the source
    for each position $t$ do // teacher forcing on the gold prefix
      $p_t \gets p_\theta(\cdot \mid y_{<t}, c)$ // next-token distribution
  $L \gets -\frac{1}{\abs{\text{batch}}} \sum_i \sum_t \log p_\theta(y^{(i)}_t \mid y^{(i)}_{<t}, c^{(i)})$
  $\theta \gets \theta - \eta\,\nabla_\theta L$ // gradient step
until converged
return $\theta$
```

At inference the decoder generates autoregressively, and exact maximization of
$p_\theta(y \mid s)$ over all sequences is intractable, so we approximate it with
**beam search**: keep the $k$ highest-scoring partial sequences (beams) at each
step and extend them. Raw log-probability favors short sequences, since every
additional token multiplies in a probability $< 1$, so a **length penalty**
normalizes the score.

> **Definition (Beam search with length penalty).** Score a candidate output
> $y = (y_1, \dots, y_L)$ by
> $$
> s(y \mid x) = \frac{1}{\text{lp}(L)} \sum_{t=1}^{L} \log p_\theta\parens{y_t \mid y_{<t}, x},
> \qquad
> \text{lp}(L) = \parens{\frac{5 + L}{5 + 1}}^{\alpha},
> $$
> with $\alpha \in [0, 1]$ a tunable exponent ($\alpha = 0$ recovers raw
> log-probability). Beam search keeps the $k$ highest-scoring prefixes at each
> step; the penalty $\text{lp}(L)$ divides out the systematic bias against long
> sequences.

Larger $\alpha$ rewards longer outputs, which matters for translation (where
truncation drops content) and is dialed down for summarization (where brevity is
the goal). The beam width $k$ trades quality for compute; widths of $4$ to $6$ are
typical for both tasks.[^chollet-decode]

| Task | Typical beam width $k$ | Length penalty $\alpha$ | Why |
| --- | --- | --- | --- |
| Translation | $4$–$5$ | $0.6$–$1.0$ | avoid truncating content of the source |
| Summarization | $4$–$6$ | $\le 1.0$ with a min-length floor | reward complete but concise output |
| Question answering | $1$–$4$ | small | answers are short; greedy often suffices |

### Training dynamics and defaults

Seq2seq fine-tuning inherits the pretrained encoder–decoder, so the loss starts
far below a from-scratch model and the useful learning rates are small. BART and
T5 fine-tune with AdamW at a peak learning rate near $1\times10^{-4}$ to
$3\times10^{-5}$, a short linear warmup of a few hundred steps, and linear decay to
zero, for one to a few epochs on the labeled set. Label smoothing of $\varepsilon
\approx 0.1$ is standard: it replaces the one-hot target with $(1-\varepsilon)$ on
the gold token and $\varepsilon/(|V|-1)$ spread over the rest, which caps the
target logit gap and improves beam-search calibration. Dropout of $0.1$ on
attention and feed-forward layers, plus early stopping on validation ROUGE or BLEU,
keep the small fine-tuning sets from overfitting.

The per-token cross-entropy falls quickly because most of the language modeling was
already learned in pretraining; what fine-tuning adapts is the mapping from the
task's input distribution to its output distribution. A summarization run reaches
most of its final ROUGE within the first epoch, and further training mainly sharpens
length control and reduces repetition.

> **Definition (Exposure bias).** The mismatch between teacher forcing at training
> time, where the decoder always conditions on the _gold_ prefix $y_{<t}$, and
> inference, where it conditions on its _own_ generated prefix $\hat y_{<t}$. A
> single wrong token at step $t$ shifts every later distribution off the training
> manifold, so errors compound left to right.

Exposure bias is the characteristic failure mode of autoregressive decoding, and it
surfaces as three symptoms with standard remedies:

- **Repetition and loops.** The decoder re-enters a high-probability n-gram and
  cycles (`the the the`, or a repeated clause). Blocking repeated n-grams during
  beam search (forbid any 3-gram from recurring) and a mild repetition penalty on
  already-emitted tokens break the loop.
- **Length collapse.** Raw log-probability favors short outputs, so summaries end
  after one clause. The length penalty $\text{lp}(L)$ and a hard minimum-length
  floor counter it; too large an $\alpha$ overshoots into padding and hallucinated
  filler.
- **Hallucination under distribution shift.** When the source is longer or more
  technical than anything seen in fine-tuning, the decoder falls back on its
  language-model prior and invents fluent but unsupported content. Bidirectional
  source encoding mitigates this (every source token is fully in context), but it
  does not eliminate it; grounding checks and shorter beams help.

## Why denoising seq2seq beats a decoder-only LM here

When the output is conditioned on a _full_ input, summarize an article, translate
a sentence, answer a question from a passage, a decoder-only language model and an
encoder–decoder differ in one structural way: **how the input is attended**. The
encoder–decoder reads the input **bidirectionally**; a decoder-only model reads it
**causally**, through the same left-to-right mask it uses for generation.

> **Theorem (Bidirectional conditioning dominates causal conditioning).** Let the
> task be to model $p(y \mid s)$ for a fixed source $s$ and a generated target
> $y$. An encoder–decoder represents each source position $s_i$ using all of $s$,
> whereas a decoder-only model that concatenates $s \mathbin{\Vert} y$ under a
> causal mask represents $s_i$ using only $s_{\le i}$. The encoder–decoder's
> hypothesis class therefore contains the decoder-only model's as the special
> case where every source representation ignores its right context.

> **Proof.** In the concatenated decoder-only model, the causal mask forces the
> representation $h_i$ of source token $s_i$ to be a function of $s_1, \dots, s_i$
> only. In the encoder–decoder, the encoder is unmasked, so $h_i$ is a function of
> all of $s_1, \dots, s_{\abs{s}}$. Any causal source encoding is realizable by an
> unmasked encoder that happens to zero out attention to the right, so the
> causal-conditioning family is a subset of the bidirectional-conditioning family.
> When the target depends on source tokens that appear _after_ position $i$, which
> is generic for summarization and translation, only the bidirectional encoder can
> represent the dependency, so its optimum is no worse and generically strictly
> better. $\qed$

Three consequences follow, and they are why the encoder–decoder design persists
for conditional generation even as decoder-only models dominate open-ended
generation.

- **Full-context source encoding.** Each source token is encoded with both its
  left and right neighbors, so a word's representation reflects the whole sentence
  before any output token is produced.
- **Decoupled lengths.** The encoder consumes a length-$\abs{s}$ source and the
  decoder produces a length-$\abs{y}$ target independently; a summary can be far
  shorter than its article without the model rebudgeting its causal context.
- **A denoising prior matched to transduction.** Reconstructing a corrupted input
  during pretraining is itself a transduction task (input sequence to output
  sequence), so the pretrained weights transfer with little adaptation, while a
  decoder-only LM must repurpose a next-token objective.[^chollet-seq2seq]

The decoder-only model is optimal for the unconditioned case, where there is no
separate input to read bidirectionally and the only task is to continue text. The seq2seq advantage is specific to conditional generation, and it
narrows as decoder-only models scale and learn to attend over long prompts. But for
a fixed budget on translation, summarization, and grounded QA, the bidirectional
encoder remains the stronger inductive bias.

$$
% caption: Conditioning the output on the source. The encoder--decoder reads the source
% bidirectionally (blue); a decoder-only model reads it causally (red), losing right context.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  s/.style={draw, circle, minimum size=6mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % encoder-decoder: bidirectional source
  \node[anchor=south, font=\footnotesize] at (1.8,1.5) {\texttt{encoder + decoder (bidirectional source)}};
  \foreach \i/\x in {1/0,2/1.2,3/2.4,4/3.6} \node[s, draw=acc, text=acc] (a\i) at (\x,0) {s\i};
  \draw[<->, acc, thick] (a1) to[bend left=40] (a2);
  \draw[<->, acc, thick] (a2) to[bend left=40] (a3);
  \draw[<->, acc, thick] (a3) to[bend left=40] (a4);
  \draw[<->, acc, thick] (a1) to[bend right=42] (a3);
  \draw[<->, acc, thick] (a2) to[bend right=42] (a4);
  % decoder-only: causal source
  \begin{scope}[yshift=-3.2cm]
  \node[anchor=south, font=\footnotesize] at (1.8,1.5) {\texttt{decoder-only (causal source)}};
  \foreach \i/\x in {1/0,2/1.2,3/2.4,4/3.6} \node[s, draw=red, text=red] (b\i) at (\x,0) {s\i};
  \draw[->, red, thick] (b1) to[bend left=40] (b2);
  \draw[->, red, thick] (b2) to[bend left=40] (b3);
  \draw[->, red, thick] (b3) to[bend left=40] (b4);
  \draw[->, red, thick] (b1) to[bend left=46] (b3);
  \draw[->, red, thick] (b1) to[bend left=50] (b4);
  \end{scope}
\end{tikzpicture}
$$

## What happened to the encoder--decoder

The theorem above is a statement about hypothesis classes at a _fixed_ budget; the field's trajectory since concerns what happens when the budget is not fixed, and it complicates the comparison in three ways.

**T5's successors kept the encoder--decoder.** The strongest instruction-following variant of the T5 line, **FLAN-T5**, kept the encoder--decoder and fine-tuned it on a large mixture of instruction-phrased tasks, and it remains a strong, cheap baseline for exactly the transduction workloads (translation, summarization, closed-book QA) the theorem predicts it should win.[^chung] For a team that wants a small model that reads a bounded input and writes a bounded output, the bidirectional encoder is still the right inductive bias.

**Decoder-only models closed the gap by scaling the prompt.** The theorem says a causal model represents source token $s_i$ using only $s_{\le i}$. That handicap shrinks as the model gets larger and its context window longer: a big decoder-only model attending over a long, in-context prompt learns to route information rightward well enough that the missing right-context becomes a second-order effect. This is why the frontier is decoder-only despite the encoder's cleaner conditioning — scale substitutes for the architectural prior, and one stack is simpler to train, serve, and pretrain on raw text.

**Prefix-LM is the compromise.** A **prefix language model** (used in UL2 and PaLM) runs one decoder-only stack but attends bidirectionally within a designated prefix and causally after it, recovering full-context source encoding without a separate encoder.[^tay-ul2] It sits exactly between the two families the theorem contrasts: bidirectional over the source, causal over the generation. The cost is that the mask is now position-dependent, and the model must be told where the prefix ends, but it captures most of the encoder--decoder advantage inside a single stack.

In short, the theorem is correct but bounds a _per-budget_ comparison. At the frontier, the simpler decoder-only stack wins on engineering grounds; at small and medium scale, where the budget is fixed and the task is genuine transduction, the encoder--decoder's stronger prior still pays.

## Takeaways

- **T5** casts every task as **text-to-text**, selecting the task by a natural-
  language prefix, and pretrains with **span corruption** using unique **sentinel
  tokens**; the target is only the dropped spans, about a fifth the length of BART's
  full reconstruction, which is a roughly $25\times$ saving in the decoder's
  $O(T^2)$ self-attention.
- **PEGASUS** masks whole principal sentences (**gap-sentence generation**) to match
  abstractive summarization; **MASS** masks a single contiguous span as a midpoint
  between BERT and BART. The pattern across BERT, MASS, BART, T5, and PEGASUS is one
  architecture with different corruptions.
- **Fine-tuning** reuses the encoder–decoder with cross-entropy and no new head;
  **decoding** uses **beam search** with a **length penalty**
  $\text{lp}(L) = \parens{(5+L)/6}^{\alpha}$ to counter the systematic bias toward
  short outputs.
- **Exposure bias** — training on gold prefixes but decoding on the model's own —
  produces repetition/loops, length collapse, and hallucination under distribution
  shift; n-gram blocking, length floors, and full-context source encoding are the
  standard remedies.
- When the output is conditioned on a **full input**, the encoder–decoder reads the
  source **bidirectionally** and strictly contains the decoder-only model's
  causal-conditioning family, which is why **denoising seq2seq beats a decoder-only
  LM** on translation, summarization, and grounded QA, while decoder-only
  [large language models](/deep-learning/large-models-and-agents/large-language-models)
  dominate open-ended generation.
- **Since then:** the encoder--decoder prior still wins _per budget_ on
  transduction (FLAN-T5), but at the frontier a decoder-only stack closes the gap by
  scaling context, and **prefix-LM** (UL2, PaLM) captures most of the encoder's
  advantage inside one stack with a position-dependent mask.

[^chung]: **Chung et al.**, _Scaling Instruction-Finetuned Language Models_ (FLAN-T5), 2022 — instruction-finetunes the T5 encoder--decoder on a large task mixture; a strong, cheap baseline for transduction workloads.
[^tay-ul2]: **Tay et al.**, _UL2: Unifying Language Learning Paradigms_, 2023 — a mixture-of-denoisers objective and a prefix-LM mode that is bidirectional in the prefix and causal after it, blending the encoder--decoder and decoder-only regimes in one stack.
[^t5]: **Raffel et al.**, _Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer_, 2020 — recasts every NLP task as text-to-text and pretrains with span corruption using sentinel tokens; a large systematic ablation of objectives.
[^pegasus]: **Zhang et al.**, _PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization_, 2020 — masks principal (ROUGE-salient) sentences and generates them, a pretraining objective matched to abstractive summarization.
[^mass]: **Song et al.**, _MASS: Masked Sequence to Sequence Pre-training for Language Generation_, 2019 — masks a single contiguous span in the encoder and reconstructs exactly that span in the decoder, midway between BERT and full seq2seq.
[^bart]: **Lewis et al.**, _BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension_, 2020 — a standard Transformer encoder–decoder pretrained as a denoising autoencoder; the part-one model this lesson builds on.
[^bert]: **Devlin et al.**, _BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding_, 2019 — bidirectional encoder pretrained by masked language modeling.
[^chollet-seq2seq]: **Chollet**, _Deep Learning with Python_, Ch. 11 — sequence-to-sequence models with an encoder and a decoder, and why conditional generation tasks pair a bidirectional reader with an autoregressive writer.
[^chollet-decode]: **Chollet**, _Deep Learning with Python_, Ch. 11 — decoding sequence models autoregressively, including beam search and the length normalization needed to avoid favoring short outputs.