--- title: "Abstractive Summarization and Evaluation" module: Applications moduleNumber: 7 lessonNumber: 8 order: 708 summary: > Extractive methods can only reuse the source's own sentences; to compress within a sentence or paraphrase, a summarizer has to generate. This part covers abstractive summarization: the sequence-to-sequence approach, the pointer-generator's copy switch and coverage mechanism, pretrained summarizers (BART, PEGASUS) and zero-shot LLM prompting, the long-document and factuality problems, and ROUGE evaluation with a worked example and its limits — closing on the abstractive lineage from See 2017 through faithfulness metrics. topics: [Applications] sources: - book: Jurafsky ref: "§9.9 Contextual Generation and Summarization (abstractive)" - book: Jurafsky ref: "§9.9.1 Applying Transformers to other NLP tasks; Ch. 10 §10.8 (overlap-metric methodology)" --- This builds on [Text Summarization](/natural-language-processing/applications/text-summarization), which defined the task and its flavors and worked through **extractive** summarization — scoring sentences by position and centrality, ranking them with TextRank/LexRank PageRank, and learning the selection. Extraction is safe but can only reuse the source's own sentences. This part takes up the other family, **abstractive** summarization, which generates fresh text, and the evaluation that tells whether either kind of summary is any good. ## Abstractive summarization Extractive methods can never say anything the document did not already say in one of its sentences. To compress within a sentence, fuse two sentences into one, or paraphrase, the summarizer has to _generate_ — the job an [encoder-decoder](/natural-language-processing/applications/machine-translation) does: read the source, then produce a fresh string one token at a time. ### The sequence-to-sequence approach Summarization is conditional generation, and J&M cast it as ordinary language modeling. Take a corpus of article–summary pairs — the standard one is **CNN/DailyMail**, news stories each paired with their human-written bullet highlights — and for each pair $(x_1, \ldots, x_m)$, $(y_1, \ldots, y_n)$ concatenate them into one long training sequence with a special separator token $\delta$ between article and summary: $$ (x_1, \ldots, x_m,\; \delta,\; y_1, \ldots, y_n), \qquad \text{total length } m + n + 1. $$ Train an autoregressive language model on these long sequences by [teacher forcing](/natural-language-processing/applications/machine-translation), exactly as any language model is trained to predict the next word.[^jm-ctxgen] At inference, feed a new article ending in $\delta$ as the priming **context** and let the model generate the summary token by token until it emits an end-of-sequence marker. Because a [transformer](/natural-language-processing/transformers/transformers-and-attention) attends over its whole input, the model has direct access to the entire article _and_ to everything it has generated so far at every step — the property that makes context-based generation work. $$ % caption: Summarization as context-primed autoregressive generation. The article % x1..xm, ending in the separator delta, primes the model; it then generates the % summary y1..yn one token at a time, each token conditioned on the full article % and the summary tokens produced so far. Training concatenates article and % summary into one sequence and uses teacher forcing. \begin{tikzpicture}[>=stealth, font=\footnotesize, tok/.style={draw, minimum width=10mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}, gen/.style={draw=acc, text=acc, fill=acc!8, minimum width=10mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}, sep/.style={draw=red, text=red, minimum width=8mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[tok] (x1) at (0,0) {x1}; \node[tok] (x2) at (1.2,0) {x2}; \node[font=\scriptsize] (xd) at (2.4,0) {. . .}; \node[tok] (xm) at (3.6,0) {xm}; \node[sep] (d) at (4.8,0) {delta}; \node[gen] (y1) at (6.0,0) {y1}; \node[gen] (y2) at (7.2,0) {y2}; \node[font=\scriptsize, text=acc] (yd) at (8.4,0) {. . .}; \node[gen] (yn) at (9.6,0) {yn}; \node[anchor=north, font=\scriptsize, text=black] at (1.8,-0.55) {article (priming context)}; \node[anchor=north, font=\scriptsize, text=acc] at (7.8,-0.55) {generated summary}; % autoregressive generation arrows \draw[->, acc] (d) -- (y1); \draw[->, acc] (y1) -- (y2); \draw[->, acc] (yd) -- (yn); \end{tikzpicture} $$ This scheme is the basis not just for summarization but for the whole family of text-to-text tasks — translation, question answering, summarization — that share the encoder-decoder shape.[^jm-ctxgen] ### The pointer-generator: copying and coverage Plain sequence-to-sequence summarizers have two chronic faults. First, they **cannot reliably reproduce facts** copied from the source — a name, a number, a rare technical term the softmax has barely seen — because generating it requires the exact token to win over the entire vocabulary. Second, they **repeat themselves**, looping on a phrase because nothing tracks what has already been said. The **pointer-generator network** fixes both.[^jm-ptrgen] The copy fix is a soft switch. At each decoder step the model computes a **generation probability** $p_{\text{gen}} \in [0,1]$ — how much to trust the vocabulary softmax versus how much to _copy_ from the source — and forms the final next-word distribution as a mixture of generating a vocabulary word and pointing at a source word via the attention weights $\alpha$: $$ P(w) \;=\; p_{\text{gen}}\,P_{\text{vocab}}(w) \;+\; (1 - p_{\text{gen}}) \sum_{j:\, x_j = w} \alpha_{j}. $$ When $p_{\text{gen}} \to 0$ the model _points_ — it copies the source word its attention is focused on, so a name or number the vocabulary softmax could never have produced is reproduced exactly. When $p_{\text{gen}} \to 1$ it _generates_ a paraphrase from the vocabulary. The single scalar lets the model copy and paraphrase within one sentence, which is the essence of abstractive compression. $$ % caption: The pointer-generator switch. At each step the model computes p_gen: it % either generates a word from the vocabulary distribution (p_gen high) or copies % a source word by pointing through the attention weights (p_gen low). The final % distribution is their mixture, so rare names and numbers can be copied exactly. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=26mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[box, draw=acc, text=acc] (sw) at (0,0) {p-gen switch}; \node[box] (gen) at (4.4,1.1) {generate from\\vocabulary}; \node[box] (cpy) at (4.4,-1.1) {copy from source\\(via attention)}; \node[box] (mix) at (8.8,0) {f\/inal next-word\\distribution}; \draw[->, acc, thick] (sw) -- (gen) node[midway, above, font=\scriptsize] {p-gen high}; \draw[->, acc, thick] (sw) -- (cpy) node[midway, below, font=\scriptsize] {p-gen low}; \draw[->, black] (gen) -- (mix); \draw[->, black] (cpy) -- (mix); \end{tikzpicture} $$ The repetition fix is **coverage**. Keep a running **coverage vector** $c_t$ that accumulates the attention each source word has received over all previous decoder steps, $c_t = \sum_{t' < t} \alpha^{t'}$, and add a penalty to the loss whenever a new step attends _again_ to a source word that has already been heavily attended. The coverage vector records which source words have already been summarized, and penalizing re-attention stops the decoder from generating the same clause twice.[^jm-ptrgen] ### Pretrained summarizers and prompting an LLM The pointer-generator was trained from scratch on the summarization corpus. The modern approach instead **pretrains** a large encoder-decoder on generic text with a self-supervised objective, then **fine-tunes** it on the summarization corpus — the same pretrain-then-finetune recipe J&M describe for applying transformers to downstream tasks.[^jm-pretrain] Two pretraining objectives are purpose-built to teach a model to reconstruct and compress text before it ever sees a summary. $$ % caption: Two denoising pretraining objectives for summarization. BART-style % corrupts the input (masking spans, shuffling, deleting) and trains the decoder % to reconstruct the original text. PEGASUS-style masks whole important % sentences (gap-sentence generation) and trains the model to regenerate them, % which mimics writing a summary. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=34mm, minimum height=9mm, align=center, font=\scriptsize}, outb/.style={draw=acc, text=acc, minimum width=34mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % BART row \node[box] (bin) at (0,1.4) {corrupt text\\(mask spans, reorder)}; \node[outb] (bout) at (5.4,1.4) {reconstruct the\\original text}; \draw[->, acc, thick] (bin) -- (bout) node[midway, above, font=\scriptsize] {BART-style}; % PEGASUS row \node[box] (pin) at (0,-1.0) {mask whole\\important sentences}; \node[outb] (pout) at (5.4,-1.0) {regenerate the\\masked sentences}; \draw[->, acc, thick] (pin) -- (pout) node[midway, above, font=\scriptsize] {PEGASUS-style}; \end{tikzpicture} $$ A **BART**-style model corrupts its input — masking spans of tokens, deleting tokens, shuffling sentence order — and trains the decoder to reconstruct the clean original. This makes it a general text **denoiser**, a strong starting point for any generation task. A **PEGASUS**-style model targets summarization directly with **gap-sentence generation**: it removes the document's most important whole sentences and trains the model to regenerate them from the rest, so the pretraining task already _is_ a kind of summarization. Fine-tuned on CNN/DailyMail, both far surpass a from-scratch pointer-generator, because they already model fluent, coherent text before fine-tuning begins. The most recent shift removes fine-tuning entirely. A sufficiently large [language model](/natural-language-processing/transformers/large-language-models) can summarize **zero-shot**: append an instruction to the document — literally `Summarize the above article in three sentences:` — and let the model generate, with no gradient update and no summarization-specific training. A few demonstrations of article-then-summary pairs in the prompt (**few-shot**) sharpens length and style further. Prompted summarization is just contextual generation where the context is an instruction; it is the same mechanism as the append-marker scheme, with a natural-language instruction playing the role of the separator $\delta$.[^jm-pretrain] $$ % caption: Prompting an LLM to summarize, zero-shot. The document and a % natural-language instruction form the priming context; the model generates the % summary autoregressively with no fine-tuning. The instruction plays the role of % the separator delta in the append-marker scheme. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=30mm, minimum height=10mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[box] (doc) at (0,0) {document text}; \node[box] (instr) at (0,-1.6) {instruction:\\"Summarize the above"}; \node[box, draw=acc, text=acc] (llm) at (4.6,-0.8) {LLM}; \node[box] (sum) at (9.2,-0.8) {generated\\summary}; \draw[->, black] (doc) -- (llm); \draw[->, black] (instr) -- (llm); \draw[->, acc, thick] (llm) -- (sum); \end{tikzpicture} $$ ## Long documents and factuality Two problems dog abstractive summarization in practice, and both get worse the longer the source. **Long input.** A transformer's self-attention costs grow quadratically with input length, and every model has a fixed **context window**, so a book, a court filing, or a long transcript will not fit. Three responses: | Strategy | Idea | Cost | | --- | --- | --- | | Truncation | keep the first $k$ tokens, drop the rest | loses everything after the cutoff | | Hierarchical | summarize each chunk, then summarize the summaries | multi-pass; may lose cross-chunk links | | Long-context | a model with an enlarged / sparse-attention window | expensive; still bounded | Truncation is the crude default and is often adequate for news (whose important material is front-loaded, the same position signal the extractive methods exploit). Hierarchical summarization — split into sections, summarize each, then summarize the section-summaries — scales to arbitrary length but can drop connections that span chunks. Purpose-built long-context models push the window out but never make it infinite. **Factuality.** The signature failure of abstractive summarization is **hallucination**: a fluent summary that states something the source does not support, or contradicts. Because the model generates rather than copies, nothing guarantees its output stays faithful to the input — it can invent a number, misattribute a quote, or merge two entities into one. > **Definition (Faithfulness / factuality).** A summary is **faithful** if every > claim it makes is entailed by the source document. Abstractive systems can be > fluent and _un_faithful at once; extractive systems are faithful by construction > because they only copy, which is a large part of why they remain in use. Faithfulness is precisely what overlap-based evaluation (next) fails to measure — a hallucinated summary can share plenty of n-grams with the reference and still be wrong — so factuality is usually assessed separately, by entailment models that check whether the source entails each summary sentence, or by human judgment. ## Evaluation: ROUGE Judging a summary means comparing it against one or more human **reference** summaries. The standard automatic metric is **ROUGE** (Recall-Oriented Understudy for Gisting Evaluation), the summarization counterpart of the [BLEU](/natural-language-processing/applications/machine-translation) metric used for translation. Where BLEU is precision-oriented (what fraction of the _candidate_'s n-grams are in the reference), ROUGE is **recall-oriented** (what fraction of the _reference_'s n-grams the candidate recovered) — the natural emphasis for summarization, where the worry is leaving important content _out_.[^jm-rouge] > **Definition (ROUGE-N).** The recall of reference n-grams: of all the n-grams in > the reference summary, the fraction that also appear in the candidate, > $$ > \text{ROUGE-N} = \frac{\sum_{g \,\in\, \text{ref n-grams}} \text{count}_{\text{match}}(g)}{\sum_{g \,\in\, \text{ref n-grams}} \text{count}(g)}, > $$ > where $\text{count}_{\text{match}}(g)$ is the number of times n-gram $g$ appears > in both. ROUGE-1 uses unigrams (content overlap), ROUGE-2 uses bigrams (a proxy > for fluency and word order). > **Definition (ROUGE-L).** Overlap measured by the **longest common subsequence** > (LCS) of candidate and reference. The LCS need not be contiguous, so it rewards > in-order word matches without demanding an exact run, capturing sentence-level > structure more forgivingly than a fixed n-gram. Reported as an F-measure over > LCS-based precision and recall. Work one example. Take a reference and a candidate summary sentence: > Reference: _the man ships Boston snow to warm states_ > Candidate: _the man ships Boston snow online_ $$ % caption: ROUGE on the worked example. Of the reference's 8 unigrams, 5 appear in % the candidate (the, man, ships, Boston, snow), so ROUGE-1 recall = 5/8. Of its 7 % bigrams, 4 match, so ROUGE-2 = 4/7. The longest common subsequence is "the man % ships Boston snow" (length 5), giving ROUGE-L. ROUGE counts what the candidate % recovered, not what it added. \begin{tikzpicture}[>=stealth, font=\footnotesize, b/.style={draw, minimum width=26mm, minimum height=9mm, align=center, font=\scriptsize}, resb/.style={draw=acc, text=acc, minimum width=30mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[b] (r1) at (0,1.6) {ROUGE-1 recall\\5/8 = 0.63}; \node[b] (r2) at (0,0) {ROUGE-2 recall\\4/7 = 0.57}; \node[b] (rl) at (0,-1.6) {ROUGE-L (LCS)\\length 5}; \node[resb] (res) at (4.8,0) {overlap with\\reference}; \draw[->, black] (r1) -- (res); \draw[->, black] (r2) -- (res); \draw[->, black] (rl) -- (res); \end{tikzpicture} $$ Carry the numbers through. The reference has $8$ unigrams; the candidate shares five of them (_the, man, ships, Boston, snow_) and misses three (_to, warm, states_), so **ROUGE-1** recall is $5/8 = 0.63$. The reference's $7$ bigrams are _the man, man ships, ships Boston, Boston snow, snow to, to warm, warm states_; the candidate reproduces the first four, so **ROUGE-2** recall is $4/7 \approx 0.57$. The longest common subsequence is _the man ships Boston snow_, length $5$, which drives **ROUGE-L**. The candidate's extra word _online_ costs nothing in recall (recall only measures how much of the reference was recovered), which is why full ROUGE reports precision and F-measures too, so padding a summary with unmatched words is penalized.[^jm-rouge] Carrying the LCS through to an F-measure shows the precision side at work. The candidate has $6$ words, the reference $8$, and the LCS is length $5$. Then LCS-precision is $P_{\text{lcs}} = 5/6 \approx 0.83$ (five of the candidate's six words lie on the common subsequence) and LCS-recall is $R_{\text{lcs}} = 5/8 = 0.63$. The ROUGE-L F-measure combines them, $$ F_{\text{lcs}} \;=\; \frac{(1+\beta^2)\,P_{\text{lcs}}\,R_{\text{lcs}}}{R_{\text{lcs}} + \beta^2\,P_{\text{lcs}}} \;\xrightarrow{\;\beta=1\;}\; \frac{2\,(0.83)(0.63)}{0.83 + 0.63} \;=\; \frac{1.05}{1.46} \;\approx\; 0.72, $$ so the balanced score sits between the two, and a candidate that recovered the same content but ran on for twenty extra words would keep the recall of $0.63$ while its precision — and therefore its $F_{\text{lcs}}$ — dropped. That is the mechanism the recall-only ROUGE-1 figure above hides: recall alone rewards saying more, and only the precision term penalizes it. ### The limits of ROUGE ROUGE inherits every weakness of surface overlap, and adds one of its own. - It is **lexical**: a correct paraphrase that uses different words scores zero overlap, so an abstractive summary is systematically undercredited relative to an extractive one that copies the reference's exact words. - It is **local**: it counts n-grams, so it barely notices whether the summary is globally coherent or self-contradictory. - Most seriously, it **does not measure faithfulness**. A hallucinated summary can share many n-grams with the reference and still assert something false; ROUGE cannot tell the difference between a faithful summary and a fluent fabrication with the same words. For these reasons ROUGE, like BLEU, is best used to track changes to a **single** system during development, and is least trustworthy when comparing very different systems. Serious evaluation pairs it with **human judgment** of the two axes that matter — is the summary _coherent_ and does it cover the important content? — and, increasingly, with dedicated **factuality** checks: entailment models, or an [LLM](/natural-language-processing/transformers/large-language-models) asked to verify each summary sentence against the source. Embedding-based metrics such as BERTScore (introduced for [translation evaluation](/natural-language-processing/applications/machine-translation)) also transfer to summarization, crediting paraphrase where ROUGE cannot.[^jm-rouge] ## The abstractive-summarization lineage Jurafsky & Martin present the pointer-generator, denoising pretraining, and prompting as a sequence of ideas. The ideas came from specific systems: the field moved through them in a clear order, and each paper isolated one problem. **Copying and coverage (See et al., 2017).** The pointer-generator network the lesson describes is due to See, Liu, and Manning, _Get To The Point: Summarization with Pointer-Generator Networks_ (ACL 2017). Their model was an LSTM encoder-decoder with attention, and the two additions match the two faults exactly: the $p_{\text{gen}}$ soft switch to copy source tokens, and the coverage vector plus coverage loss to stop repetition. On CNN/DailyMail they reported ROUGE-1 near $39$ and ROUGE-L near $36$, beating the abstractive baselines of the day, and they showed that the copy mechanism cut the rate of factually garbled rare words. The paper also documented the failure mode that motivated everything after it: even with copying, the model still occasionally produced fluent sentences unsupported by the source.[^ptrgen-see] **Pretraining takes over (Lewis et al., 2020; Zhang et al., 2020).** Two 2020 papers replaced the from-scratch encoder-decoder with a pretrained one, the split the lesson draws as BART-style versus PEGASUS-style. **BART** (Lewis et al., _BART: Denoising Sequence-to-Sequence Pre-training_, ACL 2020) pretrains a standard Transformer encoder-decoder as a denoiser: corrupt the input with several noise functions — token masking, token deletion, text-span infilling, sentence permutation, document rotation — and train the decoder to reconstruct the clean text. Fine-tuned on CNN/DailyMail it set a new state of the art, and the ablation showed span-infilling plus sentence-permutation was the most effective corruption for summarization. **PEGASUS** (Zhang et al., _PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization_, ICML 2020) instead chose a pretraining objective shaped like the target task: mask whole sentences that a salience heuristic judges important (gap-sentence generation, choosing the sentences by ROUGE overlap with the rest of the document) and regenerate them. Its headline result was sample efficiency — with as few as $1000$ fine-tuning examples it matched prior systems trained on the full corpus — which is why gap-sentence pretraining became the standard route for low-resource summarization domains.[^bart-lewis][^pegasus-zhang] $$ % caption: The abstractive-summarization lineage. See 2017 added copying and % coverage to a from-scratch encoder-decoder; BART 2020 pretrained a general text % denoiser; PEGASUS 2020 pretrained with a summarization-shaped gap-sentence % objective; and large instruction-tuned LLMs summarize zero-shot. Each step % pushed fluency up and moved the open problem toward faithfulness. \begin{tikzpicture}[>=stealth, font=\footnotesize, b/.style={draw, minimum width=28mm, minimum height=13mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[b] (ptr) at (0,0) {See 2017\\pointer-generator\\(copy + coverage)}; \node[b] (bart) at (3.6,0) {Lewis 2020\\BART\\(denoise pretrain)}; \node[b] (peg) at (7.2,0) {Zhang 2020\\PEGASUS\\(gap-sentence)}; \node[b, draw=acc, text=acc] (llm) at (10.8,0) {LLM\\zero-shot\\prompting}; \draw[->, acc, thick] (ptr) -- (bart); \draw[->, acc, thick] (bart) -- (peg); \draw[->, acc, thick] (peg) -- (llm); \node[font=\scriptsize, text=black, anchor=north] at (5.4,-0.95) {f\/luency rises; faithfulness stays the open problem}; \end{tikzpicture} $$ **Measuring faithfulness (Kryscinski et al., 2020; Wang et al., 2020; Maynez et al., 2020).** Once fluency was largely solved, hallucination became the measured problem. Maynez et al. (_On Faithfulness and Factuality in Abstractive Summarization_, ACL 2020) had humans annotate system summaries and found that a large fraction of abstractive outputs contained content not entailed by the source, and that ROUGE correlated poorly with faithfulness, so a higher-ROUGE system could be _less_ faithful. Two automatic checks answered this. **FactCC** (Kryscinski et al., _Evaluating the Factual Consistency of Abstractive Text Summarization_, EMNLP 2020) trains an entailment-style classifier to judge whether a summary sentence is supported by the document. **QAGS** (Wang, Cho, and Lewis, _Asking and Answering Questions to Evaluate the Factual Consistency of Summaries_, ACL 2020) generates questions from the summary, answers them against both the summary and the source, and flags disagreement — a summary is unfaithful where the two answer streams diverge. **SummaC** (Laban et al., TACL 2022) later showed that aggregating sentence-level entailment scores across the document is a strong, simple consistency detector. The takeaway matches the lesson's definition box: faithfulness is a separate axis from overlap, and it needs its own metric.[^faith-maynez][^factcc-kry][^qags-wang] **Prompted and instruction-tuned LLMs.** The lesson's zero-shot route is the current default. Instruction-tuned models (the InstructGPT line, Ouyang et al., _Training Language Models to Follow Instructions_, NeurIPS 2022) summarize from a plain instruction with no summarization-specific fine-tuning, and human raters often prefer their summaries to those of dedicated fine-tuned systems — while the same faithfulness caveat persists, since a fluent model can still assert what the source does not. The machinery is now good enough that the remaining problem is keeping the summary _true_, not writing it fluently.[^instruct-ouyang] Summarization compresses a document to its essential meaning in one of two ways: **extract** the sentences that carry that meaning, or **generate** new ones that express it. Extraction is safe but choppy; generation is fluent but risky. The same context-primed autoregressive machinery that drives [translation](/natural-language-processing/applications/machine-translation) and [question answering](/natural-language-processing/applications/question-answering) does the generating, and keeping the output faithful to its source remains the open problem. [^jm-ctxgen]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), §9.9 — Contextual Generation and Summarization: text summarization as context-based autoregressive generation; the append-a-summary-with-a-separator scheme converting each article–summary pair $(x_1,\ldots,x_m),(y_1,\ldots,y_n)$ into one training instance $(x_1,\ldots,x_m,\delta,y_1,\ldots,y_n)$ trained with teacher forcing; priming with the article at inference; the CNN/DailyMail news corpus; and single- vs. multi-document and generic vs. query-focused as task variants. [^jm-ptrgen]: **Jurafsky & Martin**, §9.9 (abstractive summarization) — abstractive summarization as encoder-decoder generation, the pointer-generator refinement with a $p_{\text{gen}}$ switch mixing vocabulary generation with copying source words through the attention distribution, and a coverage mechanism accumulating past attention to suppress repetition. [^jm-pretrain]: **Jurafsky & Martin**, §9.9.1 — Applying Transformers to other NLP tasks: pretraining a transformer language model self-supervised on a large corpus and then finetuning on a smaller task-specific dataset; denoising pretraining objectives (BART-style corruption/reconstruction, PEGASUS-style gap-sentence generation) and zero-/few-shot prompting of large language models as the modern route to summarization. [^jm-rouge]: **Jurafsky & Martin**, §9.9 and Ch. 10 §10.8 (MT Evaluation) — recall-oriented n-gram and longest-common-subsequence overlap against human reference summaries (ROUGE-N, ROUGE-L), following the overlap-metric methodology J&M develop for BLEU/chrF; the limitations of surface-overlap metrics (lexical, local, tokenization-sensitive, blind to faithfulness) and the need for human and factuality evaluation, with embedding metrics like BERTScore crediting paraphrase. [^ptrgen-see]: **See, Liu & Manning (2017)**, _Get To The Point: Summarization with Pointer-Generator Networks_, ACL 2017. The pointer-generator LSTM encoder-decoder with a $p_{\text{gen}}$ copy switch and a coverage vector plus coverage loss; ROUGE gains on CNN/DailyMail over prior abstractive baselines, and the residual factual-error failure mode. [^bart-lewis]: **Lewis et al. (2020)**, _BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension_, ACL 2020. A Transformer encoder-decoder pretrained to reconstruct text corrupted by masking, deletion, span infilling, sentence permutation, and document rotation; span infilling plus sentence permutation best for summarization fine-tuning. [^pegasus-zhang]: **Zhang et al. (2020)**, _PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization_, ICML 2020. Gap-sentence generation — masking whole ROUGE-salient sentences and regenerating them — as a summarization-shaped pretraining objective, with strong low-resource (~1000-example) fine-tuning performance. [^faith-maynez]: **Maynez et al. (2020)**, _On Faithfulness and Factuality in Abstractive Summarization_, ACL 2020. Human annotation showing a large share of abstractive summaries contain content not entailed by the source, and that ROUGE correlates poorly with faithfulness. [^factcc-kry]: **Kryscinski et al. (2020)**, _Evaluating the Factual Consistency of Abstractive Text Summarization_ (FactCC), EMNLP 2020 — a weakly-supervised entailment-style classifier judging whether a summary sentence is supported by the document. [^qags-wang]: **Wang, Cho & Lewis (2020)**, _Asking and Answering Questions to Evaluate the Factual Consistency of Summaries_ (QAGS), ACL 2020 — question generation and answering against summary and source, flagging faithfulness where the answers disagree; and **Laban et al. (2022)**, _SummaC_, TACL, aggregating sentence-level NLI scores as a consistency detector. [^instruct-ouyang]: **Ouyang et al. (2022)**, _Training Language Models to Follow Instructions with Human Feedback_ (InstructGPT), NeurIPS 2022 — instruction-tuned models summarize zero-shot from a plain instruction, often preferred by human raters to dedicated fine-tuned summarizers, with the faithfulness caveat persisting.