Abstractive Summarization and Evaluation

This builds on Text Summarization, which defined the task and its flavors and worked through extractive summarization — scoring sentences by position and centrality, ranking them with TextRank/LexRank PageRank, and learning the selection. Extraction is safe but can only reuse the source's own sentences. This part takes up the other family, abstractive summarization, which generates fresh text, and the evaluation that tells whether either kind of summary is any good.

Abstractive summarization

Extractive methods can never say anything the document did not already say in one of its sentences. To compress within a sentence, fuse two sentences into one, or paraphrase, the summarizer has to generate — the job an encoder-decoder does: read the source, then produce a fresh string one token at a time.

The sequence-to-sequence approach

Summarization is conditional generation, and J&M cast it as ordinary language modeling. Take a corpus of article–summary pairs — the standard one is CNN/DailyMail, news stories each paired with their human-written bullet highlights — and for each pair $(x_1,\dots ,x_m)$, $(y_1,\dots ,y_n)$ concatenate them into one long training sequence with a special separator token $\delta$ between article and summary:

Train an autoregressive language model on these long sequences by teacher forcing, exactly as any language model is trained to predict the next word.¹ At inference, feed a new article ending in $\delta$ as the priming context and let the model generate the summary token by token until it emits an end-of-sequence marker. Because a transformer attends over its whole input, the model has direct access to the entire article and to everything it has generated so far at every step — the property that makes context-based generation work.

This scheme is the basis not just for summarization but for the whole family of text-to-text tasks — translation, question answering, summarization — that share the encoder-decoder shape.¹

The pointer-generator: copying and coverage

Plain sequence-to-sequence summarizers have two chronic faults. First, they cannot reliably reproduce facts copied from the source — a name, a number, a rare technical term the softmax has barely seen — because generating it requires the exact token to win over the entire vocabulary. Second, they repeat themselves, looping on a phrase because nothing tracks what has already been said. The pointer-generator network fixes both.²

The copy fix is a soft switch. At each decoder step the model computes a generation probability $p_{\text{gen}}\in [0,1]$ — how much to trust the vocabulary softmax versus how much to copy from the source — and forms the final next-word distribution as a mixture of generating a vocabulary word and pointing at a source word via the attention weights $\alpha$:

When $p_{\text{gen}}\to 0$ the model points — it copies the source word its attention is focused on, so a name or number the vocabulary softmax could never have produced is reproduced exactly. When $p_{\text{gen}}\to 1$ it generates a paraphrase from the vocabulary. The single scalar lets the model copy and paraphrase within one sentence, which is the essence of abstractive compression.

The repetition fix is coverage. Keep a running coverage vector $c_t$ that accumulates the attention each source word has received over all previous decoder steps, $c_t={\sum }_{t^{\prime }<t}{\alpha }^{t^{\prime }}$, and add a penalty to the loss whenever a new step attends again to a source word that has already been heavily attended. The coverage vector records which source words have already been summarized, and penalizing re-attention stops the decoder from generating the same clause twice.²

Pretrained summarizers and prompting an LLM

The pointer-generator was trained from scratch on the summarization corpus. The modern approach instead pretrains a large encoder-decoder on generic text with a self-supervised objective, then fine-tunes it on the summarization corpus — the same pretrain-then-finetune recipe J&M describe for applying transformers to downstream tasks.³ Two pretraining objectives are purpose-built to teach a model to reconstruct and compress text before it ever sees a summary.

A BART-style model corrupts its input — masking spans of tokens, deleting tokens, shuffling sentence order — and trains the decoder to reconstruct the clean original. This makes it a general text denoiser, a strong starting point for any generation task. A PEGASUS-style model targets summarization directly with gap-sentence generation: it removes the document's most important whole sentences and trains the model to regenerate them from the rest, so the pretraining task already is a kind of summarization. Fine-tuned on CNN/DailyMail, both far surpass a from-scratch pointer-generator, because they already model fluent, coherent text before fine-tuning begins.

The most recent shift removes fine-tuning entirely. A sufficiently large language model can summarize zero-shot: append an instruction to the document — literally Summarize the above article in three sentences: — and let the model generate, with no gradient update and no summarization-specific training. A few demonstrations of article-then-summary pairs in the prompt (few-shot) sharpens length and style further. Prompted summarization is just contextual generation where the context is an instruction; it is the same mechanism as the append-marker scheme, with a natural-language instruction playing the role of the separator $\delta$.³

Long documents and factuality

Two problems dog abstractive summarization in practice, and both get worse the longer the source.

Long input. A transformer's self-attention costs grow quadratically with input length, and every model has a fixed context window, so a book, a court filing, or a long transcript will not fit. Three responses:

Truncation is the crude default and is often adequate for news (whose important material is front-loaded, the same position signal the extractive methods exploit). Hierarchical summarization — split into sections, summarize each, then summarize the section-summaries — scales to arbitrary length but can drop connections that span chunks. Purpose-built long-context models push the window out but never make it infinite.

Factuality. The signature failure of abstractive summarization is hallucination: a fluent summary that states something the source does not support, or contradicts. Because the model generates rather than copies, nothing guarantees its output stays faithful to the input — it can invent a number, misattribute a quote, or merge two entities into one.

Faithfulness is precisely what overlap-based evaluation (next) fails to measure — a hallucinated summary can share plenty of n-grams with the reference and still be wrong — so factuality is usually assessed separately, by entailment models that check whether the source entails each summary sentence, or by human judgment.

Evaluation: ROUGE

Judging a summary means comparing it against one or more human reference summaries. The standard automatic metric is ROUGE (Recall-Oriented Understudy for Gisting Evaluation), the summarization counterpart of the BLEU metric used for translation. Where BLEU is precision-oriented (what fraction of the candidate's n-grams are in the reference), ROUGE is recall-oriented (what fraction of the reference's n-grams the candidate recovered) — the natural emphasis for summarization, where the worry is leaving important content out.⁴

Work one example. Take a reference and a candidate summary sentence:

Reference: the man ships Boston snow to warm states Candidate: the man ships Boston snow online

Carry the numbers through. The reference has $8$ unigrams; the candidate shares five of them (the, man, ships, Boston, snow) and misses three (to, warm, states), so ROUGE-1 recall is $5/8=0.63$. The reference's $7$ bigrams are the man, man ships, ships Boston, Boston snow, snow to, to warm, warm states; the candidate reproduces the first four, so ROUGE-2 recall is $4/7\approx 0.57$. The longest common subsequence is the man ships Boston snow, length $5$, which drives ROUGE-L. The candidate's extra word online costs nothing in recall (recall only measures how much of the reference was recovered), which is why full ROUGE reports precision and F-measures too, so padding a summary with unmatched words is penalized.⁴

Carrying the LCS through to an F-measure shows the precision side at work. The candidate has $6$ words, the reference $8$, and the LCS is length $5$. Then LCS-precision is $P_{\text{lcs}}=5/6\approx 0.83$ (five of the candidate's six words lie on the common subsequence) and LCS-recall is $R_{\text{lcs}}=5/8=0.63$. The ROUGE-L F-measure combines them,

so the balanced score sits between the two, and a candidate that recovered the same content but ran on for twenty extra words would keep the recall of $0.63$ while its precision — and therefore its $F_{\text{lcs}}$ — dropped. That is the mechanism the recall-only ROUGE-1 figure above hides: recall alone rewards saying more, and only the precision term penalizes it.

The limits of ROUGE

ROUGE inherits every weakness of surface overlap, and adds one of its own.

• It is lexical: a correct paraphrase that uses different words scores zero overlap, so an abstractive summary is systematically undercredited relative to an extractive one that copies the reference's exact words.
• It is local: it counts n-grams, so it barely notices whether the summary is globally coherent or self-contradictory.
• Most seriously, it does not measure faithfulness. A hallucinated summary can share many n-grams with the reference and still assert something false; ROUGE cannot tell the difference between a faithful summary and a fluent fabrication with the same words.

For these reasons ROUGE, like BLEU, is best used to track changes to a single system during development, and is least trustworthy when comparing very different systems. Serious evaluation pairs it with human judgment of the two axes that matter — is the summary coherent and does it cover the important content? — and, increasingly, with dedicated factuality checks: entailment models, or an LLM asked to verify each summary sentence against the source. Embedding-based metrics such as BERTScore (introduced for translation evaluation) also transfer to summarization, crediting paraphrase where ROUGE cannot.⁴

The abstractive-summarization lineage

Jurafsky & Martin present the pointer-generator, denoising pretraining, and prompting as a sequence of ideas. The ideas came from specific systems: the field moved through them in a clear order, and each paper isolated one problem.

Copying and coverage (See et al., 2017). The pointer-generator network the lesson describes is due to See, Liu, and Manning, Get To The Point: Summarization with Pointer-Generator Networks (ACL 2017). Their model was an LSTM encoder-decoder with attention, and the two additions match the two faults exactly: the $p_{\text{gen}}$ soft switch to copy source tokens, and the coverage vector plus coverage loss to stop repetition. On CNN/DailyMail they reported ROUGE-1 near $39$ and ROUGE-L near $36$, beating the abstractive baselines of the day, and they showed that the copy mechanism cut the rate of factually garbled rare words. The paper also documented the failure mode that motivated everything after it: even with copying, the model still occasionally produced fluent sentences unsupported by the source.⁵

Pretraining takes over (Lewis et al., 2020; Zhang et al., 2020). Two 2020 papers replaced the from-scratch encoder-decoder with a pretrained one, the split the lesson draws as BART-style versus PEGASUS-style. BART (Lewis et al., BART: Denoising Sequence-to-Sequence Pre-training, ACL 2020) pretrains a standard Transformer encoder-decoder as a denoiser: corrupt the input with several noise functions — token masking, token deletion, text-span infilling, sentence permutation, document rotation — and train the decoder to reconstruct the clean text. Fine-tuned on CNN/DailyMail it set a new state of the art, and the ablation showed span-infilling plus sentence-permutation was the most effective corruption for summarization. PEGASUS (Zhang et al., PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization, ICML 2020) instead chose a pretraining objective shaped like the target task: mask whole sentences that a salience heuristic judges important (gap-sentence generation, choosing the sentences by ROUGE overlap with the rest of the document) and regenerate them. Its headline result was sample efficiency — with as few as $1000$ fine-tuning examples it matched prior systems trained on the full corpus — which is why gap-sentence pretraining became the standard route for low-resource summarization domains.⁶⁷

Measuring faithfulness (Kryscinski et al., 2020; Wang et al., 2020; Maynez et al., 2020). Once fluency was largely solved, hallucination became the measured problem. Maynez et al. (On Faithfulness and Factuality in Abstractive Summarization, ACL 2020) had humans annotate system summaries and found that a large fraction of abstractive outputs contained content not entailed by the source, and that ROUGE correlated poorly with faithfulness, so a higher-ROUGE system could be less faithful. Two automatic checks answered this. FactCC (Kryscinski et al., Evaluating the Factual Consistency of Abstractive Text Summarization, EMNLP 2020) trains an entailment-style classifier to judge whether a summary sentence is supported by the document. QAGS (Wang, Cho, and Lewis, Asking and Answering Questions to Evaluate the Factual Consistency of Summaries, ACL 2020) generates questions from the summary, answers them against both the summary and the source, and flags disagreement — a summary is unfaithful where the two answer streams diverge. SummaC (Laban et al., TACL 2022) later showed that aggregating sentence-level entailment scores across the document is a strong, simple consistency detector. The takeaway matches the lesson's definition box: faithfulness is a separate axis from overlap, and it needs its own metric.⁸⁹¹⁰

Prompted and instruction-tuned LLMs. The lesson's zero-shot route is the current default. Instruction-tuned models (the InstructGPT line, Ouyang et al., Training Language Models to Follow Instructions, NeurIPS 2022) summarize from a plain instruction with no summarization-specific fine-tuning, and human raters often prefer their summaries to those of dedicated fine-tuned systems — while the same faithfulness caveat persists, since a fluent model can still assert what the source does not. The machinery is now good enough that the remaining problem is keeping the summary true, not writing it fluently.¹¹

Summarization compresses a document to its essential meaning in one of two ways: extract the sentences that carry that meaning, or generate new ones that express it. Extraction is safe but choppy; generation is fluent but risky. The same context-primed autoregressive machinery that drives translation and question answering does the generating, and keeping the output faithful to its source remains the open problem.