Entity-Based and Global Coherence

This builds on discourse coherence: relations and parsing, which tied clauses together with coherence relations (RST and the PDTB) and recovered them with discourse parsers. Relational coherence is one of three local sources; here we take up the other two — being about stable entities and sharing a topic — and then the global structure a whole genre imposes.

Centering and entity-based coherence

The second flavor of coherence is being about something. A discourse coheres by keeping the same salient entities in focus rather than swinging among them. Grosz et al.'s minimal pair makes the point with two passages of identical propositional content but different saliences:¹

(i) John went to his favorite music store to buy a piano. He had frequented the store for years. He was excited that he could finally buy a piano. He arrived just as the store was closing.

(ii) John went to his favorite music store to buy a piano. It was a store John had frequented for years. He was excited that he could finally buy a piano. It was closing just as John arrived.

Both are grammatical, but (i) is more coherent: it is steadily about John, whereas (ii) swings John / store / John / store and lacks a stable aboutness.¹

Centering Theory formalizes this as a theory of both salience and coherence. Each utterance $U_n$ carries two structures:¹

• The backward-looking center $C_b(U_n)$: the single entity currently in focus, the one being centered on.
• The forward-looking centers $C_f(U_n)$: the set of entities $U_n$ evokes, any of which could be the center of the next utterance, ranked by grammatical role (subject > object > everything else).

The top-ranked forward center is the preferred center $C_p(U_n)$ — a prediction of what the next utterance will be about. The relationship between successive utterances falls into four transition types, classified by two questions: does $C_b$ stay the same from $U_n$ to $U_{n+1}$, and does the new $C_b$ equal the predicted $C_p$?

Two rules drive the algorithm:¹

A worked centering trace

Run the algorithm on the coherent passage (i), taking its first two utterances. Rank each $C_f$ by grammatical role.

Because $C_b(U_2)=C_p(U_2)=\text{John}$ and $C_b(U_1)$ is undefined, the transition is a Continue — the most coherent type. Passage (i) is a chain of Continues on John; passage (ii), which realizes John and the store to keep swapping which is the subject, forces Shifts, and those Shifts are why it reads worse.

The entity grid

Centering is a top-down theory: it stipulates that salient entities appear as subjects or pronouns and that continuing on them is coherent. The entity grid model of Barzilay and Lapata learns the same intuition bottom-up, inducing the patterns of entity mention that make a discourse coherent.²

An entity grid is a two-dimensional array: one row per sentence, one column per discourse entity. Each cell records the entity's grammatical role in that sentence — Subject, Object, X (neither), or – (absent). An entity appearing more than once in a sentence takes its highest-ranked role. Because the columns are discourse entities, building a grid requires coreference resolution to cluster mentions and parsing to read off grammatical roles — so this stage depends directly on coreference.

Coherence is read off the local entity transitions — a length-2 vertical slice of a column, like $[\text{S}\text{–}]$ (subject then absent) or $[\text{O}\text{X}]$. There are $4^2=16$ possible transitions, and each document becomes a feature vector of their probabilities: the fraction of column-slices that take each value. That vector feeds a machine-learning classifier. Coherent texts have a characteristic topology — a few dense columns (entities mentioned throughout, often as S or O) and many sparse ones.²

To see the vector come out of the grid, count transitions column by column. Each column of $r$ rows yields $r-1$ length-2 slices; a $6$-row, $5$-column grid gives $5\times 5=25$ slices total. Slide down the Microsoft column $[\text{S},\text{O},\text{S},\text{S},\text{–},\text{S}]$ and read off its five adjacent pairs: $[\text{S}\text{O}]$, $[\text{O}\text{S}]$, $[\text{S}\text{S}]$, $[\text{S}\text{–}]$, $[\text{–}\text{S}]$. Do the same for every column and divide each transition's count by $25$.

The vector is dominated by the $[\text{–}\text{–}]$ transition — most entity-sentence cells are empty — with a long tail of role-bearing transitions. Barzilay and Lapata found that giving different weights to salient versus non-salient entities (split by mention frequency) and adding the coreference-based columns beats a bag-of-words baseline on the ordering tasks below; the grid captures something a word-overlap model cannot.²

Evaluating and training coherence models

Labels for coherence are expensive, so most entity-grid and neural models train and evaluate one of two ways.³ The first uses human ratings: have annotators score each document (high/mid/low, or continuous) and fit a classifier to predict those scores — best when an end task like essay grading fixes what the label should mean. The second exploits self-supervision: a naturally ordered discourse is more coherent than a random permutation of its sentences, so a good model should prefer the original. This gives three concrete tasks:³

• Sentence-order discrimination — rank the original document above a random permutation of its sentences.
• Sentence insertion — remove one sentence and find its original slot among the $n$ positions. Harder than discrimination, since the compared documents differ in just one sentence.
• Sentence-order reconstruction — shuffle a document and put the sentences back in the correct order. Hardest of the three, being a reordering rather than a classification.

Representation-learning models for local coherence

The third local flavor — topical coherence — rests on lexical cohesion: discourses cohere by discussing the same topics with words from the same semantic fields.⁴ A line of unsupervised models from the 1990s measured this directly. Lexical chains track words in a discourse that are related through a thesaurus (pine, bush, trees, trunk); their number and density correlate with the topic structure. TextTiling computes the cosine between vectors of raw word counts in neighboring text spans, high within a subtopic and low across a subtopic boundary (we use it for segmentation below). LSA coherence was the first to use embeddings: it represents a sentence $s$ by summing its word vectors and scores overall coherence as the average cosine between adjacent sentences,⁴

Modern neural models keep the how much do neighboring sentences change intuition but add self-supervision, training on natural-vs-scrambled orderings to discover the features that matter for coherence. One such model is the local coherence discriminator (LCD).⁴ LCD scores a text as the average coherence over consecutive sentence pairs. It embeds a pair $(s,t)$ with any sentence encoder and concatenates four combinations — the two vectors, their difference $\mathbf{s}-\mathbf{t}$, its absolute value $|\mathbf{s}-\mathbf{t}|$, and the element-wise product $\mathbf{s}\odot \mathbf{t}$ — then passes them through a one-layer feedforward net to a coherence score.

Training makes true consecutive pairs $(s_i,s_{i+1})$ score above negative pairs $(s_i,s^{\prime })$ with $s^{\prime }$ a sentence sampled uniformly from elsewhere in the same document. With a margin loss $\ell (f^{+},f^{-})=\max (0,\eta -f^{+}+f^{-})$ over positive $f^{+}=f_{\theta }(s_i,s_{i+1})$ and negative $f^{-}=f_{\theta }(s_i,s^{\prime })$, the objective is⁴

A related predictability baseline trains an RNN language model and measures how much the preceding context raises the log-likelihood of $s_i$ (conditional minus marginal log-likelihood) — a coherence signal for free. Pushing beyond consecutive pairs helps: a Transformer trained with a contrastive objective to predict text up to $\pm 2$ sentences away yields stronger discourse representations still.⁴

Global coherence

A discourse must cohere globally, not just between neighboring sentences. Two kinds of global structure show up: the topic organization of a text, and the conventional macro-structure of a genre.

Topic segmentation

Long documents move through subtopics, and finding the boundaries is topic segmentation. The classic unsupervised method is TextTiling: slide a window across the text, represent the block of sentences on each side of a candidate gap as a word-count vector, and compute their cosine.⁴ Similarity stays high inside a subtopic (neighbors share vocabulary) and dips at a subtopic boundary. Boundaries are placed at the deep valleys of the similarity-versus-position curve.

Discourse macro-structure: schemas

Beyond topic flow, whole genres impose conventional structure. The oldest studied case is narrative: in his Morphology of the Folktale, Propp models Russian folktales with a plot grammar of dramatis personae (character roles like Hero, Villain, Donor, Helper) and functions (events like Villain commits kidnapping,Hero is pursued) that occur in a fixed order.⁵ Every tale is a subset of the functions in that order — a discourse grammar of stories. Recent work induces such Proppian functions from folktale corpora, and learns latent personae from movie plot summaries using features like the actions a character takes (Villains strangle), what is done to them (Villains are foiled), and how they are described (Villains are evil).⁵

Argumentation structure

A second, widely studied kind of global structure is the structure of arguments; analyzing it computationally is called argumentation mining.⁶ Aristotle named three appeals — pathos (emotion), ethos (character), and logos (logic) — and most computational work targets logos. Persuasive-essay corpora annotate argumentative components: claims (the controversial central assertions that need support) and premises (reasons offered to support or attack a claim), plus the argumentative relations between them, SUPPORT and ATTACK. In the essay

(1) Museums and art galleries provide a better understanding of arts than > Internet. (2) In most museums and art galleries, detailed descriptions of the > background, history and author are provided. (3) Seeing an artwork online is not > the same as watching it with our own eyes, as (4) the picture online does not show > the texture or three-dimensional structure of the art.

sentence (1) is the claim, (2) and (3) are premises supporting it, and (4) is a premise supporting (3): three argumentative relations, SUPPORT(2,1), SUPPORT(3,1), SUPPORT(4,3).

Detection systems combine a component classifier (claim / premise / non-argument) with a relation classifier (SUPPORT / ATTACK / neither).⁶ Richer work labels argumentation schemes — larger patterns like argument from example or argument from cause to effect — and studies which features make an argument persuasive, connecting logos back to Aristotle's ethos and pathos through social cues like reciprocity, authority, and scarcity.⁶ Argument structure is markedly less local than coherence relations: a persuasive essay has one main claim with premises scattered throughout, without the adjacent-clause linking of RST.

Scientific discourse structure

Scientific papers carry their own conventional macro-structure: somewhere the authors state a goal, develop a method, present evidence, and compare to prior work.⁶ The argumentative zoning scheme captures these rhetorical roles by assigning each sentence one of 15 labels — AIM, OWN_METHOD, OWN_RESULTS, USE, GAP_WEAK, SUPPORT, ANTISUPPORT, and others — reflecting the idea that a paper makes a knowledge claim and positions it against the field. Labeled corpora of computational-linguistics and chemistry articles supervise a standard sentence-classification model to assign the tags.⁶

Neural discourse parsing and coherence

The models above are the classical backbone; the last decade reshaped each of them around learned representations. Four public lines of work stand out.

Representation learning for RST. Ji and Eisenstein (2014), Representation Learning for Text-level Discourse Parsing (ACL), replaced hand-built lexical and syntactic features with learned vector representations of each discourse span, projecting the surface features into a latent space jointly with the parsing objective (their DPLP parser). Learned span vectors, rather than sparse features, have driven RST parsing accuracy ever since and are the reason the shift-reduce RST parser of the companion lesson uses a hierarchical biLSTM to encode EDUs.⁷

Top-down parsing with pointer networks. That shift-reduce parser builds the tree bottom-up. Lin et al. (2019) and Kobayashi et al. (2020) showed a top-down alternative: a pointer-network decoder repeatedly splits a span by pointing at the boundary where it divides into two subtrees, recursing until it reaches EDUs. Top-down splitting sees the whole span before committing to a division, which helps on the long-range attachments that trip a greedy bottom-up parser, and these systems set strong numbers on the RST Discourse Treebank.⁸

Implicit relations under pretrained models. Task 4 of shallow parsing — labeling the implicit sense of an adjacent sentence pair — was for years the field's hardest number. Feeding both arguments to a pretrained masked language model and classifying off the pooled representation (the BERT [CLS] baseline the companion lesson describes, from Nie et al. 2019) jumped implicit-relation accuracy well past the feature-engineered systems that preceded it, and later work adds the connective as an auxiliary prediction target so the model learns the word that could have appeared. Implicit relation recognition is still open — the four-way top-level split is far from solved — but pretrained encoders reset the baseline.⁹

Neural coherence models and their benchmark. Nguyen and Joty (2017) gave the entity grid a neural form, running a convolutional network over the grid so the model learns which local transition patterns matter rather than enumerating all sixteen, and it beat the classic grid on the ordering tasks. More broadly, the self-supervised discriminators the lesson describes (the LCD, the contrastive Transformer) are now evaluated on shared discourse benchmarks such as DiscoEval (Chen et al. 2019), a suite of probing tasks — sentence position, binary sentence ordering, discourse coherence, RST-relation prediction — that measure how much discourse structure a sentence encoder captures. The through-line is the same in every case: the classical model states what to measure (relations, entity transitions, adjacent-sentence continuity) and the neural version learns the features that measure it.¹⁰

The through-line

A text is more than a set of sentences, and these two lessons are the catalog of what binds them. Coherence relations tie clauses; Centering and the entity grid thread salient entities through the sentences; lexical cohesion and neural discriminators track shared topic; and above all of it, genres impose narrative, argumentative, and scientific macro-structure. The entity-based half depends squarely on the earlier coreference machinery — whether a text stays about John cannot be checked until the mentions of John are identified — and lifts it from the sentence to the discourse. Downstream, the same structure is used directly: a summarizer keeps nuclei and drops satellites, an essay grader reads the argument structure, and a coherence score flags text that fails to hang together.