--- title: "Coherence and Discourse Structure" module: Linguistic Structure moduleNumber: 6 lessonNumber: 10 order: 610 summary: > Coherence is what makes a run of sentences a discourse rather than an arbitrary collection. This lesson develops coherence relations and Rhetorical Structure Theory trees, discourse-structure parsing, Centering and the entity grid for entity-based coherence, and representation-learning models of local coherence, measured in part over the coreference chains recovered in the companion lesson. topics: [Structure] sources: - book: Jurafsky ref: "Ch. 22 — Discourse Coherence; §22.1 Coherence Relations; §22.2 Discourse Structure Parsing; §22.3 Centering and Entity-Based Coherence; §22.4 Representation Learning Models" --- This builds on [coreference resolution](/natural-language-processing/linguistic-structure/coreference-and-discourse), which recognized one entity under its many names across a text. Coreference is only the entity half of tracking a discourse; the other half is what makes a sequence of sentences hang together at all — the subject of this lesson. ## Discourse: what makes a text cohere Language comes in **collocated, structured groups** of sentences — a **discourse** — and **coherence** is the relationship among sentences that separates a real discourse from a random pile.[^jm-coh] Compare Hobbs's minimal pair: > (a) John took a train from Paris to Istanbul. He likes spinach. > (b) Jane took a train from Paris to Istanbul. She had to attend a conference. (a) is incoherent — what has spinach to do with train trips? — while (b) coheres because the second sentence gives a **reason** for the first.[^jm-coh] That readers _strain_ to find a connection in (a) shows coherence is something comprehension actively constructs. Real discourses are coherent in three local ways and one global way:[^jm-coh] | Kind | What ties the text together | Model | | --- | --- | --- | | Relational | **coherence relations** between clauses (Reason, Elaboration, …) | RST, PDTB | | Entity-based | the discourse stays "about" the same salient entities | Centering, entity grid | | Topical | nearby sentences share a topic and vocabulary (**lexical cohesion**) | TextTiling, embeddings | | Global | conventional structure of a genre (Methods/Results, plot) | discourse structure | ## Coherence relations A **coherence relation** is a structured link between two spans of text.[^jm-rst] The dominant model is **Rhetorical Structure Theory (RST)**. RST relations hold between a **nucleus** (central, independently interpretable) and a **satellite** (peripheral, interpretable only relative to the nucleus); some symmetric relations hold between two nuclei.[^jm-rst] A few examples: | Relation | Nucleus / satellite | Example | | --- | --- | --- | | Reason | NUC action, SAT its reason | _Jane took a train._ / _She had to attend a conference._ | | Elaboration | SAT adds detail | _Dorothy was from Kansas._ / _She lived in the prairies._ | | Evidence | SAT convinces of NUC | _Kevin must be here._ / _His car is parked outside._ | | Attribution | SAT sources reported speech | _Analysts estimated_ / _that sales declined._ | | List | multinuclear series | _Billy was the mate;_ / _Long John was quartermaster._ | The base units — leaves of an RST tree — are **elementary discourse units (EDUs)**, roughly clauses.[^jm-rst] Relations compose hierarchically into a tree over the whole text; asymmetric relations are drawn as an arrow from satellite to nucleus. $$ % caption: An RST-style relation tree over a short text. EDUs are leaves; an % arrow runs from each satellite to its nucleus, and relations nest: the % Elaboration over spans 2-3 is the satellite of a Background under the whole. \begin{tikzpicture}[>=stealth, font=\footnotesize, edu/.style={draw, minimum width=24mm, minimum height=11mm, align=center, font=\scriptsize, inner sep=2pt}, rel/.style={font=\scriptsize, text=acc}] \definecolor{acc}{HTML}{2348F2} \node[edu] (e1) at (0,0) {(1) Mars has a\\distant orbit and\\slim atmosphere,}; \node[edu] (e2) at (3.0,0) {(2) so Mars has\\frigid weather.}; \node[edu] (e3) at (6.4,0) {(3) Temperatures\\average about\\-60 C.}; % internal node for 1-2 \coordinate (n12) at (1.5,1.8); \draw[black] (e1.north) -- (n12); \draw[->, acc] (e2.north) -- (n12) node[pos=0.5, right, rel] {evidence}; % root spanning (1-2) and (3) \coordinate (root) at (3.7,3.1); \draw[black] (n12) -- (root); \draw[->, acc] (e3.north) to[out=90,in=-20] node[pos=0.5, right, rel] {elaboration} (root); \node[anchor=south, black, font=\scriptsize] at (3.7,3.15) {discourse}; \end{tikzpicture} $$ A second, more lexically grounded corpus is the **Penn Discourse TreeBank (PDTB)**.[^jm-pdtb] Rather than tag relations directly, annotators start from **discourse connectives** — words like _because_, _although_, _since_, _as a result_ — that signal a relation between two arguments **Arg1** and **Arg2**. Where a relation has no explicit connective, the annotator supplies the one that _could_ have appeared (marked _implicit_) and labels its sense. PDTB senses form a four-way top level — **Temporal**, **Contingency**, **Comparison**, **Expansion** — refined into types and subtypes.[^jm-pdtb] Unlike RST, PDTB annotates only span pairs and makes no commitment to a global tree.[^jm-pdtb] ## Discourse structure parsing **Discourse parsing** recovers these relations automatically.[^jm-parse] For RST it runs in two stages. First, **EDU segmentation** labels each token as a break or not, typically with a neural sequence model over an encoder: > _[Mr. Rambo says]_ _[that a 3.2-acre property]_ _[overlooking the San Fernando > Valley]_ _[is priced at \$4 million]_ _[because the late actor Erroll Flynn once > lived there.]_ Second, an RST **parser** builds the tree. Modern parsers are **shift-reduce** transition systems with a stack and a queue and two core actions:[^jm-parse] - **shift** — pop the front EDU off the queue onto the stack as a single-node subtree. - **reduce**$(l, d)$ — merge the top two subtrees, labeling the relation $l$ and the nuclearity direction $d \in \{\text{NN}, \text{NS}, \text{SN}\}$. plus a **pop-root** that removes the final tree. The Yu et al. parser encodes words and EDUs with a hierarchical biLSTM, then a feedforward decoder chooses each action from the top few stack subtrees and the front-of-queue EDU. Below is the action trace that builds a four-EDU tree. $$ % caption: A shift-reduce trace building an RST tree over four EDUs. Each row is % a parser state (stack contents) and the action taken; SH shifts an EDU, RD % reduces the top two subtrees with a labeled relation, PR pops the finished root. \begin{tikzpicture}[>=stealth, font=\footnotesize, row/.style={font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[row, anchor=west] at (0,3.0) {\textbf{stack}}; \node[row, anchor=west] at (5.4,3.0) {\textbf{action}}; \node[row, anchor=west] at (0,2.4) {e1}; \node[row, anchor=west, text=acc] at (5.4,2.4) {SH}; \node[row, anchor=west] at (0,1.8) {e1 e2}; \node[row, anchor=west, text=acc] at (5.4,1.8) {SH}; \node[row, anchor=west] at (0,1.2) {e1:2}; \node[row, anchor=west, text=acc] at (5.4,1.2) {RD(attr, SN)}; \node[row, anchor=west] at (0,0.6) {e1:2 e3 e4}; \node[row, anchor=west, text=acc] at (5.4,0.6) {SH SH}; \node[row, anchor=west] at (0,0.0) {e1:2 e3:4}; \node[row, anchor=west, text=acc] at (5.4,0.0) {RD(elab, NS)}; \node[row, anchor=west] at (0,-0.6) {e1:4}; \node[row, anchor=west, text=acc] at (5.4,-0.6) {RD(elab, SN)}; \node[row, anchor=west] at (0,-1.2) {(root)}; \node[row, anchor=west, text=acc] at (5.4,-1.2) {PR}; \draw[black] (-0.1,2.15) -- (7.7,2.15); \end{tikzpicture} $$ Training maps each gold tree to its oracle action sequence and fits the decoder with cross-entropy. RST parsers report four $F_1$ scores of increasing strictness: **Span** (bracketing only) $\le$ **Nuclearity** (bracketing + N/S direction) $\le$ **Relation** (+ label) $\le$ **Full** (all three).[^jm-parse] **PDTB parsing** is shallower — flat relations, no tree — and splits into four subtasks: find discourse connectives (disambiguating _and_ the connective from _and_ the NP conjunction), find each connective's two spans, label explicit relations, and label the relation between every adjacent sentence pair (the implicit case).[^jm-parse] A strong baseline for the implicit task encodes each span with BERT, takes the `[CLS]` state, and passes it through a small feedforward net and softmax.[^jm-parse] ## Centering and entity-based coherence The second flavor of coherence is being **about** something. A discourse coheres by keeping the same salient entities in focus rather than lurching among them. Consider Grosz et al.'s minimal pair, same propositions, different realization:[^jm-center] > (i) John went to his favorite music store to buy a piano. He had frequented the > store for years. He was excited that he could finally buy a piano. He arrived just > as the store was closing. > > (ii) John went to his favorite music store to buy a piano. It was a store John had > frequented for years. He was excited that he could finally buy a piano. It was > closing just as John arrived. Both are grammatical, but (i) is more coherent — it is steadily _about John_, while (ii) swings John / store / John / store.[^jm-center] **Centering Theory** formalizes this. Each utterance $U_n$ carries two structures: a **backward-looking center** $C_b(U_n)$, the single entity currently in focus, and a set of **forward-looking centers** $C_f(U_n)$, the entities $U_n$ evokes, ranked by grammatical role (subject $>$ object $>$ other).[^jm-center] The top-ranked forward center is the **preferred center** $C_p(U_n) = \max C_f(U_n)$, a prediction of what comes next. The relationship between successive utterances falls into four transition types, ordered by how coherent they are: $$ % caption: The four Centering transitions, classified by whether the % backward-looking center stays the same across utterances (columns) and whether % it equals the predicted preferred center (rows). Continue is most coherent, % Rough-Shift least. \begin{tikzpicture}[>=stealth, font=\footnotesize, cell/.style={draw, minimum width=34mm, minimum height=10mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[anchor=south, font=\scriptsize, align=center] at (2.0,1.4) {Cb stays same}; \node[anchor=south, font=\scriptsize, align=center] at (6.0,1.4) {Cb changes}; \node[anchor=east, font=\scriptsize, align=right] at (-0.3,0.5) {Cb = Cp}; \node[anchor=east, font=\scriptsize, align=right] at (-0.3,-1.1) {Cb != Cp}; \node[cell, draw=acc, text=acc] (c11) at (2.0,0.5) {Continue}; \node[cell] (c12) at (6.0,0.5) {Smooth-Shift}; \node[cell] (c21) at (2.0,-1.1) {Retain}; \node[cell, draw=red, text=red] (c22) at (6.0,-1.1) {Rough-Shift}; \node[anchor=west, black, font=\scriptsize] at (-0.2,-2.2) {preference: Continue $>$ Retain $>$ Smooth-Shift $>$ Rough-Shift}; \end{tikzpicture} $$ Two rules drive the theory. **Rule 1**: if any forward center is pronominalized in the next utterance, then $C_b$ must be pronominalized too — pronouns mark the most salient entity. **Rule 2**: transitions are preferred in the order Continue > Retain > Smooth-Shift > Rough-Shift.[^jm-center] Discourse (i) is a chain of Continues on John; (ii) forces Shifts, which is why it reads worse. ### The entity grid Centering is a top-down theory. The **entity grid** learns the same intuition bottom-up.[^jm-grid] A grid is a table: one **row per sentence**, one **column per entity**, each cell recording the entity's grammatical role in that sentence — **S**ubject, **O**bject, **X** (other), or **–** (absent). Columns come from coreference resolution, so this stage depends on the first half of the lesson. $$ % caption: A fragment of an entity grid. Rows are sentences, columns are entities, % cells give the grammatical role: S subject, O object, X other, dash absent. % Dense columns (Microsoft) mark salient entities; local transitions like [S -] % become features for a coherence classifier. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \def\cw{1.5} \def\rh{0.62} % headers \foreach \c/\name in {1/Dept, 2/Trial, 3/Microsoft, 4/Netscape, 5/Suit} \node[font=\scriptsize] at (\c*\cw, 0.55) {\name}; \foreach \r in {1,...,6} \node[font=\scriptsize, anchor=east] at (0.4,-\r*\rh) {S\r}; % grid cell values: rows are sentences % row1 \foreach \c/\v in {1/S, 2/O, 3/S, 4/-, 5/-} \node[font=\scriptsize] at (\c*\cw,-1*\rh) {\v}; \foreach \c/\v in {1/-, 2/-, 3/O, 4/-, 5/-} \node[font=\scriptsize] at (\c*\cw,-2*\rh) {\v}; \foreach \c/\v in {1/-, 2/-, 3/S, 4/O, 5/-} \node[font=\scriptsize] at (\c*\cw,-3*\rh) {\v}; \foreach \c/\v in {1/-, 2/-, 3/S, 4/-, 5/-} \node[font=\scriptsize] at (\c*\cw,-4*\rh) {\v}; \foreach \c/\v in {1/-, 2/-, 3/-, 4/-, 5/O} \node[font=\scriptsize] at (\c*\cw,-5*\rh) {\v}; \foreach \c/\v in {1/-, 2/X, 3/S, 4/-, 5/-} \node[font=\scriptsize] at (\c*\cw,-6*\rh) {\v}; % highlight the dense Microsoft column \draw[acc, thick] (3*\cw-0.55,-6*\rh-0.3) rectangle (3*\cw+0.55,-1*\rh+0.3); \node[acc, anchor=west, font=\scriptsize] at (5*\cw+0.5,-3.5*\rh) {dense = salient}; \draw[->, acc] (5*\cw+0.45,-3.5*\rh) -- (3*\cw+0.6,-3.5*\rh); \end{tikzpicture} $$ Coherence is read off the **local transitions** — a length-2 slice of a column, like $[\text{S}\ \text{–}]$ (subject then absent). With role set $R = \{\text{S}, \text{O}, \text{X}, \text{–}\}$, each of the $|R|^2 = 16$ transition types $(r, r')$ becomes a feature: its frequency across all columns, $$ \phi_{(r,r')} = \frac{\#\,\text{column pairs with transition } (r,r')}{\#\,\text{column transitions}}, $$ collected into a document vector $\boldsymbol{\phi}$ fed to a classifier.[^jm-grid] Coherent texts have a characteristic topology: a few **dense** columns (entities mentioned throughout, often as S or O) and many **sparse** ones.[^jm-grid] Labels are expensive, so entity-grid and neural coherence models train by **self-supervision**: a naturally-ordered discourse is more coherent than a permutation of its sentences, so the model learns to prefer the original.[^jm-grid] This is realized as sentence-order **discrimination** (rank original above a permutation), **insertion** (find the original slot for a removed sentence), or **reconstruction** (put shuffled sentences back in order).[^jm-eval2] ## Local vs. global coherence Everything above is **local** coherence — links between adjacent or nearby sentences. Discourses also exhibit **global** coherence: genres carry conventional structures. Academic articles have Methods and Results; stories follow plot motifs; persuasive essays state a claim and marshal premises for and against it.[^jm-coh] Coherence detection matters wherever text quality matters — essay grading, summarization (drop satellites, keep nuclei), even clinical measures of disordered language.[^jm-coh] > **Note (Representation-learning coherence models).** The third local flavor, > **topical** coherence, rests on **lexical cohesion**: nearby sentences share words > from one semantic field.[^jm-repl] Early unsupervised models measured it directly — > lexical chains, TextTiling's cosine between neighboring spans, and LSA coherence as > the average cosine between adjacent-sentence embeddings. Modern neural models keep > the "how much do neighboring sentences change" intuition but add self-supervision. > The **local coherence discriminator** scores a text as the mean pairwise score over > consecutive pairs, $\text{coh}(d) = \frac{1}{n-1}\sum_{i=1}^{n-1} f(s_i, s_{i+1})$, > trained to rank true consecutive pairs above pairs $(s_i, s')$ with $s'$ sampled > from elsewhere in the document. The pair scorer $f$ embeds both sentences and feeds > the vector $[\,s \,;\, t \,;\, s - t \,;\, |s - t| \,;\, s \odot t\,]$ — concatenation, > difference, absolute difference, and element-wise product — through a feedforward > net.[^jm-repl] ## The through-line A text is more than a bag of sentences. Coreference is the machinery that recognizes one entity under its many names; coherence is what ties clauses and entities across a whole document. The two halves meet in the entity grid, where coherence is measured _over_ the coreference chains: whether a text stays "about John" cannot be assessed until the mentions of John are identified. Both take the entities and relations of [semantic roles and information extraction](/natural-language-processing/linguistic-structure/semantic-roles-and-information-extraction) and lift them from the sentence to the discourse. [^jm-coh]: **Jurafsky & Martin**, Ch. 22 intro — Discourse Coherence: discourse and coherence; Hobbs's spinach/conference minimal pair; local coherence (relational, entity-based, topical/lexical cohesion) vs. global coherence and conventional genre structure; uses in essay grading, summarization, clinical language. [^jm-rst]: **Jurafsky & Martin**, §22.1.1 — Rhetorical Structure Theory: nucleus/satellite relations (Reason, Elaboration, Evidence, Attribution, List); elementary discourse units (EDUs); hierarchical trees with satellite-to-nucleus arrows; the RST Discourse Treebank. [^jm-pdtb]: **Jurafsky & Martin**, §22.1.2 — Penn Discourse TreeBank: lexically grounded annotation via discourse connectives linking Arg1 and Arg2; implicit relations; the four-class sense hierarchy (Temporal, Contingency, Comparison, Expansion); no global tree, unlike RST. [^jm-parse]: **Jurafsky & Martin**, §22.2 — Discourse Structure Parsing: EDU segmentation as neural sequence labeling; shift-reduce RST parsing with shift/reduce($l$,$d$)/pop-root actions and a hierarchical biLSTM encoder; RST-Pareval Span/Nuclearity/Relation/Full metrics; PDTB shallow parsing's four subtasks and a BERT `[CLS]` baseline. [^jm-center]: **Jurafsky & Martin**, §22.3.1 — Centering: the Grosz et al. piano-store minimal pair; backward-looking center $C_b$, forward-looking centers $C_f$ ranked by grammatical role, preferred center $C_p$; the Continue/Retain/Smooth-Shift/Rough-Shift transitions; Rule 1 (pronominalization) and Rule 2 (transition ordering). [^jm-grid]: **Jurafsky & Martin**, §22.3.2 — Entity Grid model (Barzilay & Lapata 2008): the sentence-by-entity grid with S/O/X/– roles from coreference and parsing; local transitions as features; dense vs. sparse columns; training by self-supervision to prefer natural over permuted orderings. [^jm-repl]: **Jurafsky & Martin**, §22.4 — Representation learning models for local coherence: lexical cohesion; lexical chains, TextTiling, and LSA coherence as average adjacent-sentence cosine; the self-supervised local coherence discriminator with its four sentence-pair features (concatenation, difference, absolute difference, element-wise product). [^jm-eval2]: **Jurafsky & Martin**, §22.3.3 — Evaluating Neural and Entity-based coherence: human ratings vs. self-supervision; the sentence-order discrimination, insertion, and reconstruction tasks.