---
title: "Relations, Events, and Templates"
module: Linguistic Structure
moduleNumber: 6
lessonNumber: 8
order: 608
summary: >
  Semantic roles answer "who did what" for one predicate; information extraction
  scales the idea to a whole corpus. This lesson turns unstructured text into
  structured data: relation extraction pulls entity-relation-entity triples out of
  sentences by patterns, supervision, and distant supervision; event and temporal
  extraction place those facts on a timeline; and template filling and knowledge-base
  population assemble them into a database a downstream system can query.
topics: [Structure]
sources:
  - book: Jurafsky
    ref: "Ch. 17 — Information Extraction; §17.1 Relation Extraction; §17.2 Relation Extraction Algorithms; §17.3 Extracting Times; §17.4 Extracting Events; §17.5 Template Filling"
---

This builds on [semantic roles](/natural-language-processing/linguistic-structure/semantic-roles-and-information-extraction),
which recovered who played what part in a single event. Information extraction takes
the same question — who did what to whom — and runs it across an entire corpus, so
the answers accumulate into a queryable store rather than a single predicate's
argument list.

## Information extraction

Semantic roles answer "who did what" for one predicate. **Information extraction
(IE)** scales the idea to a whole corpus: it turns the unstructured information in
text into structured data — populating a relational database or a **knowledge
graph** that downstream applications can query.[^jm-ie] The running example is a
news snippet about airlines:

> Citing high fuel prices, [ORG United Airlines] said [TIME Friday] it has
> increased fares by [MONEY $6] per round trip… [ORG American Airlines], a unit of
> [ORG AMR Corp.], immediately matched the move, spokesman [PER Tim Wagner] said.

**Named-entity recognition** — the
[sequence-labeling](/natural-language-processing/sequences/sequence-labeling) task
that finds and types the bracketed spans — is the first stage; IE builds on top of
it. Model-theoretically, NER is the identification of a class of _unary_ relations
(is-an-organization), and the relations we extract next are _binary_ ones: sets of
ordered tuples over the entities. From the snippet we recover that United is a unit
of UAL, that Tim Wagner works for American, and that United serves several cities.

$$
% caption: Information extraction as free text to a structured model. NER identifies
% the entities (unary relations / classes); relation extraction adds the binary
% relations. The right-hand side is a knowledge-graph fragment ready for a database.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  txt/.style={draw, fill=black!3, minimum width=34mm, minimum height=20mm, align=left, font=\scriptsize},
  ent/.style={draw=acc, text=acc, minimum width=15mm, minimum height=6mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % free text
  \node[txt] (T) at (0,0) {United Airlines\\said Friday it\\increased fares.\\American, a unit\\of AMR, matched.};
  \node[anchor=south, font=\scriptsize] at (0,1.15) {free text};
  \draw[->, acc, very thick] (2.1,0) -- (3.5,0) node[midway, above, font=\scriptsize] {IE};
  % knowledge graph
  \begin{scope}[xshift=5.3cm]
    \node[ent] (u)  at (0,1.4)   {United};
    \node[ent] (ual)at (2.6,1.4) {UAL};
    \node[ent] (am) at (0,-1.4)  {American};
    \node[ent] (amr)at (2.6,-1.4){AMR};
    \node[ent, draw=red, text=red] (tw) at (0,0) {Tim Wagner};
    \draw[->, black] (u) -- (ual)  node[midway, above, font=\scriptsize] {unit-of};
    \draw[->, black] (am) -- (amr) node[midway, below, font=\scriptsize] {unit-of};
    \draw[->, red]   (tw) -- (am)  node[midway, left, font=\scriptsize] {works-for};
  \end{scope}
  \node[anchor=south, font=\scriptsize] at (6.6,2.15) {knowledge graph};
\end{tikzpicture}
$$

These relations coincide with the model-theoretic tuples of formal semantics: a
relation $R$ is a set of ordered tuples over a domain $D$, $R \subseteq D \times D$.
_United is a unit of UAL_ is the pair $\langle a, b\rangle \in \texttt{PartOf}$;
_United serves Chicago, Dallas, Denver, San Francisco_ is four pairs in
$\texttt{Serves}$. Standard schemas exist: the ACE evaluation defines 17 relations
(part-whole, org-affiliation, physical, …); UMLS defines 54 medical relations
(_Injury disrupts Physiological-Function_); Wikipedia infoboxes yield millions of
facts, packaged as **RDF triples** $\langle \text{subject}, \text{predicate}, \text{object}\rangle$
(_Golden Gate Park — location — San Francisco_) in datasets like DBpedia and Freebase.

## Relation extraction

**Relation extraction** finds and classifies the semantic relations among the
entities in a text. There are five families of algorithm, trading annotation cost
against precision and recall.[^jm-re-algo]

### Patterns

The oldest method is hand-written **lexico-syntactic patterns**, introduced by
Hearst for the hyponym (is-a) relation. The pattern

$$
\text{NP}_0 \ \text{such as} \ \text{NP}_1 \{, \text{NP}_2 \dots (\text{and} \mid \text{or}) \, \text{NP}_i\}
$$

licenses the inference $\text{hyponym}(\text{NP}_i, \text{NP}_0)$, so _red algae
such as Gelidium_ yields $\text{hyponym}(\text{Gelidium}, \text{red algae})$. Other
**Hearst patterns** include _NP, including NP_ and _such NP as NP_. Adding named-
entity constraints tailors the idea to specific relations — _PER, POSITION of ORG_
matches _George Marshall, Secretary of State of the United States_. Patterns are
high-precision but low-recall, and building enough of them is laborious.

### Supervised learning

With a hand-annotated corpus, relation extraction becomes ordinary supervised
classification: find pairs of entities in a sentence, and classify the relation (if
any) that holds between them.

```algorithm
caption: $\textsc{Find-Relations}(\text{words})$ — classify every within-sentence entity pair
$\text{relations} \gets \varnothing$
$\text{entities} \gets \textsc{Find-Entities}(\text{words})$
for all entity pairs $\langle e_1, e_2 \rangle$ in $\text{entities}$ do
  if $\textsc{Related?}(e_1, e_2)$ then
    $\text{relations} \gets \text{relations} \cup \textsc{Classify-Relation}(e_1, e_2)$
return $\text{relations}$
```

An optional `Related?` gate is a cheap binary filter that skips unrelated pairs
before the expensive N-way classifier runs. Feature-based classifiers use the
headwords of the two mentions, the bag of words between them, their named-entity
types (`M1:ORG, M2:PER`), and the syntactic path between them. The neural version
delexicalizes and fine-tunes: feed the sentence to a pretrained encoder like BERT,
**replace each entity span with its NER tag** so the model cannot memorize specific
names, and put a linear classifier on top of the sentence representation.

$$
% caption: Neural relation extraction. The sentence is de-lexified by replacing the
% subject and object spans with their NER tags, encoded by BERT, and a linear layer
% over the sentence representation predicts the relation, yielding the triple
% $\langle \text{subject}, \text{relation}, \text{object}\rangle$.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  tok/.style={draw, minimum height=6mm, align=center, font=\scriptsize, inner sep=2pt},
  enc/.style={draw=acc, text=acc, thick, minimum width=68mm, minimum height=8mm, align=center, font=\small},
  lin/.style={draw, minimum width=30mm, minimum height=6mm, align=center, font=\scriptsize},
  outbox/.style={draw=acc, text=acc, minimum width=44mm, minimum height=7mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % de-lexified input
  \node[tok] (t1) at (-3.4,0) {[SUBJ-PER]};
  \node[tok, right=1.5mm of t1] (t2) {was};
  \node[tok, right=1.5mm of t2] (t3) {born};
  \node[tok, right=1.5mm of t3] (t4) {in};
  \node[tok, right=1.5mm of t4] (t5) {[OBJ-LOC]};
  \node[tok, right=1.5mm of t5] (t6) {, Michigan};
  \node[anchor=north, font=\scriptsize] at (0.7,-0.6) {de-lexif\/ied input};
  % encoder
  \node[enc] (E) at (0.7,1.3) {ENCODER (BERT)};
  \foreach \t in {t1,t2,t3,t4,t5,t6} \draw[->, acc] (\t.north) -- (E.south);
  % linear classifier
  \node[lin] (L) at (0.7,2.6) {Linear classif\/ier};
  \draw[->, acc] (E) -- (L);
  % output triple
  \node[outbox] (O) at (0.7,3.9) {triple: ([SUBJ], per:city-of-birth, [OBJ])};
  \draw[->, acc, thick] (L) -- (O);
\end{tikzpicture}
$$

Supervised systems are accurate when the test set resembles the training set, but
labeling is expensive and the models are brittle across genres — which motivates
the semi-supervised methods.

### Bootstrapping

**Bootstrapping** starts from a handful of high-precision **seed tuples** and grows
the relation itself. Given the seed _Ryanair has a hub at Charleroi_, it finds
sentences mentioning both entities, generalizes the surrounding context into
patterns (_[ORG], which uses [LOC] as a hub_), and greps for new tuples with those
patterns — which supply still more patterns, and so on.[^jm-boot]

$$
% caption: The bootstrapping loop. Seed tuples find sentences, sentences yield
% generalized patterns, patterns extract new tuples, and confidence filtering guards
% against semantic drift before the new tuples re-enter the loop.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  stepbox/.style={draw, minimum width=27mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[stepbox, draw=acc, text=acc] (seed) at (0,2.2)    {seed tuples\\(Ryanair, Charleroi)};
  \node[stepbox] (sent) at (5.4,2.2)  {f\/ind sentences\\with both entities};
  \node[stepbox] (pat)  at (5.4,-0.4) {generalize to\\patterns};
  \node[stepbox] (new)  at (0,-0.4)   {extract new\\tuples};
  \draw[->, acc, thick] (seed) -- (sent);
  \draw[->, acc, thick] (sent) -- (pat);
  \draw[->, acc, thick] (pat)  -- (new);
  \draw[->, acc, thick] (new)  -- (seed) node[midway, left, font=\scriptsize, red] {conf\/idence f\/ilter};
\end{tikzpicture}
$$

The danger is **semantic drift**: one bad pattern admits a wrong tuple (_Sydney has
a ferry hub at Circular Quay_ $\to \langle$Sydney, Circular Quay$\rangle$), which
spawns worse patterns, and the relation's meaning wanders. Bootstrapping systems
therefore attach **confidence values** to patterns and tuples. A pattern $p$'s
confidence trades its accuracy on known tuples against its productivity,
$\mathrm{Conf}(p) = \frac{\text{hits}_p}{\text{finds}_p} \cdot \log(\text{finds}_p)$
(hits among the tuples it matches, scaled by how many it finds), and evidence across
the patterns $P'$ supporting a tuple $t$ combines by a noisy-or,

$$
\mathrm{Conf}(t) = 1 - \prod_{p \in P'} \bigl(1 - \mathrm{Conf}(p)\bigr),
$$

so a tuple supported by several independent patterns is trusted more than one
resting on a single match. Conservative thresholds keep the system from drifting.

### Distant supervision

**Distant supervision** combines bootstrapping's use of seed facts with supervised
learning's rich features.[^jm-distant] Instead of a few seeds, it takes a large
database — Freebase has over 100,000 `place-of-birth` pairs — and, for every pair,
collects _every_ sentence in a big corpus that mentions both entities:

- _…Hubble was born in Marshfield…_
- _…Einstein, born (1879), Ulm…_
- _…Hubble's birthplace in Marshfield…_

Each occurrence becomes a training instance for the tuple's relation, and features
are pooled across _all_ the sentences for a tuple, yielding rich conjunctions like
`M1=PER & M2=LOC & nextword="born" & path=…`. A supervised classifier trained on
these, plus a `no-relation` class from random unrelated pairs, needs no hand-labeled
sentences at all.

$$
% caption: Distant supervision. A relation database provides many seed tuples; a NER
% pass over a large corpus finds every sentence mentioning a tuple's entities;
% features pooled across those sentences train a supervised classifier — no
% hand-labeled sentences required.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  box/.style={draw, minimum width=30mm, minimum height=12mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box, draw=acc, text=acc] (db)  at (0,0)   {relation DB\\(Freebase)};
  \node[box] (corp) at (4.2,1.3)  {corpus + NER\\f\/ind both entities};
  \node[box] (feat) at (4.2,-1.3) {pool features\\over all sentences};
  \node[box, draw=acc, text=acc] (clf) at (8.6,0) {supervised\\classif\/ier};
  \draw[->, acc, thick] (db) -- (corp) node[midway, above, font=\scriptsize, sloped] {tuples};
  \draw[->, black] (corp) -- (feat);
  \draw[->, acc, thick] (feat) -- (clf);
\end{tikzpicture}
$$

Distant supervision inherits the strengths of each parent — many features, high-
precision evidence, no genre-bound labeled corpus, and training tuples usable by
neural models — but it is limited to relations for which a database already exists,
and it tends toward lower precision. For relations with no database at all,
**unsupervised** methods (Open IE, e.g. the ReVerb system) extract relations as raw
strings — usually verb phrases — straight from the web:
$\langle$United, has a hub in, Chicago$\rangle$. Open IE handles an unbounded set
of relations but leaves the strings un-canonicalized.

| Method | What it needs | Precision | Recall / coverage |
| --- | --- | --- | --- |
| Patterns | hand-written rules | high | low |
| Supervised | labeled corpus | high (in-genre) | limited, brittle |
| Bootstrapping | a few seed tuples | drifts if unchecked | grows with iterations |
| Distant supervision | a relation database | moderate | broad, database-bound |
| Unsupervised / Open IE | nothing but text | lower | very broad, uncanonical |

## Events, times, and templates

Relations are static facts; a news story is also a sequence of **events** in time.
Three further IE tasks put events on a timeline.[^jm-events]

**Event extraction** finds mentions of events — expressions denoting something that
happened at a point or interval. Most events are verbs (_increased_, _matched_,
_said_), but noun phrases introduce events too (_the move_, _the increase_), and
some verbs do not (_took effect_ marks a boundary, not an event). Event extraction
is modeled as BIO sequence labeling plus a classifier for the event class
(occurrence, state, **reporting** event like _said_) and its tense and aspect.

**Temporal expressions** — _Friday_, _last week_, _two days from now_, _3:30 P.M._
— are extracted (again by BIO tagging over `TIMEX3`-delimited spans) and then
**normalized** to a standard form so a machine can compute with them. Normalization
uses the ISO 8601 calendar: a fully qualified date _April 24, 1916_ becomes
`1916-04-24`; a relative expression is resolved against the document's **temporal
anchor** (its dateline). If the article is dated `2007-07-02` (ISO week 27), then
_last week_ normalizes to `2007-W26` and _the weekend_ to the duration `P1WE`
anchored to that week. The TimeML scheme records these as XML attributes:

```
<TIMEX3 type="DATE"     value="2007-W26" anchorTimeID="t1"> last week </TIMEX3>
<TIMEX3 type="DURATION" value="P1WE"     anchorTimeID="t1"> the weekend </TIMEX3>
```

With events and times both tagged, a system can **order** them: the relation
between two events is one of Allen's 13 temporal relations (before, overlaps, during,
meets, …), classified by a feature-based model trained on the TimeBank corpus. The
result is a partial timeline — _the American fare increase came after United's_.

$$
% caption: A partial event timeline built from the airline story. Temporal
% expressions are normalized to ISO dates and events are ordered by Allen relations
% (here, before), turning the prose into a queryable sequence.
\begin{tikzpicture}[>=stealth, font=\scriptsize,
  ev/.style={draw, minimum width=30mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % timeline axis
  \draw[->, black, thick] (-0.4,0) -- (11.2,0);
  \node[anchor=west, black, font=\scriptsize] at (11.0,-0.35) {time};
  % ticks
  \foreach \x/\d in {1.2/Thursday, 5.4/Friday, 9.4/weekend} {
    \draw[black] (\x,0.12) -- (\x,-0.12);
    \node[anchor=north, black, font=\scriptsize] at (\x,-0.18) {\d};
  }
  % events above
  \node[ev, draw=acc, text=acc] (e1) at (1.2,1.2)  {United raises fare\\(ef\/fective)};
  \node[ev] (e2) at (5.4,1.2)  {United announces};
  \node[ev] (e3) at (9.4,1.2)  {American matches};
  \draw[->, black] (1.2,0.05) -- (e1.south);
  \draw[->, black] (5.4,0.05) -- (e2.south);
  \draw[->, black] (9.4,0.05) -- (e3.south);
  \draw[->, acc] (e1.east) to[out=0,in=180] node[midway, above, font=\scriptsize] {before} (e2.west);
  \draw[->, acc] (e2.east) to[out=0,in=180] node[midway, above, font=\scriptsize] {before} (e3.west);
\end{tikzpicture}
$$

### Template filling and knowledge-base population

Many texts describe **stereotypical situations** — a **script**, in the older
terminology — whose structure we know in advance. A fare raise, for instance, has a
lead airline, an amount, an effective date, and a follower. **Template filling**
finds documents that invoke such a script and fills a fixed set of slots:

```
FARE-RAISE-ATTEMPT:
  LEAD-AIRLINE:   United Airlines
  AMOUNT:         $6
  EFFECTIVE-DATE: 2006-10-26
  FOLLOWER:       American Airlines
```

The standard approach trains two systems: a **template recognizer** (a text
classifier deciding whether the script is present) and, for each slot, a
**role-filler extractor** (a classifier or sequence model that pulls the slot's
value). Multiple mentions of one filler (_United_, _United Airlines_) are reconciled
by coreference. Older systems like FASTUS handled richer, hierarchically nested
templates with cascades of finite-state transducers over hand-written rules.

All of this feeds the largest goal, **knowledge-base population**: the tuples,
events, and filled templates accumulate into a structured store — a relational
database or knowledge graph — that a later system can query. **Knowledge-based
[question answering](/natural-language-processing/applications/question-answering)**
runs against that store: a question is parsed into a query over the relations IE deposited,
and the answer is looked up rather than read off a passage. Semantic roles told us
who did what in one sentence; information extraction wrote it all down in a form a
machine can reason over.
[^jm-ie]: **Jurafsky & Martin**, Ch. 17 intro; §17.1 — information extraction as text-to-structure, NER as unary relations, relations as model-theoretic tuples, ACE / UMLS schemas, and RDF triples from Wikipedia infoboxes.
[^jm-re-algo]: **Jurafsky & Martin**, §17.2.1–17.2.2, §17.2.5 — the five families of relation-extraction algorithm; Hearst lexico-syntactic patterns; supervised feature-based and neural (BERT, NER-delexified) classifiers; and Open IE.
[^jm-boot]: **Jurafsky & Martin**, §17.2.3 — bootstrapping from seed tuples, pattern generalization, semantic drift, and confidence values combined by the noisy-or model.
[^jm-distant]: **Jurafsky & Martin**, §17.2.4 — distant supervision: a large relation database supplies many tuples, features are pooled across every matching sentence, and a supervised classifier (plus a `no-relation` class) is trained without hand-labeled sentences.
[^jm-events]: **Jurafsky & Martin**, §17.3–17.5 — temporal expression extraction and ISO 8601 normalization (TimeML `TIMEX3`, temporal anchor), event extraction, Allen temporal relations and TimeBank ordering, template filling, and knowledge-base population.