--- title: "Extracting Times, Events, and Templates" module: Linguistic Structure moduleNumber: 6 lessonNumber: 16 order: 616 summary: > Once relation extraction has produced typed triples, the information-extraction pipeline still has to place facts in time and assemble them into records. This lesson detects and normalizes temporal expressions to ISO 8601 values, detects events and orders them on a timeline with the 13 Allen relations, and fills slot-and-filler templates — flat and hierarchical — for stereotyped situations, closing the loop from text to a queryable database. topics: [Structure] sources: - book: Jurafsky ref: "§17.3 Extracting Times; §17.4 Extracting Events and their Times; §17.5 Template Filling" --- This builds on [information extraction: relation extraction](/natural-language-processing/linguistic-structure/information-extraction), which turned tagged entities into relation triples. Relations are only part of what a database stores: facts also have to be **placed in time** and gathered into **records**. Here we detect and normalize temporal expressions, order events on a timeline, and fill templates for recurring situations — the stages that finish the pipeline from free text to a queryable database. ## Extracting times Times and dates are a special, important kind of entity for question answering and calendar assistants. To reason about them, extracted **temporal expressions** must be **normalized** — converted to a standard format.[^jm-times] Temporal expressions come in three flavors. **Absolute** expressions map directly to a calendar date or clock time (_April 24, 1916_; _10:15 AM_). **Relative** expressions locate a time only through a reference point (_yesterday_; _next semester_; _two weeks from yesterday_). **Durations** denote a span at some granularity (_four hours_; _three weeks_; _the last three quarters_). $$ % caption: The three kinds of temporal expression. Absolute times fix a calendar % point; relative times need a reference; durations name a span. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \node[font=\scriptsize] at (0,2.1) {\textbf{Absolute}}; \node[font=\scriptsize] at (4.0,2.1) {\textbf{Relative}}; \node[font=\scriptsize] at (8.4,2.1) {\textbf{Durations}}; \draw[black] (-1.5,1.8) -- (10.3,1.8); \node[font=\scriptsize] at (0,1.3) {April 24, 1916}; \node[font=\scriptsize] at (4.0,1.3) {yesterday}; \node[font=\scriptsize] at (8.4,1.3) {four hours}; \node[font=\scriptsize] at (0,0.7) {10:15 AM}; \node[font=\scriptsize] at (4.0,0.7) {next semester}; \node[font=\scriptsize] at (8.4,0.7) {three weeks}; \node[font=\scriptsize] at (0,0.1) {The summer of '77}; \node[font=\scriptsize] at (4.0,0.1) {two weeks from yesterday}; \node[font=\scriptsize] at (8.4,0.1) {the last three quarters}; \end{tikzpicture} $$ Temporal expressions have **lexical triggers** as their heads — nouns (_morning_, _noon_, _winter_), proper nouns (_January_, _Monday_, _Ramadan_), adjectives (_recent_, _annual_), and adverbs (_hourly_, _daily_) — projected into full noun, adjective, and adverbial phrases. ### Recognition and TIMEX3 The **TimeML** annotation scheme wraps each temporal expression in a `TIMEX3` XML tag. Recognition — finding the start and end of each such span — is done either by **rule-based** cascades of automata (POS-tag, then chunk by trigger-word patterns) or by **sequence labeling** with the same IOB scheme used for named entities: $$ % caption: IOB tagging of a temporal expression. Tokens inside a TIMEX3 span get % I or B tags, everything else O, exactly as in named-entity recognition. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \foreach \w/\t/\x in {A/O/0, fare/O/1.2, increase/O/2.6, initiated/O/4.2, last/B/5.6, week/I/6.6, by/O/7.6, UAL/O/8.4}{ \node[font=\scriptsize] at (\x,0.4) {\w}; \node[font=\footnotesize\ttfamily, text=acc] at (\x,-0.2) {\t}; } \end{tikzpicture} $$ A difficulty unique to this task is avoiding false positives from expressions that _look_ temporal but are not — _1984_ in _1984 tells the story of Winston Smith_, or _Sunday_ in _U2's classic Sunday Bloody Sunday_. Recognizers are scored with the usual precision, recall, and F-measure. ### Temporal normalization **Temporal normalization** maps a recognized expression to a point or a duration in the **ISO 8601** standard. For example, take a document dated _July 2, 2007_ (its `CREATION_TIME`) containing _A fare increase initiated last week ... over the weekend, marking the second successful fare increase in two weeks_. $$ % caption: Worked temporal normalization. Each expression is anchored to the July 2, % 2007 document date and mapped to an ISO 8601 value: a date, an ISO week, or a % PnU duration. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \node[font=\scriptsize] at (-0.2,3.0) {\textbf{Expression}}; \node[font=\scriptsize] at (3.4,3.0) {\textbf{Type}}; \node[font=\scriptsize] at (6.0,3.0) {\textbf{ISO value}}; \node[font=\scriptsize] at (9.2,3.0) {\textbf{Why}}; \draw[black] (-2.2,2.7) -- (11.4,2.7); \node[anchor=west, font=\scriptsize] at (-2.2,2.2) {July 2, 2007 (dateline)}; \node[anchor=west, font=\scriptsize] at (3.0,2.2) {DATE}; \node[anchor=west, font=\scriptsize, text=acc] at (5.5,2.2) {2007-07-02}; \node[anchor=west, font=\scriptsize] at (8.2,2.2) {YYYY-MM-DD}; \node[anchor=west, font=\scriptsize] at (-2.2,1.5) {last week}; \node[anchor=west, font=\scriptsize] at (3.0,1.5) {DATE}; \node[anchor=west, font=\scriptsize, text=acc] at (5.5,1.5) {2007-W26}; \node[anchor=west, font=\scriptsize] at (8.2,1.5) {week 27 minus 1}; \node[anchor=west, font=\scriptsize] at (-2.2,0.8) {the weekend}; \node[anchor=west, font=\scriptsize] at (3.0,0.8) {DURATION}; \node[anchor=west, font=\scriptsize, text=acc] at (5.5,0.8) {P1WE}; \node[anchor=west, font=\scriptsize] at (8.2,0.8) {one weekend}; \node[anchor=west, font=\scriptsize] at (-2.2,0.1) {two weeks}; \node[anchor=west, font=\scriptsize] at (3.0,0.1) {DURATION}; \node[anchor=west, font=\scriptsize, text=acc] at (5.5,0.1) {P2W}; \node[anchor=west, font=\scriptsize] at (8.2,0.1) {two weeks}; \draw[black] (-2.2,-0.25) -- (11.4,-0.25); \end{tikzpicture} $$ Several conventions are at work. ISO **weeks** run `YYYY-Wnn`, numbered 01-53 with week 01 the one holding the year's first Thursday; the document date falls in week 27, so _last week_ normalizes to `2007-W26`. **Durations** use the pattern `PnU` — $n$ an integer, $U$ a unit — so `P3Y` is three years, `P2W` two weeks, and the weekend is `P1WE`. When a duration is also anchored to a specific week, an `anchorTimeID` attribute records it. Fully qualified expressions (a year, month, and day) are handled by compositional rules with temporal arithmetic: a pattern $\textit{FQTE} \to \textit{Month Date}, \textit{Year}$ computes its value from the values of its parts. Most news expressions are only implicitly anchored to the **temporal anchor** $A$ — the document date. Relative expressions resolve by offset arithmetic from $A$: _today_ $\mapsto A$, _tomorrow_ $\mapsto A + 1\,\text{day}$, _yesterday_ $\mapsto A - 1\,\text{day}$ (modulo for month and clock rollover). Even _the weekend_ is subtle: it normally refers to the weekend just past, but tense can flip it forward — _Random security checks that began yesterday ... will continue at least through the weekend_ points at the _coming_ weekend, signaled by _continue_. Most current normalizers are rule-based, pairing each matching pattern with a semantic procedure. ## Extracting events and their times **Event extraction** identifies mentions of events — any expression denoting an event or state that can be located at a point or interval in time. Marking up the airline text tags many events: _citing_, _said_, _increased_, _matched_, _the move_, _took effect_, _applies_, _competes_.[^jm-events] Most event mentions are verbs, but not all: events can be introduced by noun phrases (_the move_, _the increase_), and some verbs fail to introduce events — the phrasal verb _took effect_ refers to _when_ an event began, and light verbs like _make_, _take_, _have_ push the event onto their nominal object (_took a flight_). Each event mention $e$ is annotated with a class and attributes, $\langle \text{class}, \text{tense}, \text{aspect}, \text{modality} \rangle$; a _said_ event is $\langle \texttt{REPORTING}, \texttt{PAST}, \texttt{PERFECTIVE}, - \rangle$. The TempEval classes group event predicates by function: | Class | Trigger examples | | --- | --- | | Reporting | _say_, _report_, _tell_, _explain_ | | Perception | _see_, _hear_, _watch_ | | Aspectual | _begin_, _continue_, _stop_ | | State | _love_, _resemble_, _cost_ | | Occurrence | _increase_, _match_, _take effect_ | Event extraction is supervised: sequence models with IOB tagging detect events, and multi-class classifiers assign class and attributes, using features like part-of-speech, nominalization suffixes (_-tion_), light-verb flags, morphological stem, and WordNet hypernyms. ### Temporal ordering and Allen relations With events and times both detected, the next task is to fit the events onto a timeline. The full timeline is beyond current systems, but a useful weaker goal is a **partial ordering** — for instance, that _American Airlines_' fare increase came _after_ United's. Determining such an ordering is a binary relation-classification task, deciding which of the standard set of **Allen relations** holds between two intervals. $$ % caption: The 13 Allen relations between two time intervals A and B, from A % strictly before B, through overlaps, meets, starts, finishes, and during, to A % equal to B. Each names one way two intervals can sit on the timeline. \begin{tikzpicture}[>=stealth, font=\footnotesize, ev/.style={draw, fill=acc!10, minimum height=4mm, inner sep=2pt, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % before / after \node[ev, minimum width=12mm] (a1) at (0,3.3) {A}; \node[ev, minimum width=12mm] (b1) at (2.1,3.0) {B}; \node[font=\scriptsize, anchor=west] at (3.4,3.15) {A before B}; % meets \node[ev, minimum width=12mm] (a2) at (0,2.3) {A}; \node[ev, minimum width=12mm] (b2) at (1.35,2.0) {B}; \node[font=\scriptsize, anchor=west] at (3.4,2.15) {A meets B}; % overlaps \node[ev, minimum width=14mm] (a3) at (0,1.3) {A}; \node[ev, minimum width=14mm] (b3) at (0.9,1.0) {B}; \node[font=\scriptsize, anchor=west] at (3.4,1.15) {A overlaps B}; % starts \node[ev, minimum width=10mm] (a4) at (-0.1,0.3) {A}; \node[ev, minimum width=16mm] (b4) at (0.2,0.0) {B}; \node[font=\scriptsize, anchor=west] at (3.4,0.15) {A starts B}; % during \node[ev, minimum width=8mm] (a5) at (0.2,-0.7) {A}; \node[ev, minimum width=18mm] (b5) at (0.2,-1.0) {B}; \node[font=\scriptsize, anchor=west] at (3.4,-0.85) {A during B}; % finishes \node[ev, minimum width=10mm] (a6) at (0.5,-1.7) {A}; \node[ev, minimum width=16mm] (b6) at (0.2,-2.0) {B}; \node[font=\scriptsize, anchor=west] at (3.4,-1.85) {A finishes B}; % equals \node[ev, minimum width=14mm] (a7) at (0,-2.7) {A}; \node[ev, minimum width=14mm] (b7) at (0,-3.0) {B}; \node[font=\scriptsize, anchor=west] at (3.4,-2.85) {A equals B}; \draw[->, black] (-1.2,-3.5) -- (7.0,-3.5) node[right, font=\scriptsize] {time}; \end{tikzpicture} $$ An event is an interval $A = [A^-, A^+]$ with $A^- < A^+$; each Allen relation is a constraint on the four endpoint orderings (e.g. $A \text{ before } B \iff A^+ < B^-$; $A \text{ meets } B \iff A^+ = B^-$). The 13 relations are the six asymmetric ones (before, meets, overlaps, starts, during, finishes), their inverses, and equals. These relations are classified with feature-based classifiers trained on the **TimeBank** corpus — 183 news articles annotated with events, times, and the temporal links between them, using TimeML. A TimeBank sentence like _Delta Air Lines earnings soared 33% to a record in the fiscal first quarter, bucking the industry trend toward declining profits_ carries three events (_soared_, _bucking_, _declining_) and two times, plus links such as: soaring is _included_ in the first quarter; soaring is _before_ 1989-10-26; soaring is _simultaneous_ with bucking; declining _includes_ soaring. Classifier features include words/embeddings, parse paths, tense, and aspect. ## Template filling Many texts describe recurring, stereotyped situations — a **script**, in the sense of a prototypical sequence of sub-events with participants and roles. The strong expectations a script provides let a system classify entities, assign them to roles, and infer facts left unsaid. In their simplest form scripts become **templates**: a fixed set of **slots** $\{r_1, \dots, r_m\}$, each slot $r_k$ constrained to fillers of a type $\tau_k$. Filling a template means selecting a text span (or normalized value) $v_k$ with $\text{type}(v_k) = \tau_k$ for each slot. **Template filling** finds documents that invoke a template and fills its slots from the text.[^jm-templates] Our airline story is a stereotyped fare-raise situation. Reading it, we identify United as the lead airline that first raised fares, $6 as the amount, Thursday as the effective date, and American as a follower — a filled template: $$ % caption: A filled template for the fare-raise situation. Each slot takes a % typed filler extracted from the text; effective date is normalized to ISO. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \node[draw, fill=acc!6, minimum width=95mm, minimum height=34mm] (box) at (0,0) {}; \node[font=\scriptsize, anchor=west] at (-4.6,1.2) {\textbf{FARE-RAISE ATTEMPT}}; \draw[black] (-4.6,0.85) -- (4.4,0.85); \node[font=\scriptsize, anchor=west] at (-4.4,0.35) {LEAD AIRLINE:}; \node[font=\scriptsize, anchor=west, text=acc] at (-0.6,0.35) {United Airlines}; \node[font=\scriptsize, anchor=west] at (-4.4,-0.15) {AMOUNT:}; \node[font=\scriptsize, anchor=west, text=acc] at (-0.6,-0.15) {USD 6}; \node[font=\scriptsize, anchor=west] at (-4.4,-0.65) {EFFECTIVE DATE:}; \node[font=\scriptsize, anchor=west, text=acc] at (-0.6,-0.65) {2006-10-26}; \node[font=\scriptsize, anchor=west] at (-4.4,-1.15) {FOLLOWER:}; \node[font=\scriptsize, anchor=west, text=acc] at (-0.6,-1.15) {American Airlines}; \end{tikzpicture} $$ The task is modeled with two supervised systems. **Template recognition** (also, confusingly, called _event recognition_) decides whether the template is present in a sentence — a text-classification task over the labeled spans. **Role-filler extraction** trains a separate classifier per role (LEAD-AIRLINE, AMOUNT, and so on), run over noun phrases or as a sequence model, to detect each filler. Both use the usual features: tokens, embeddings, word shapes, POS, chunk tags, and named entities. When several text segments carry the same slot label — _United_ and _United Airlines_ both label LEAD-AIRLINE — coreference resolution reconciles them. ### Complex and hierarchical templates The fare-raise template is flat. Consider instead a joint-venture report: _Bridgestone Sports Co. said Friday it has set up a joint venture in Taiwan ... to produce golf clubs ... Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990_. Capturing all of this needs a **hierarchical** template, where one slot's filler is _itself_ a template. $$ % caption: A hierarchical joint-venture template: the ACTIVITY slot of the tie-up % record is filled by a nested activity template with its own company, product, % and start-date slots. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \node[draw, fill=acc!6, minimum width=52mm, minimum height=30mm] (t1) at (-3.0,0) {}; \node[font=\scriptsize, anchor=west] at (-5.3,1.15) {\textbf{Tie-up-1}}; \draw[black] (-5.3,0.8) -- (-0.7,0.8); \node[font=\scriptsize, anchor=west] at (-5.2,0.35) {RELATIONSHIP: tie-up}; \node[font=\scriptsize, anchor=west] at (-5.2,-0.1) {ENTITIES: Bridgestone, ...}; \node[font=\scriptsize, anchor=west] at (-5.2,-0.55) {JOINT VENTURE: BST Co.}; \node[font=\scriptsize, anchor=west, text=acc] at (-5.2,-1.0) {ACTIVITY: Activity-1}; \node[draw, fill=acc!12, minimum width=48mm, minimum height=26mm] (t2) at (4.0,-0.2) {}; \node[font=\scriptsize, anchor=west, text=acc] at (2.0,0.85) {\textbf{Activity-1}}; \draw[acc!40] (2.0,0.5) -- (6.0,0.5); \node[font=\scriptsize, anchor=west] at (2.1,0.1) {COMPANY: BST Taiwan Co.}; \node[font=\scriptsize, anchor=west] at (2.1,-0.35) {PRODUCT: golf clubs}; \node[font=\scriptsize, anchor=west] at (2.1,-0.8) {START DATE: January 1990}; \draw[->, acc, thick] (-1.0,-1.0) to[bend left=10] (2.0,-0.3); \end{tikzpicture} $$ Early systems for such templates used cascades of finite-state transducers over hand-built rules. The FASTUS pipeline ran six stages, each extracting one kind of information and passing it up: | Stage | Output | | --- | --- | | 1. Tokens | tokenized stream | | 2. Complex words | multiword names, numbers | | 3. Basic phrases | noun and verb groups | | 4. Complex phrases | attached modifiers, prepositional phrases | | 5. Semantic patterns | entities and events inserted into templates | | 6. Merging | coreference-resolve references to the same entity/event | Modern learning-based systems handle flat templates well; the complex, hierarchical case remains partly the province of these earlier rule-based systems, and recent work induces templates automatically as sets of linked events. ## From text to a database The through-line is the pipeline that opened the [first lesson](/natural-language-processing/linguistic-structure/information-extraction): unstructured text becomes structured records. Relations among entities come from pattern-based methods, supervised classifiers when labels exist, semi-supervised bootstrapping and distant supervision when they are scarce, and unsupervised Open IE when the relations themselves are unknown. Temporal expressions are detected and normalized to ISO values so a system can compute with them. Events are detected and ordered on a timeline with Allen relations. Template filling assembles the pieces into slot-and-filler records for stereotyped situations. Every stage starts from tagged entities, so IE sits directly downstream of [sequence labeling](/natural-language-processing/sequences/sequence-labeling): named-entity recognition supplies the arguments that relation, event, and template extraction connect. And it sits directly upstream of applications that need a knowledge base. Those triples and templates arrive already in the form of rows a knowledge base stores, which is why IE is the standard method for **populating** one — and a populated knowledge base is what lets a [question-answering](/natural-language-processing/applications/question-answering) system look up _who founded Apple_ or _when did the fare increase take effect_ without re-reading the source text. [^jm-times]: **Jurafsky & Martin**, Ch. 17, §17.3 — Extracting Times: absolute, relative, and duration temporal expressions; TimeML and TIMEX3 recognition; and normalization to ISO 8601 values anchored to the document date. [^jm-events]: **Jurafsky & Martin**, Ch. 17, §17.4 — Extracting Events and their Times: event detection and classification, the TimeBank corpus, and temporal ordering via the 13 Allen relations between intervals. [^jm-templates]: **Jurafsky & Martin**, Ch. 17, §17.5 — Template Filling: scripts and templates for stereotyped situations, template recognition and role-filler extraction, and the earlier finite-state (FASTUS) approach to hierarchical templates.