---
title: "Question Answering: Knowledge Bases and Language Models"
module: Applications
moduleNumber: 7
lessonNumber: 4
order: 704
summary: >
  The retrieve-and-read pipeline extracts an answer span from prose, but not all
  knowledge lives in prose. This part covers the rest of the QA stack: entity
  linking (Wikification) that grounds a question's entities to a knowledge base,
  knowledge-based QA by semantic parsing a question into an executable query,
  and the modern default — closed-book QA and retrieval-augmented generation
  with a large language model — closing on the DPR/RAG/fusion-in-decoder lineage
  and how factoid answers are scored by exact match and F1.
topics: [Applications]
sources:
  - book: Jurafsky
    ref: "§23.3 Entity Linking; §23.4 Knowledge-based Question Answering"
  - book: Jurafsky
    ref: "§23.5 Using Language Models to do QA; §23.2 (factoid answer evaluation)"
---

This builds on [Question Answering](/natural-language-processing/applications/question-answering),
which developed the retrieve-and-read pipeline: an information retriever (sparse
tf-idf/BM25 or a dense bi-encoder) narrows a collection to a few passages, and a
BERT reader extracts the answer span. That covers answers that sit verbatim in some
document. This part handles the two cases it does not — answers that live in a
structured knowledge base, and answers a language model can generate directly — and
ends with how factoid answers are scored.

## Entity linking and Wikification

Before a question can be answered from a structured knowledge base, the system has
to know _which_ entities the question is about. When the question mentions _Ada
Lovelace_, some component must map that string of characters to the specific
real-world person — the row in the database, the node in the graph, the page in
Wikipedia. That is **entity linking**: associating a **mention** in text with the
representation of a real-world entity in an **ontology**.[^jm-el] Every
knowledge-based QA system depends on it.

The most common ontology for factoid QA is **Wikipedia**, where each page stands
for one entity and doubles as its unique id. Deciding which Wikipedia page a mention
refers to has its own name: **Wikification**. Since the earliest systems, entity
linking has run in two stages: **mention detection**, finding the spans of text that
name entities, and **mention disambiguation**, choosing the right entity for each
ambiguous span. We follow the classic TAGME baseline, which links to Wikipedia using
an anchor dictionary and the link structure of the encyclopedia itself.

$$
% caption: The entity-linking pipeline. Mention detection finds candidate spans in
% the question; candidate generation looks each span up in an anchor dictionary to
% get possible Wikipedia entities; disambiguation ranks them by prior probability
% and coherence to pick one entity per mention.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (m) at (0,0) {mention\\detection};
  \node[box] (c) at (4.0,0) {candidate\\generation};
  \node[box, draw=acc, text=acc, thick] (d) at (8.0,0) {disambiguation\\(rank + prune)};
  \node[align=center, font=\footnotesize] (e) at (11.6,0) {linked\\entity};
  \draw[->, acc, thick] (m) -- (c);
  \draw[->, acc, thick] (c) -- (d);
  \draw[->, thick] (d) -- (e);
  \node[font=\scriptsize, anchor=north, align=center] at (0,-0.9) {spans in\\text};
  \node[font=\scriptsize, anchor=north, align=center] at (4.0,-0.9) {anchor\\dictionary};
  \node[font=\scriptsize, anchor=north, align=center] at (8.0,-0.9) {prior +\\relatedness};
\end{tikzpicture}
$$

### Mention detection and candidate generation

TAGME starts offline by building two resources over a Wikipedia dump. For every
page $e$ it counts $\text{in}(e)$, the number of other Wikipedia pages that link to
$e$ — a measure of the entity's prominence. It also builds an **anchor
dictionary**: for each page, its title plus every **anchor text** — the hyperlinked
span used on other pages to point at it. The Stanford University page is pointed to
by anchors like _Stanford_ and _Stanford University_. For each anchor string $a$ it
records how often $a$ occurs at all, $\text{freq}(a)$, how often it occurs _as a
link_, $\text{link}(a)$, and the ratio

$$
\text{linkprob}(a) = \frac{\text{link}(a)}{\text{freq}(a)},
$$

the probability that an occurrence of $a$ is being used as a link — a rough gauge of
whether $a$ names an entity at all.

Given a question, **mention detection** queries the anchor dictionary for every
token sequence up to six words long, then prunes the large candidate set with simple
heuristics (drop substrings with tiny linkprob). For _When was Ada Lovelace born?_
the span _Ada Lovelace_ surfaces as a strong anchor, _Ada_ as a weaker one, while
_Lovelace_ is pruned for low linkprob and _born_ never enters the dictionary at all.
Each surviving mention span $a$ carries its set $E(a)$ of **candidate entities** —
the Wikipedia pages that anchor $a$ has ever linked to.

### Disambiguation: prior and coherence

If a span points to only one entity, linking is done. Most interesting spans are
ambiguous, and TAGME ranks $E(a)$ by two factors.[^jm-tagme] The first is the
**prior probability** $p(e \mid a)$, how often the anchor $a$ links to entity $e$
relative to all of $a$'s link uses:

$$
\text{prior}(a \rightarrow e) = p(e \mid a) = \frac{\text{count}(a \rightarrow e)}{\text{link}(a)}.
$$

The prior alone is not enough. In _What Chinese Dynasty came before the Yuan?_, the
span _Yuan_ most often links to the Chinese currency, not the dynasty — so
$p(e \mid a)$ would pick the wrong entity. The second factor,
**relatedness** (coherence), corrects this: the correct entity should cohere with
the _other_ entities in the question. Here the question also contains _Chinese Dynasty_, which
links reliably to a page on Chinese dynasties, and that page shares many in-links
with _Yuan dynasty_ — pulling the ambiguous _Yuan_ toward the right reading. Two
entities count as related to the degree their Wikipedia pages share in-links:

$$
\text{rel}(A, B) = \frac{\log\bigl(\max(|\text{in}(A)|, |\text{in}(B)|)\bigr) - \log\bigl(|\text{in}(A) \cap \text{in}(B)|\bigr)}{\log(|W|) - \log\bigl(\min(|\text{in}(A)|, |\text{in}(B)|)\bigr)},
$$

where $\text{in}(x)$ is the set of pages linking to $x$ and $W$ is the whole
collection. Every other mention $b$ in the question casts a **vote** for a candidate
$X$, averaging $X$'s relatedness to $b$'s own candidates, weighted by their priors,
and the total relatedness of $a \rightarrow X$ sums those votes:

$$
\text{vote}(b, X) = \frac{1}{|E(b)|} \sum_{Y \in E(b)} \text{rel}(X, Y)\, p(Y \mid b),
\qquad
\text{relatedness}(a \rightarrow X) = \sum_{b} \text{vote}(b, X).
$$

To choose the entity for span $a$, TAGME takes the candidates with the highest
relatedness, keeps those within a small margin of the top, and among _those_ picks
the one with the highest prior $p(X \mid a)$ — coherence first, prior as tie-breaker.
A final pruning step discards spurious links whose combined linkprob-and-coherence
score falls below a held-out threshold $\lambda$, so a marginal mention is dropped
rather than linked to a doubtful entity.

$$
% caption: Disambiguating the span "Yuan". Prior probability alone favors the
% currency; but "Chinese Dynasty" elsewhere in the question votes, through shared
% Wikipedia in-links (relatedness), for "Yuan dynasty", which wins on coherence.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  span/.style={draw, minimum width=26mm, minimum height=9mm, align=center},
  cand/.style={draw, minimum width=30mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[span] (yuan) at (0,0.7) {span: Yuan};
  \node[cand, draw=red, text=red] (cur) at (4.6,1.6) {Yuan (currency): high prior};
  \node[cand, draw=acc, text=acc] (dyn) at (4.6,0.4) {Yuan dynasty: high coherence};
  \node[span] (ctx) at (0,-1.5) {span: Chinese Dynasty};
  \draw[->, black] (yuan.east) -- (cur.west);
  \draw[->, black] (yuan.east) -- (dyn.west);
  \draw[->, acc, thick] (ctx.east) to[out=0, in=210] (dyn.west) node[midway, below, font=\scriptsize, text=acc] {votes (shared in-links)};
  \node[anchor=west, font=\scriptsize, text=acc] at (6.55,0.4) {$\gets$ chosen};
\end{tikzpicture}
$$

### Neural linking and the role in KB-QA

Modern systems replace the anchor dictionary with a **bi-encoder**, the same
approach the reader uses for span extraction.[^jm-elq] A system like ELQ runs the question
through [BERT](/natural-language-processing/transformers/large-language-models),
scores each span $[i,j]$ for being a mention (from learned start, end, and
mention-token vectors), and separately encodes every Wikipedia entity from its title
and the first tokens of its page, taking the `[CLS]` output as the entity vector.
A mention span is linked to the entity whose vector has the highest dot product with
the span's, softmaxed over all entities:

$$
p(e \mid [i,j]) = \frac{\exp\bigl(s(e, [i,j])\bigr)}{\sum_{e'} \exp\bigl(s(e', [i,j])\bigr)}.
$$

Because the entity vectors are precomputed and cached, linking a new question only
requires encoding the question. For QA, entity linking supplies
the **subject** of the knowledge-base query. In the semantic-parsing pipeline below,
_Ada Lovelace_ is linked to its canonical KB entity, and a separate relation
detector supplies the predicate — together they specify the triple that holds the
answer. Wikification also grounds a system's answers, letting it cite the Wikipedia
page an entity came from. Without entity linking, a knowledge-based system would have
a relation but no entity to apply it to.

## Knowledge-based QA

Not all knowledge lives in prose. A great deal sits in structured databases and
knowledge bases as clean facts, and **knowledge-based QA** answers a question by
translating it into a query over that structure rather than reading text.[^jm-kb]
A knowledge base is often a set of **RDF triples**, each a _(subject, predicate,
object)_ tuple asserting one relation:

| subject | predicate | object |
| --- | --- | --- |
| Ada Lovelace | birth-year | 1815 |

This single triple answers _"When was Ada Lovelace born?"_ and _"Who was born in
1815?"_. The task is to map the question onto the right triple pattern.

### Semantic parsing

The general method is **semantic parsing**: map the natural-language question to a
logical form — a query in SQL or SPARQL, a lambda-calculus expression, or another
executable program — and run it against the store. For a simple relational
question this decomposes into two steps that mirror the parts of the triple:
**entity linking** identifies the subject (mapping the string _Ada Lovelace_ to
its canonical KB entity) and **relation detection** identifies the predicate
(mapping _"When was ... born?"_ to the relation `birth-year`).

$$
% caption: Knowledge-based QA by semantic parsing. The question is parsed into an
% executable query — here a relation applied to a linked entity with a variable —
% which is run against the knowledge base to return the answer.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=30mm, minimum height=11mm, align=center},
  db/.style={draw, cylinder, shape aspect=0.3, minimum width=14mm, minimum height=17mm, align=center, font=\scriptsize, shape border rotate=90}]
  \definecolor{acc}{HTML}{2348F2}
  \node[align=center, font=\footnotesize] (q) at (0,0) {"When was\\Ada Lovelace born?"};
  \node[box, draw=acc, text=acc, thick] (parse) at (4.0,0) {semantic\\parser};
  \node[box, font=\ttfamily\footnotesize] (lf) at (8.4,0) {birth-year\\(Ada Lovelace, ?x)};
  \node[db] (kb) at (8.4,-2.5) {KB\\triples};
  \node[align=center, font=\footnotesize] (a) at (12.3,0) {?x = 1815};
  \draw[->, thick] (q) -- (parse);
  \draw[->, acc, thick] (parse) -- (lf);
  \draw[->, thick] (kb) -- (lf);
  \draw[->, thick] (lf) -- (a);
\end{tikzpicture}
$$

Modern relation detection reuses the same encoder trick as the reader: run the
question through BERT, take the `[CLS]` output $m_r$ as the question's
representation, learn a vector $w_{r_i}$ per candidate relation, and pick the
relation by a softmax over dot products,

$$
p(r_i \mid q) = \frac{\exp(m_r \cdot w_{r_i})}{\sum_k \exp(m_r \cdot w_{r_k})}.
$$

For richer questions the logical form is a full program. A semantic parser can
emit a lambda-calculus predicate, a multi-clause SQL query joining several tables,
or a SPARQL query over a graph — a sequence-to-sequence model with a BERT encoder
maps question tokens to logical-form tokens. Compared with reading text, the
appeal of KB-QA is precision: the answer is computed from curated facts, not
guessed from prose. Its limit is coverage — a fact absent from the knowledge base
is simply unanswerable.

## QA with large language models

A pretrained [large language model](/natural-language-processing/transformers/large-language-models)
has already been trained on enormous amounts of text and stored a great deal of it
in its weights. This suggests skipping retrieval and querying the model directly.

### Closed-book QA

In **closed-book** QA the model answers from its parameters alone — no passage, no
retrieval, just the question in, the answer out.[^jm-lm] An encoder-decoder like T5,
pretrained to fill masked spans, is fine-tuned on _(question, answer)_ pairs and
learns to emit the answer text directly. With enough scale this is competitive.
It has two failure modes. The model can only recall what its training
data happened to contain and what its weights happened to compress, so it cannot
answer anything recent or rare; and when it does answer, it cannot cite a source —
the fact is distributed across billions of weights with no passage to point to. A
confident wrong answer is indistinguishable from a right one.

### Retrieval-augmented generation

**Retrieval-augmented generation** (RAG) addresses both failures by putting
retrieval back, but feeding the retrieved passages to a _generator_ instead of a
span extractor: it retrieves passages relevant to $q$, concatenates them with the
question into the language model's context, and the model _generates_ the answer
conditioned on that evidence.[^jm-lm] It is retrieve-and-read with the extractive
reader swapped for a generative one, and it is the modern default for open-domain QA.

$$
% caption: Retrieval-augmented generation. The retriever pulls passages for $q$;
% the language model generates the answer conditioned on the question together with
% the retrieved evidence placed in its context.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=24mm, minimum height=13mm, align=center},
  db/.style={draw, cylinder, shape aspect=0.3, minimum width=14mm, minimum height=18mm, align=center, font=\scriptsize, shape border rotate=90}]
  \definecolor{acc}{HTML}{2348F2}
  \node[align=center, font=\footnotesize] (q) at (0,0) {question $q$};
  \node[db] (idx) at (0,-2.5) {indexed\\docs};
  \node[box, draw=acc, text=acc, thick] (ret) at (3.4,-1.2) {retriever};
  \node[box, align=center] (ctx) at (7.2,-1.2) {context:\\$q$ + passages};
  \node[box, draw=acc, text=acc, thick] (lm) at (7.2,1.4) {language\\model};
  \node[align=center, font=\footnotesize] (a) at (11.2,1.4) {generated\\answer};
  \draw[->, thick] (q) -- (ret);
  \draw[->, thick] (idx) -- (ret);
  \draw[->, acc, thick] (ret) -- (ctx) node[midway, above, font=\scriptsize] {passages};
  \draw[->, thick] (q.north) to[out=60, in=180] (lm.west);
  \draw[->, acc, thick] (ctx) -- (lm);
  \draw[->, thick] (lm) -- (a);
\end{tikzpicture}
$$

RAG combines the advantages of both approaches:

- **Attribution and freshness**: the answer is generated conditioned on retrieved
  text, so the model can quote a source and stay current by updating the index, with
  no retraining.
- **Free-form answers**: generation handles complex and abstractive questions that
  no single span could answer.
- **Fewer fabrications**: the evidence is supplied in-context rather than recalled
  from weights, which reduces the closed-book model's rate of confidently stated
  falsehoods.

The modern QA stack, from factoid lookup to document-grounded assistant, follows
this loop: retrieve, then generate.

## DPR, RAG, and fusion-in-decoder

The retrieve-and-read and retrieve-and-generate patterns this lesson describes
come from a line of public work that turned open-domain QA from a
BM25-plus-reader pipeline into the standard neural stack.

**Dense retrieval was shown to beat BM25.** The bi-encoder this lesson sketches is
from Karpukhin, Oğuz, Min, Lewis, Wu, Edunov, Chen, and Yih, "Dense Passage Retrieval
for Open-Domain Question Answering" (**DPR**, EMNLP 2020). They trained two BERT
encoders — one for questions, one for passages — with a contrastive objective that
pulls a question toward its gold passage and pushes it from in-batch negatives, and
reported top-20 passage retrieval accuracy of $78.4\%$ on Natural Questions against
BM25's $59.1\%$, with the downstream QA accuracy rising in step. This is the concrete
evidence for the claim above that a dense retriever can beat the strong BM25 baseline,
and DPR is the model the "dense retrieval" section describes.[^bb-dpr]

**RAG made retrieval part of a generator's training.** Lewis, Perez, Piktus, Petroni,
Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel, and Kiela,
"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (**RAG**, NeurIPS
2020), coupled a DPR retriever to a BART generator and trained them together, so the
generator learns to condition on retrieved passages and the retriever is tuned by the
generation signal. RAG set state-of-the-art open-domain QA numbers at the time and,
because the knowledge lives in a swappable index rather than the weights, could be
updated by replacing the index — the property the RAG section attributes to the
architecture.[^bb-rag]

**Fusion-in-Decoder scaled the number of passages.** A limit of stuffing passages into
one context is that attention cost grows with the concatenated length. Izacard and
Grave, "Leveraging Passage Retrieval with Generative Models for Open Domain Question
Answering" (**FiD**, EACL 2021), encode each retrieved passage _separately_ with the
encoder and let the decoder attend over the concatenation of all passage encodings —
_fusion in the decoder_. This lets the model condition on $100$ passages instead of a
handful, and pushed exact-match on Natural Questions and TriviaQA above prior systems.
FiD is why modern RAG systems retrieve many passages rather than two or three.[^bb-fid]

$$
% caption: The public open-domain QA lineage. Sparse BM25 retrieval feeding an
% extractive reader gave way to dense passage retrieval (DPR, 2020), which was jointly
% trained with a generator in RAG (2020); fusion-in-decoder (FiD, 2021) encodes many
% passages separately so the decoder can attend over 100 of them at once.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  stg/.style={draw, minimum width=25mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[stg] (bm25) at (0,0)   {BM25\\+ span reader};
  \node[stg, draw=acc, text=acc] (dpr) at (3.5,0) {DPR\\dense retrieval\\2020};
  \node[stg] (rag)  at (7.2,0)  {RAG\\joint retriever\\+ generator, 2020};
  \node[stg] (fid)  at (10.9,0) {FiD\\100 passages\\2021};
  \draw[->, acc, thick] (bm25) -- (dpr);
  \draw[->, acc, thick] (dpr) -- (rag);
  \draw[->, acc, thick] (rag) -- (fid);
\end{tikzpicture}
$$

**Grounded does not mean correct.** RAG reduces but does not remove
hallucination. A generator can still misread a retrieved passage, blend two passages
into a false composite, or answer confidently when retrieval returned nothing relevant.
Public evaluations of retrieval-augmented systems report residual unsupported-claim
rates, which is why production systems layer answer attribution (pointing each claim at
the passage that supports it) and abstention on top of the retrieve-then-generate loop.
The pattern is the modern default, not a solved problem.[^bb-rag]

## Evaluation

How an answer is scored depends on whether it was extracted or generated. For
extractive, factoid answers there are two standard metrics against a set of gold
answers.[^jm-eval] **Exact match** is the strictest: the fraction of questions for
which the predicted string, after normalization (lowercasing, stripping articles
and punctuation), equals a gold answer exactly. It is unforgiving — _"in Paris"_
versus gold _"Paris"_ scores zero.

**F1** relaxes that by scoring partial overlap at the token level. Treating the
predicted and gold answers as bags of tokens, precision is the fraction of
predicted tokens that appear in the gold answer, recall the fraction of gold tokens
recovered, and F1 their harmonic mean,

$$
\text{F1} = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}.
$$

F1 gives partial credit — _"in Paris"_ against _"Paris"_ scores well — and is
averaged over the evaluation set, taking the maximum over gold answers when several
are acceptable. Generative and abstractive answers, which need not reuse the
source's words at all, are harder to score this way and lean on overlap metrics or
model-based judgments, but exact match and F1 remain the standard metrics for factoid QA
benchmarks.

One architectural commitment runs through the whole subject: separate _finding_
the evidence from _using_ it. The retriever narrows a large collection to a few
passages; a reader or generator turns those passages into an answer. Swapping the
retriever (sparse to dense) and the second stage (span extractor to language
model) covers everything from a 1990s IR system to a modern RAG assistant.


[^jm-el]: **Jurafsky & Martin**, §23.3 — Entity Linking: associating a text mention with a real-world entity in an ontology (Ji and Grishman 2011); Wikipedia as the ontology and **Wikification** (Mihalcea and Csomai 2007) as linking to Wikipedia pages; the two stages of mention detection and mention disambiguation.
[^jm-tagme]: **Jurafsky & Martin**, §23.3.1 — Linking based on Anchor Dictionaries and Web Graph (the TAGME linker, Ferragina and Scaiella 2011): in-link counts $\text{in}(e)$, the anchor dictionary with $\text{linkprob}(a)$, mention detection over token sequences up to six words, and disambiguation by prior $p(e \mid a)$ and in-link relatedness / coherence with a final linkprob-and-coherence pruning threshold.
[^jm-elq]: **Jurafsky & Martin**, §23.3.2 — Neural Graph-based Linking (the ELQ algorithm, Li et al. 2020): a BERT bi-encoder scoring mention spans and encoding each entity from its Wikipedia title and description, with cached entity embeddings and a softmax over entities per span.
[^jm-kb]: **Jurafsky & Martin**, §23.4 — Knowledge-based Question Answering: RDF triples, semantic parsing to a logical form, entity linking, and relation detection.
[^jm-lm]: **Jurafsky & Martin**, §23.5 — Using Language Models to do QA: closed-book QA with a fine-tuned T5, and generating answers from retrieved evidence.
[^jm-eval]: **Jurafsky & Martin**, §23.2 — evaluation of factoid answers by exact match and token-level F1 against gold answer strings.
[^bb-dpr]: **Karpukhin, Oğuz, Min, Lewis, Wu, Edunov, Chen, and Yih**, "Dense Passage Retrieval for Open-Domain Question Answering," _EMNLP_ 2020. Trained a dual-BERT bi-encoder with a contrastive in-batch-negatives objective, reporting top-20 retrieval accuracy of 78.4% on Natural Questions versus 59.1% for BM25, with corresponding gains in end-to-end QA; the source of the "dense retriever can beat BM25" claim and the bi-encoder in this lesson.
[^bb-rag]: **Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel, and Kiela**, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," _NeurIPS_ 2020. Coupled a DPR retriever with a BART generator trained jointly, setting open-domain QA state of the art at the time and allowing the knowledge source to be updated by swapping the index; retrieval-augmented systems reduce but do not eliminate unsupported (hallucinated) claims, motivating answer attribution and abstention.
[^bb-fid]: **Izacard and Grave**, "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering," _EACL_ 2021. Fusion-in-Decoder encodes each retrieved passage separately and lets the decoder attend over all passage encodings jointly, scaling conditioning to about 100 passages and improving exact-match on Natural Questions and TriviaQA over prior systems.