Question Answering: Knowledge Bases and Language Models

This builds on Question Answering, which developed the retrieve-and-read pipeline: an information retriever (sparse tf-idf/BM25 or a dense bi-encoder) narrows a collection to a few passages, and a BERT reader extracts the answer span. That covers answers that sit verbatim in some document. This part handles the two cases it does not — answers that live in a structured knowledge base, and answers a language model can generate directly — and ends with how factoid answers are scored.

Entity linking and Wikification

Before a question can be answered from a structured knowledge base, the system has to know which entities the question is about. When the question mentions Ada Lovelace, some component must map that string of characters to the specific real-world person — the row in the database, the node in the graph, the page in Wikipedia. That is entity linking: associating a mention in text with the representation of a real-world entity in an ontology.¹ Every knowledge-based QA system depends on it.

The most common ontology for factoid QA is Wikipedia, where each page stands for one entity and doubles as its unique id. Deciding which Wikipedia page a mention refers to has its own name: Wikification. Since the earliest systems, entity linking has run in two stages: mention detection, finding the spans of text that name entities, and mention disambiguation, choosing the right entity for each ambiguous span. We follow the classic TAGME baseline, which links to Wikipedia using an anchor dictionary and the link structure of the encyclopedia itself.

Mention detection and candidate generation

TAGME starts offline by building two resources over a Wikipedia dump. For every page $e$ it counts $\text{in}(e)$, the number of other Wikipedia pages that link to $e$ — a measure of the entity's prominence. It also builds an anchor dictionary: for each page, its title plus every anchor text — the hyperlinked span used on other pages to point at it. The Stanford University page is pointed to by anchors like Stanford and Stanford University. For each anchor string $a$ it records how often $a$ occurs at all, $\text{freq}(a)$, how often it occurs as a link, $\text{link}(a)$, and the ratio

the probability that an occurrence of $a$ is being used as a link — a rough gauge of whether $a$ names an entity at all.

Given a question, mention detection queries the anchor dictionary for every token sequence up to six words long, then prunes the large candidate set with simple heuristics (drop substrings with tiny linkprob). For When was Ada Lovelace born? the span Ada Lovelace surfaces as a strong anchor, Ada as a weaker one, while Lovelace is pruned for low linkprob and born never enters the dictionary at all. Each surviving mention span $a$ carries its set $E(a)$ of candidate entities — the Wikipedia pages that anchor $a$ has ever linked to.

Disambiguation: prior and coherence

If a span points to only one entity, linking is done. Most interesting spans are ambiguous, and TAGME ranks $E(a)$ by two factors.² The first is the prior probability $p(e\mid a)$, how often the anchor $a$ links to entity $e$ relative to all of $a$'s link uses:

The prior alone is not enough. In What Chinese Dynasty came before the Yuan?, the span Yuan most often links to the Chinese currency, not the dynasty — so $p(e\mid a)$ would pick the wrong entity. The second factor, relatedness (coherence), corrects this: the correct entity should cohere with the other entities in the question. Here the question also contains Chinese Dynasty, which links reliably to a page on Chinese dynasties, and that page shares many in-links with Yuan dynasty — pulling the ambiguous Yuan toward the right reading. Two entities count as related to the degree their Wikipedia pages share in-links:

where $\text{in}(x)$ is the set of pages linking to $x$ and $W$ is the whole collection. Every other mention $b$ in the question casts a vote for a candidate $X$, averaging $X$'s relatedness to $b$'s own candidates, weighted by their priors, and the total relatedness of $a\to X$ sums those votes:

To choose the entity for span $a$, TAGME takes the candidates with the highest relatedness, keeps those within a small margin of the top, and among those picks the one with the highest prior $p(X\mid a)$ — coherence first, prior as tie-breaker. A final pruning step discards spurious links whose combined linkprob-and-coherence score falls below a held-out threshold $\lambda$, so a marginal mention is dropped rather than linked to a doubtful entity.

Neural linking and the role in KB-QA

Modern systems replace the anchor dictionary with a bi-encoder, the same approach the reader uses for span extraction.³ A system like ELQ runs the question through BERT, scores each span $[i,j]$ for being a mention (from learned start, end, and mention-token vectors), and separately encodes every Wikipedia entity from its title and the first tokens of its page, taking the [CLS] output as the entity vector. A mention span is linked to the entity whose vector has the highest dot product with the span's, softmaxed over all entities:

Because the entity vectors are precomputed and cached, linking a new question only requires encoding the question. For QA, entity linking supplies the subject of the knowledge-base query. In the semantic-parsing pipeline below, Ada Lovelace is linked to its canonical KB entity, and a separate relation detector supplies the predicate — together they specify the triple that holds the answer. Wikification also grounds a system's answers, letting it cite the Wikipedia page an entity came from. Without entity linking, a knowledge-based system would have a relation but no entity to apply it to.

Knowledge-based QA

Not all knowledge lives in prose. A great deal sits in structured databases and knowledge bases as clean facts, and knowledge-based QA answers a question by translating it into a query over that structure rather than reading text.⁴ A knowledge base is often a set of RDF triples, each a (subject, predicate, object) tuple asserting one relation:

This single triple answers When was Ada Lovelace born? and Who was born in 1815?. The task is to map the question onto the right triple pattern.

Semantic parsing

The general method is semantic parsing: map the natural-language question to a logical form — a query in SQL or SPARQL, a lambda-calculus expression, or another executable program — and run it against the store. For a simple relational question this decomposes into two steps that mirror the parts of the triple: entity linking identifies the subject (mapping the string Ada Lovelace to its canonical KB entity) and relation detection identifies the predicate (mapping When was ... born? to the relation birth-year).

Modern relation detection reuses the same encoder trick as the reader: run the question through BERT, take the [CLS] output $m_r$ as the question's representation, learn a vector $w_{r_i}$ per candidate relation, and pick the relation by a softmax over dot products,

For richer questions the logical form is a full program. A semantic parser can emit a lambda-calculus predicate, a multi-clause SQL query joining several tables, or a SPARQL query over a graph — a sequence-to-sequence model with a BERT encoder maps question tokens to logical-form tokens. Compared with reading text, the appeal of KB-QA is precision: the answer is computed from curated facts, not guessed from prose. Its limit is coverage — a fact absent from the knowledge base is simply unanswerable.

QA with large language models

A pretrained large language model has already been trained on enormous amounts of text and stored a great deal of it in its weights. This suggests skipping retrieval and querying the model directly.

Closed-book QA

In closed-book QA the model answers from its parameters alone — no passage, no retrieval, just the question in, the answer out.⁵ An encoder-decoder like T5, pretrained to fill masked spans, is fine-tuned on (question, answer) pairs and learns to emit the answer text directly. With enough scale this is competitive. It has two failure modes. The model can only recall what its training data happened to contain and what its weights happened to compress, so it cannot answer anything recent or rare; and when it does answer, it cannot cite a source — the fact is distributed across billions of weights with no passage to point to. A confident wrong answer is indistinguishable from a right one.

Retrieval-augmented generation

Retrieval-augmented generation (RAG) addresses both failures by putting retrieval back, but feeding the retrieved passages to a generator instead of a span extractor: it retrieves passages relevant to $q$, concatenates them with the question into the language model's context, and the model generates the answer conditioned on that evidence.⁵ It is retrieve-and-read with the extractive reader swapped for a generative one, and it is the modern default for open-domain QA.

RAG combines the advantages of both approaches:

• Attribution and freshness: the answer is generated conditioned on retrieved text, so the model can quote a source and stay current by updating the index, with no retraining.
• Free-form answers: generation handles complex and abstractive questions that no single span could answer.
• Fewer fabrications: the evidence is supplied in-context rather than recalled from weights, which reduces the closed-book model's rate of confidently stated falsehoods.

The modern QA stack, from factoid lookup to document-grounded assistant, follows this loop: retrieve, then generate.

DPR, RAG, and fusion-in-decoder

The retrieve-and-read and retrieve-and-generate patterns this lesson describes come from a line of public work that turned open-domain QA from a BM25-plus-reader pipeline into the standard neural stack.

Dense retrieval was shown to beat BM25. The bi-encoder this lesson sketches is from Karpukhin, Oğuz, Min, Lewis, Wu, Edunov, Chen, and Yih, Dense Passage Retrieval for Open-Domain Question Answering (DPR, EMNLP 2020). They trained two BERT encoders — one for questions, one for passages — with a contrastive objective that pulls a question toward its gold passage and pushes it from in-batch negatives, and reported top-20 passage retrieval accuracy of $78.4\%$ on Natural Questions against BM25's $59.1\%$, with the downstream QA accuracy rising in step. This is the concrete evidence for the claim above that a dense retriever can beat the strong BM25 baseline, and DPR is the model the dense retrieval section describes.⁶

RAG made retrieval part of a generator's training. Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel, and Kiela, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG, NeurIPS 2020), coupled a DPR retriever to a BART generator and trained them together, so the generator learns to condition on retrieved passages and the retriever is tuned by the generation signal. RAG set state-of-the-art open-domain QA numbers at the time and, because the knowledge lives in a swappable index rather than the weights, could be updated by replacing the index — the property the RAG section attributes to the architecture.⁷

Fusion-in-Decoder scaled the number of passages. A limit of stuffing passages into one context is that attention cost grows with the concatenated length. Izacard and Grave, Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (FiD, EACL 2021), encode each retrieved passage separately with the encoder and let the decoder attend over the concatenation of all passage encodings — fusion in the decoder. This lets the model condition on $100$ passages instead of a handful, and pushed exact-match on Natural Questions and TriviaQA above prior systems. FiD is why modern RAG systems retrieve many passages rather than two or three.⁸

Grounded does not mean correct. RAG reduces but does not remove hallucination. A generator can still misread a retrieved passage, blend two passages into a false composite, or answer confidently when retrieval returned nothing relevant. Public evaluations of retrieval-augmented systems report residual unsupported-claim rates, which is why production systems layer answer attribution (pointing each claim at the passage that supports it) and abstention on top of the retrieve-then-generate loop. The pattern is the modern default, not a solved problem.⁷

Evaluation

How an answer is scored depends on whether it was extracted or generated. For extractive, factoid answers there are two standard metrics against a set of gold answers.⁹ Exact match is the strictest: the fraction of questions for which the predicted string, after normalization (lowercasing, stripping articles and punctuation), equals a gold answer exactly. It is unforgiving — in Paris versus gold Paris scores zero.

F1 relaxes that by scoring partial overlap at the token level. Treating the predicted and gold answers as bags of tokens, precision is the fraction of predicted tokens that appear in the gold answer, recall the fraction of gold tokens recovered, and F1 their harmonic mean,

F1 gives partial credit — in Paris against Paris scores well — and is averaged over the evaluation set, taking the maximum over gold answers when several are acceptable. Generative and abstractive answers, which need not reuse the source's words at all, are harder to score this way and lean on overlap metrics or model-based judgments, but exact match and F1 remain the standard metrics for factoid QA benchmarks.

One architectural commitment runs through the whole subject: separate finding the evidence from using it. The retriever narrows a large collection to a few passages; a reader or generator turns those passages into an answer. Swapping the retriever (sparse to dense) and the second stage (span extractor to language model) covers everything from a 1990s IR system to a modern RAG assistant.