Question Answering

A search engine returns documents; a question-answering system returns an answer. For the question What's the official language of Algeria?, the correct output is the word Arabic, not ten Wikipedia pages ranked by relevance. Almost every modern system produces the answer with one pattern: retrieve a small set of promising passages from a large collection, then read or generate the answer from them.¹

Two distinctions organize the field. The first is what kind of question is asked. A factoid question has a short, factual answer that can be stated in a phrase — a name, a date, a place (Where is the Louvre? → in Paris). A complex question needs synthesis across several facts or a paragraph of explanation (Why did the Roman Republic fall?). The second is how the answer is produced. Extractive QA returns a literal span copied out of some source text; generative (or abstractive) QA writes the answer in its own words, possibly combining several sources. Most of this lesson is factoid QA, because it exposes the machinery most cleanly, but the retrieve-then-generate skeleton scales straight up to the complex, generative case.

Information retrieval

Everything downstream depends on the retriever handing the reader a handful of passages that actually contain the answer, so we start there. Information retrieval (IR) is the task of returning, from a large document collection, the documents most relevant to a query.² We write the query as $q$ and a document as $d$, and represent both as vectors in a shared space; relevance is then a geometric quantity — how aligned the query vector is with the document vector.

Term weighting and the vector space model

The oldest representation is a sparse word-count vector: one dimension per vocabulary term, most entries zero. Raw counts overweight common words, so each term $t$ in a document $d$ is weighted by tf-idf, the product of a term frequency and an inverse document frequency,

where $N$ is the number of documents and ${\text{df}}_t$ the number that contain $t$. The idf term does the real work here: a word appearing in every document (like the) carries $\text{idf}\approx 0$ and is downweighted to nothing, while a rare, discriminating word gets a large weight. Documents are then scored against the query by the cosine of their vectors, which for the usual simplification (query terms counted once, query normalization constant across documents) reduces to a sum over the query's terms,

with $|d|$ the document's vector length; the denominator prevents a long document from scoring highly on term count alone.

A slightly richer member of the same family is BM25, which adds two knobs: a parameter $k$ balancing term frequency against idf, and $b$ controlling how much document length is penalized. The BM25 score of $d$ given $q$ is

where $|d_{\text{avg}}|$ is the average document length. Reasonable defaults are $k\in [1.2,2]$ and $b=0.75$. At $k=0$ BM25 ignores term frequency entirely and reduces to a binary idf-weighted match; large $k$ recovers raw term frequency. BM25 remains a strong, ubiquitous baseline that dense retrievers do not always beat.

Worked example: tf-idf vs. BM25 on two documents

Run both scorers on one query against two toy documents to see the effect of $k$ and $b$. The query is tragic love; the collection has $N=100$ documents, in $10$ of which love occurs and $5$ of which tragic occurs, so the idf weights are ${\text{idf}}_{\text{love}}={\log }_{10}(100/10)=1.000$ and ${\text{idf}}_{\text{tragic}}={\log }_{10}(100/5)=1.301$. Two candidate documents, with the average document length $|d_{\text{avg}}|=120$:

$d_1$ is short and focused; $d_2$ is long and repeats love forty times. Plain tf-idf (using the log-squashed $\text{tf}={\log }_{10}(\text{count}+1)$ and summing $\text{tf}\cdot \text{idf}$ over the query terms) rewards that repetition:

against $\text{tf-idf}(d_1)={\log }_{10}(9)\cdot 1.000+{\log }_{10}(3)\cdot 1.301=0.954+0.621=1.575$. Plain tf-idf ranks the long, keyword-stuffed $d_2$ above $d_1$.

BM25 at the default $k=1.2$, $b=0.75$ (using the raw term count as $\text{tf}$ inside the saturation term, per the formula above) flips the ranking. The love term in $d_2$, despite forty occurrences, contributes almost the same as love in $d_1$ with eight, because the tf saturates and the length penalty grows the denominator:

Forty occurrences in the long document beat eight in the short one by less than a hundredth of a point. Summing both query terms gives the full scores below, and varying the knobs isolates what each controls:

Read the last three rows as controlled experiments. At $k=0$ term frequency drops out of the formula entirely and both documents score ${\text{idf}}_{\text{love}}+{\text{idf}}_{\text{tragic}}=2.301$ — a pure binary idf match, since both contain both query terms. Turning length normalization off ($b=0$) lets $d_2$'s extra occurrences win again, exactly as plain tf-idf did; turning it fully on ($b=1$) penalizes $d_2$'s length hardest and gives $d_1$ the largest margin. The default $b=0.75$ sits between, enough to stop a long document from ranking first on sheer term count. BM25's two parameters correct the two ways plain tf-idf misranks: unbounded term-frequency reward and no penalty for length.²

The inverted index

To score documents we first have to find the ones sharing a term with the query, and scanning the whole collection per query is far too slow. The inverted index is the data structure that makes it fast.³ It maps each term to a postings list — the document IDs that contain it, usually annotated with the term's count or positions — through a dictionary of terms built for quick lookup. Given the query's terms, a lookup in the dictionary yields exactly the candidate documents that could score above zero, together with the counts needed to compute tf-idf.

Dense retrieval

tf-idf and BM25 share a structural weakness: they only match when the query and document use the same words. A user asking about a tragic love story will never match a passage that says star-crossed lovers — this is the vocabulary mismatch problem.⁴ To address it, replace sparse word-count vectors with dense ones: a fixed-length embedding, a few hundred real numbers, produced by an encoder that has learned to place synonymous text nearby. Because the vectors come from a model trained on meaning rather than surface form, tragic love story and star-crossed lovers land close together even with no shared word.

Modern dense retrieval uses a bi-encoder: two encoders, one for the query and one for the document, each a transformer like BERT. We take the [CLS] output of each as the vector and score by dot product,

Encoding both sides separately is what makes this practical at scale: every document vector $h_d$ is computed once, offline, and stored, so a query only has to encode $q$ and find its nearest neighbors. That nearest-neighbor search over millions of dense vectors is itself expensive, so systems use approximate nearest neighbor libraries like Faiss instead of an exact scan.

Evaluating retrieval

A ranked retriever is judged with precision and recall: precision is the fraction of returned documents that are relevant, recall the fraction of relevant documents returned. Because both ignore order, ranked systems are usually summarized by mean average precision (MAP), which averages the precision measured at each rank where a relevant document appears, then averages that over a set of queries. Higher precision across all recall levels is the goal; MAP folds the whole precision-recall curve into one comparable number.

IR-based factoid QA: retrieve and read

With a retriever in hand, the dominant IR-based architecture is retrieve and read.⁵ Stage one retrieves relevant passages from the collection using the IR machinery above. Stage two runs a neural reading-comprehension model over each passage: given the question and a passage that may contain the answer, it extracts the answer span — a contiguous run of tokens.

Span extraction with BERT

The reading model computes, for a question $q$ of $n$ tokens $q_1,\dots ,q_n$ and a passage $p$ of $m$ tokens $p_1,\dots ,p_m$, the probability $P(a\mid q,p)$ that each candidate span $a$ is the answer. It factors that span probability into a start and an end: for span from token $a_s$ to token $a_e$,

So the model only has to compute, for every passage token $p_i$, the probability $p_{\text{start}}(i)$ that it begins the answer and $p_{\text{end}}(i)$ that it ends it. The standard baseline concatenates the question and passage into one BERT input, separated by a [SEP] token and prefixed by [CLS], yielding an embedding $p_i^{\prime }$ for every passage token.⁶ This reuses the fine-tuning recipe wholesale — a pretrained encoder plus a small task head trained on labelled data.

The head is two learned vectors, a span-start embedding $S$ and a span-end embedding $E$. The start probability of token $i$ is a softmax of the dot product between $S$ and that token's embedding, and the end probability is the same with $E$,

The score of a candidate span from $i$ to $j$ is $S\cdot p_i^{\prime }+E\cdot p_j^{\prime }$, and the model returns the highest-scoring span with $j\ge i$. Fine-tuning minimizes the negative log-likelihood of the gold start and end positions,

Unanswerable questions get a simple treatment. Datasets like SQuAD 2.0 include questions whose answer is not in the passage; the model learns to point both the start and end at the [CLS] token in that case, so [CLS] serves as the no-answer prediction. And because passages are often longer than BERT's token limit, a long document is chopped into overlapping windows, each treated as its own passage, and the answer is the highest-scoring span across all of them.

A worked retrieve-and-read trace

Carry a real question through both stages. Ask When was Mozart born? against a tiny collection of three passages, with a dense bi-encoder retriever and a BERT reader.

Stage one — dense retrieval. The query encoder maps the question to a vector $h_q$, and each passage was encoded offline to $h_d$. Scoring by dot product gives a ranking; suppose the numbers come out as

Note that $d_1$ shares the word Mozart and $d_3$ shares Salzburg, but the passage that actually states the birth, $d_2$, wins because the bi-encoder places the meaning of when born near was born … on … 1756 even though the question's words when and born appear differently in the passage. A sparse BM25 retriever keying on the word born would also surface $d_2$ here, but would rank a passage using birth or native of far lower — the vocabulary-mismatch gap dense retrieval closes. The top $k$ passages (say $k=2$: $d_2,d_1$) pass to the reader.

Stage two — span extraction. The reader concatenates the question with $d_2$ into one BERT input, [CLS] when was Mozart born ? [SEP] Wolfgang Amadeus Mozart was born in Salzburg on 27 January 1756 [SEP], and produces an embedding $p_i^{\prime }$ per token. The learned start vector $S$ dots highest with the token 27 and the end vector $E$ dots highest with 1756, giving the candidate span 27 January 1756. Suppose the dot products land at

and that this beats every span the reader extracts from $d_1$ (whose best span, around 1791, scores lower because $d_1$ is about his death, not his birth) and beats the [CLS]-to-[CLS] no-answer score. The system returns 27 January 1756 — or, under a stricter answer normalization that keeps only the year, 1756. The retriever narrowed the collection to two passages; the reader extracted one span from the top passage.

Datasets

Reading-comprehension datasets are built as (passage, question, answer-span) triples. SQuAD takes Wikipedia passages and has annotators write questions whose answers are spans of the passage; HotpotQA forces reasoning across multiple documents; Natural Questions uses real Google queries paired with a Wikipedia page, annotating a long and a short answer or null. At training time the reader sees the gold passage; at inference in the full two-stage system the passage is withheld and the retriever must find it — which is why open-domain QA is harder than reading comprehension alone. The retriever-plus-reader pattern handles any answer that appears verbatim in some passage. Two cases fall outside it: answers that live in a structured knowledge base rather than prose, and answers a large language model can generate from its own weights. This continues in Question Answering: Knowledge Bases and Language Models.