--- title: Fine-Tuning and Prompting module: Transformers moduleNumber: 5 lessonNumber: 5 order: 505 summary: > A pretrained transformer is a general-purpose knowledge source; a task is what you do with it. There are two ways to adapt one, and this first part covers the one that updates the weights: fine-tuning. A bidirectional encoder like BERT is pretrained by masked language modeling, then a small task head is bolted on and the whole thing is trained on labelled data for classification, sequence labeling, or span-based question answering — with parameter-efficient variants (adapters, LoRA) that touch only a sliver of the weights. Prompting, the family that leaves the weights frozen, comes next. topics: [Transformers] sources: - book: Jurafsky ref: "Ch. 11 — Fine-tuning and Masked Language Models; Bidirectional Transformer Encoders; Masked Language Modeling; Fine-Tuning for Classification and Sequence Labeling" - book: Jurafsky ref: "§23.2 — Reading-comprehension / span-based question answering (BERT fine-tuning, eqs. 23.16–23.18)" --- A [large language model](/natural-language-processing/transformers/large-language-models) is pretrained once, on web-scale text, to do one thing: predict tokens. That single objective forces it to absorb syntax, facts, and a great deal of world knowledge, all folded into its weights. But prediction is rarely the target task itself — the target is sentiment labels, named entities, answers to questions, a helpful reply. **Adaptation** turns the general-purpose predictor into a task-specific model. Pretraining is paid for once; adaptation is cheap and is done many different ways on the same frozen starting point.[^jm-ft] There are two families of adaptation, and the difference between them is one bit: whether you change the weights. - **Fine-tuning** _updates_ the weights. Start from the pretrained model, add a small task-specific layer, and continue training on labelled examples for the target task. The result is a new, specialized model. - **Prompting** leaves the weights _frozen_. Describe the task to the model in its input — an instruction, maybe a few worked examples — and read the answer off its next-token prediction. No gradient step, no new model. These two parts build up both. This one covers fine-tuning, starting with the pretraining objective that makes it possible; the [next](/natural-language-processing/transformers/prompting-and-alignment) covers prompting and the training stages that turn a base model into an assistant. ## The pretrain-then-finetune paradigm The scheme has two phases. In **pretraining**, a model learns from raw text with a self-supervised objective — a label mined from the text itself, no human annotation. In **fine-tuning**, that pretrained model is the starting point for supervised training on a specific labelled task.[^jm-ft] The weights carry over; only the final task-specific piece is new. $$ % caption: The pretrain-then-finetune paradigm. One expensive self-supervised % pretraining run produces a base model whose weights are the starting point for % many cheap task-specific fine-tunes. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=26mm, minimum height=12mm, align=center}, small/.style={draw, minimum width=22mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[box] (corpus) at (0,0) {raw text\\(self-supervised)}; \node[box, draw=acc, text=acc, thick] (base) at (4.2,0) {pretrained\\base model}; \draw[->, acc, thick] (corpus) -- (base) node[midway, above, font=\scriptsize] {pretrain once}; \node[small] (t1) at (8.6,1.6) {sentiment}; \node[small] (t2) at (8.6,0.0) {NER}; \node[small] (t3) at (8.6,-1.6) {QA}; \draw[->, thick] (base) -- (t1) node[midway, above, font=\scriptsize] {}; \draw[->, thick] (base) -- (t2) node[midway, above, font=\scriptsize] {f\/ine-tune}; \draw[->, thick] (base) -- (t3) node[midway, below, font=\scriptsize] {}; \end{tikzpicture} $$ The cost asymmetry is large. Pretraining a modern model costs millions of dollars and weeks of compute; fine-tuning it for a new task takes hours and a few thousand labelled examples, because the general representations are already learned. The fine-tune only has to learn the thin mapping from the model's existing representations to the task's outputs. > **Definition (Pretrain-then-finetune).** A two-phase training scheme: first learn > general representations from unlabelled text by a self-supervised objective > (pretraining), then continue training on a small labelled dataset for a specific > task (fine-tuning). Pretraining is done once and amortized across every > downstream task. The [feedforward and recurrent language models](/natural-language-processing/semantics/neural-language-models) of earlier lessons were **causal**: they predict each word from the words to its _left_, because a language model that generates text must not peek at the future. That left-to-right constraint is right for generation but wrong for _understanding_. To classify a word — is "bank" a river bank or a financial one? — you want to see the words on _both_ sides. The first family of adaptation is built on an encoder that does exactly that. ## Bidirectional encoders and BERT A **bidirectional transformer encoder** drops the causal mask. Every token attends to every other token, left and right, so each position's output vector is a **contextual embedding** that summarizes the whole sentence as seen from that word.[^jm-bert] This is the architecture of **BERT** (Bidirectional Encoder Representations from Transformers) and its many descendants: a stack of transformer encoder blocks, no decoder, producing one context-sensitive vector per input token. $$ % caption: A causal (left-to-right) model masks out the future so position $i$ % attends only to positions $\le i$; a bidirectional encoder lets every position % attend to the whole sequence, giving a contextual embedding informed by both % sides. \begin{tikzpicture}[>=stealth, font=\footnotesize, tok/.style={draw, minimum width=7mm, minimum height=7mm, inner sep=1pt, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % --- causal (left) --- \begin{scope} \foreach \i/\w in {0/the, 1/bank, 2/was, 3/steep} \node[tok] (a\i) at (\i*0.95,0) {\w}; \node[font=\scriptsize, anchor=north] at (1.4,-0.55) {causal: attends left only}; \draw[->, black] (a1.north) .. controls (0.7,0.9) and (0.2,0.9) .. (a0.north); \draw[->, black] (a3.north) .. controls (2.4,1.0) and (1.6,1.0) .. (a1.north); \draw[->, black] (a3.north) .. controls (2.6,0.8) and (2.1,0.8) .. (a2.north); \end{scope} % --- bidirectional (right) --- \begin{scope}[xshift=6.4cm] \foreach \i/\w in {0/the, 1/bank, 2/was, 3/steep} \node[tok] (b\i) at (\i*0.95,0) {\w}; \node[font=\scriptsize, anchor=north] at (1.4,-0.55) {bidirectional: attends both ways}; \draw[->, acc] (b1.north) .. controls (0.7,0.85) and (0.2,0.85) .. (b0.north); \draw[->, acc] (b1.north) .. controls (1.5,0.85) and (2.1,0.85) .. (b3.north); \draw[->, acc] (b1.north) .. controls (1.3,0.7) and (1.7,0.7) .. (b2.north); \end{scope} \end{tikzpicture} $$ Because it sees the whole context, an encoder is not a text generator — you cannot sample the next word from it. Its job is to _represent_, producing embeddings that a downstream layer turns into labels. This poses a problem: a left-to-right model trains by predicting the next word, a signal available for free from any text. What is the self-supervised signal for a model that already sees every word? If BERT could see the whole sentence, predicting a word it already has as input would be trivial. The answer is to hide part of the input. ## Masked language modeling The pretraining objective for a bidirectional encoder is **masked language modeling** (MLM). Take a sentence, corrupt a random fraction of its tokens — in BERT, about $15\%$ — and train the model to reconstruct the originals from the surrounding context.[^jm-mlm] The corruption is usually replacement with a special `[MASK]` token, and the loss is the cross-entropy of predicting the true token at each masked position, exactly the softmax-over-vocabulary loss of an ordinary language model, but applied only at the hidden positions. $$ % caption: Masked language modeling. A random token is replaced by [MASK]; the % encoder reads the whole (corrupted) sentence bidirectionally, and a softmax over % the vocabulary at the masked position is trained to recover the original word, % here "gets". \begin{tikzpicture}[>=stealth, font=\footnotesize, tok/.style={draw, minimum width=9mm, minimum height=7mm, inner sep=1pt, font=\scriptsize}, mask/.style={draw, draw=acc, text=acc, minimum width=13mm, minimum height=7mm, inner sep=1pt, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % input tokens \node[tok] (t0) at (0,0) {the}; \node[tok] (t1) at (1.2,0) {cat}; \node[mask] (t2) at (2.55,0) {[MASK]}; \node[tok] (t3) at (4.0,0) {fed}; \node[font=\scriptsize, anchor=north] at (2.0,-0.5) {corrupted input}; % encoder bar \node[draw, fill=black!5, minimum width=44mm, minimum height=8mm, anchor=south] (enc) at (2.0,1.0) {bidirectional encoder}; \foreach \i in {t0,t1,t2,t3} \draw[->, black] (\i.north) -- (\i.north |- enc.south); % prediction head over the masked position \node[draw, draw=acc, minimum width=20mm, minimum height=7mm, align=center, font=\scriptsize] (head) at (2.55,3.0) {softmax over\\vocabulary}; \draw[->, acc, thick] (2.55,1.85) -- (head.south); \node[acc, anchor=west, font=\scriptsize] at (3.9,3.3) {p(gets) high}; \node[black, anchor=west, font=\scriptsize] at (3.9,2.7) {p(zebra) low}; \end{tikzpicture} $$ Masking is a clean way to force bidirectional understanding. To fill in a blank correctly the model must use both the left context ("the cat") and any right context, so it learns to build each position's embedding from the entire sentence. And the labels are free: any text is its own answer key once part of it is hidden — the same self-supervised principle from the [taxonomy of learning](/deep-learning/foundations/what-is-deep-learning). > **Definition (Masked language modeling).** A self-supervised objective for > bidirectional encoders: replace a random subset of input tokens with `[MASK]` > (or a random or unchanged token), and train the model to predict the original > tokens from the two-sided context, minimizing cross-entropy at the masked > positions only. BERT paired MLM with a second, sentence-level objective. In **next-sentence prediction** (NSP), the model is given two segments and asked to classify whether the second actually followed the first in the corpus or was randomly chosen.[^jm-mlm] The goal was to teach the relationships between sentences that tasks like question answering and entailment need. Later encoders found NSP weak and often dropped or replaced it — for example with **contrastive** sentence objectives that pull the embeddings of related sentences together and push unrelated ones apart — but the pattern is the same: a cheap self-supervised signal invented from the raw text. The tokens themselves are not words but **wordpieces**: sub-word units produced by an algorithm like BPE or WordPiece, so a rare word splits into known pieces and the vocabulary stays fixed and finite.[^jm-wp] Two special tokens matter for the tasks below: a `[CLS]` token prepended to every input, whose final embedding serves as a summary of the whole sequence, and a `[SEP]` token that separates two segments when the input is a pair. ### How subword vocabularies are built: a BPE trace The vocabulary is not hand-designed; it is _learned_ from the corpus by **byte-pair encoding** (BPE; Sennrich et al., 2016), the algorithm SLP3 works through in Chapter 2.[^jm-wp] BPE starts with a vocabulary of individual characters and repeatedly merges the most frequent adjacent pair into a new symbol, growing longer tokens until it has performed $k$ merges. It runs inside words, so each word is first split into characters plus an end-of-word marker $\_$. Take Jurafsky & Martin's tiny corpus of five distinct words with counts: `low` (5), `lowest` (2), `newer` (6), `wider` (3), `new` (2). The starting vocabulary is the eleven letters plus $\_$. The learner counts adjacent pairs across the whole corpus and merges the most frequent: - **Merge 1.** The pair `e r` occurs in `newer` (6) and `wider` (3), total $9$ — the most frequent. Merge to `er`. Now `newer` is `n e w er`, `wider` is `w i d er`. - **Merge 2.** `er _` (word-final `er`) occurs $9$ times; merge to `er_`. - **Merge 3.** `n e` occurs in `newer` (6) and `new` (2), total $8$; merge to `ne`. - **Merge 4.** `ne w` totals $8$; merge to `new`. - **Merge 5.** `l o` totals $7$ (`low` 5 + `lowest` 2); merge to `lo`. - **Merge 6.** `lo w` totals $7$; merge to `low`. After these merges the vocabulary contains the eleven characters plus `er`, `er_`, `ne`, `new`, `lo`, `low`. To tokenize new text, the learned merges are applied greedily in the order learned. The word `newer` collapses back to a single token; but an unseen word like `lower` tokenizes as `low` + `er_` — two known pieces — so the model never meets a true unknown word, which is the entire point. The vocabulary size is fixed by $k$, yet any string is representable. $$ % caption: A byte-pair-encoding trace on the corpus low(5), lowest(2), newer(6), wider(3), % new(2). Each row merges the most frequent adjacent pair (its total count on the right) into % a new token ("eow" and the trailing "." mark the end-of-word symbol); after six merges an % unseen word like "lower" splits into known pieces low + er. \begin{tikzpicture}[>=stealth, font=\footnotesize, merge/.style={draw, minimum width=30mm, minimum height=6mm, inner sep=2pt, font=\scriptsize, align=left}, cnt/.style={font=\scriptsize, text=black}] \definecolor{acc}{HTML}{2348F2} \foreach \i/\m/\c in {0/{e + r $\to$ er}/9, 1/{er + eow $\to$ er.}/9, 2/{n + e $\to$ ne}/8, 3/{ne + w $\to$ new}/8, 4/{l + o $\to$ lo}/7, 5/{lo + w $\to$ low}/7} { \node[merge] (m\i) at (0,-\i*0.82) {\m}; \node[cnt, anchor=west] at (1.7,-\i*0.82) {count \c}; } \node[merge, draw=acc, text=acc, minimum width=52mm] (res) at (1.1,-5.4) {result: lower $\to$ low + er. (two known pieces)}; \end{tikzpicture} $$ ## Fine-tuning: adding a task head Pretraining gives one contextual embedding per token, plus the `[CLS]` summary. Fine-tuning turns those embeddings into task outputs by adding a **task head** — a small layer, often a single linear map plus softmax — on top of the frozen-then- unfrozen encoder, and training the whole assembly on labelled data.[^jm-ftclass] Three task shapes cover most of NLP, and each reads off a different part of the encoder's output. $$ % caption: Three fine-tuning heads on the same encoder. Sequence classification % reads the [CLS] summary vector; sequence labeling puts a head on every token's % embedding; span extraction scores each token as a possible answer start and end. \begin{tikzpicture}[>=stealth, font=\footnotesize, tok/.style={draw, minimum width=8mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}, h/.style={draw, draw=acc, text=acc, minimum width=8mm, minimum height=6mm, inner sep=1pt, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} % --- (a) classification --- \begin{scope} \node[tok] (c0) at (0,0) {CLS}; \node[tok] (c1) at (0.9,0) {w1}; \node[tok] (c2) at (1.8,0) {w2}; \node[h] (ch) at (0,1.3) {y}; \draw[->, acc] (c0) -- (ch); \node[font=\scriptsize, anchor=north] at (0.9,-0.45) {classi\/f\/ication}; \end{scope} % --- (b) sequence labeling --- \begin{scope}[xshift=4.4cm] \node[tok] (l0) at (0,0) {CLS}; \node[tok] (l1) at (0.9,0) {w1}; \node[tok] (l2) at (1.8,0) {w2}; \node[h] (lh1) at (0.9,1.3) {t1}; \node[h] (lh2) at (1.8,1.3) {t2}; \draw[->, acc] (l1) -- (lh1); \draw[->, acc] (l2) -- (lh2); \node[font=\scriptsize, anchor=north] at (0.9,-0.45) {sequence labeling}; \end{scope} % --- (c) span --- \begin{scope}[xshift=8.8cm] \node[tok] (s0) at (0,0) {CLS}; \node[tok] (s1) at (0.9,0) {p1}; \node[tok] (s2) at (1.8,0) {p2}; \node[h] (sh1) at (0.9,1.3) {s/e}; \node[h] (sh2) at (1.8,1.3) {s/e}; \draw[->, acc] (s1) -- (sh1); \draw[->, acc] (s2) -- (sh2); \node[font=\scriptsize, anchor=north] at (0.9,-0.45) {span extraction}; \end{scope} \end{tikzpicture} $$ **Sequence classification** (sentiment, topic, entailment) attaches a head to the `[CLS]` summary vector, mapping it to the class scores. **Sequence labeling** (part-of-speech tagging, [named-entity recognition](/natural-language-processing/sequences/sequence-labeling)) attaches the same head to _every_ token's embedding, producing one label per token. The third shape, span extraction, is used for extractive question answering; SLP3 gives it in full detail, so we work through it here. ### Span-based question answering In extractive (reading-comprehension) QA, the model is given a question and a passage, and must return the **span** of the passage that answers it.[^jm-qa] The input is built by concatenating the two segments — the question first, the passage second, separated by `[SEP]` — and running them through the encoder, which produces a contextual embedding $p'_i$ for every passage token. The head is two learned vectors: a **span-start** embedding $S$ and a **span-end** embedding $E$. For each passage token $p'_i$, the probability that it begins the answer is a softmax of the dot product $S \cdot p'_i$ over all passage tokens, and the end probability uses $E$ the same way: $$ P_{\text{start}_i} = \frac{\exp(S \cdot p'_i)}{\sum_j \exp(S \cdot p'_j)}, \qquad P_{\text{end}_i} = \frac{\exp(E \cdot p'_i)}{\sum_j \exp(E \cdot p'_j)}. $$ A candidate span from token $i$ to token $j$ scores $S \cdot p'_i + E \cdot p'_j$, and the model predicts the highest-scoring span with $j \ge i$. Only $S$ and $E$ are new; everything below them is the pretrained encoder, nudged by the same gradient. The fine-tuning loss is the negative log-likelihood of the correct start and end positions, $$ L = -\log P_{\text{start}_i} - \log P_{\text{end}_i}, $$ summed over the training examples.[^jm-qa] When a dataset includes questions with _no_ answer in the passage, the trick is to point the gold start and end at the `[CLS]` token, so the model can learn to abstain. For example, take a four-token passage _Ada wrote the program_ and a question _who wrote it?_. Suppose the encoder's contextual embeddings, dotted with the learned $S$ and $E$ vectors, give start scores $S \cdot p' = (0.5, 3.0, 1.0, 0.2)$ and end scores $E \cdot p' = (0.3, 0.8, 2.5, 0.4)$ over the four tokens. Softmax turns these into start probabilities $(0.064, 0.782, 0.106, 0.048)$ — the model strongly prefers to start the answer at token $2$ (_wrote_) — and end probabilities $(0.078, 0.129, 0.706, 0.086)$, peaking at token $3$ (_the_). The predicted span maximizes $S \cdot p'_i + E \cdot p'_j$ over $j \ge i$; the best pair is $i = 2, j = 3$ with score $3.0 + 2.5 = 5.5$, returning the span _wrote the_. The constraint $j \ge i$ rules out impossible spans where the end precedes the start, and the two independent softmaxes let a single forward pass score every start and every end at once. $$ % caption: Span extraction over a four-token passage. Start scores (blue) peak at token 2 and % end scores (red) at token 3; the predicted span maximizes start(i) + end(j) with j >= i, % giving the span from token 2 to token 3. \begin{tikzpicture}[>=stealth, font=\footnotesize, tok/.style={draw, minimum width=17mm, minimum height=7mm, inner sep=1pt, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[anchor=east, font=\scriptsize, text=black] at (-0.2,1.6) {token}; \node[anchor=east, font=\scriptsize, text=acc] at (-0.2,0.8) {start score}; \node[anchor=east, font=\scriptsize, text=red] at (-0.2,0.0) {end score}; \foreach \c/\w/\s/\e in {1/Ada/0.5/0.3, 2/wrote/3.0/0.8, 3/the/1.0/2.5, 4/program/0.2/0.4} { \node[tok] at (\c*2.0,1.6) {\w}; \node[tok, text=acc] at (\c*2.0,0.8) {\s}; \node[tok, text=red] at (\c*2.0,0.0) {\e}; } % highlight chosen start (token2) and end (token3) \draw[acc, thick] (2*2.0-0.95,0.45) rectangle (2*2.0+0.95,1.15); \draw[red, thick] (3*2.0-0.95,-0.35) rectangle (3*2.0+0.95,0.35); \node[font=\scriptsize, anchor=north, text=black!70] at (5.0,-0.7) {predicted span: tokens 2 to 3 = "wrote the", score 5.5}; \end{tikzpicture} $$ ```algorithm caption: $\textsc{Finetune-QA}$ — span-based extractive question answering input: pretrained encoder, labelled pairs $(q, p, i^\star, j^\star)$, rate $\eta$ initialize span-start vector $S$, span-end vector $E$ randomly repeat sample $(q, p, i^\star, j^\star)$ // question, passage, gold start/end $x \gets$ concat($[\text{CLS}], q, [\text{SEP}], p$) $(p'_1, \ldots, p'_m) \gets \textsc{Encoder}(x)$ // contextual embeddings of passage tokens $P_{\text{start}} \gets \mathrm{softmax}_i(S \cdot p'_i)$ $P_{\text{end}} \gets \mathrm{softmax}_i(E \cdot p'_i)$ $L \gets -\log P_{\text{start}_{i^\star}} - \log P_{\text{end}_{j^\star}}$ update encoder, $S$, $E$ by $-\eta\,\nabla L$ // gradient reaches the whole model until converged ``` The pattern generalizes: whatever the task, choose which embeddings to read, add a thin head, and let the gradient tune the head and the encoder together. Because the encoder starts from a rich pretrained state, a few thousand labelled examples usually suffice — the fine-tune is refining representations, not learning language from scratch. ## Parameter-efficient fine-tuning Full fine-tuning updates every weight, and produces a full-size copy of the model per task. For a billion-parameter encoder and a dozen tasks that is wasteful in both storage and compute. **Parameter-efficient fine-tuning** (PEFT) freezes the pretrained weights and trains only a small number of new ones, recovering most of the accuracy at a fraction of the cost.[^jm-peft] Two designs dominate. **Adapters** insert tiny bottleneck layers between the frozen transformer blocks — down-project to a small dimension, apply a non-linearity, up-project back — and train only those. **LoRA** (Low-Rank Adaptation) freezes each weight matrix $W$ and learns a low-rank correction $\Delta W = BA$, where $B$ and $A$ are thin matrices, so the effective weight is $W + BA$ with only $BA$ trainable. $$ % caption: Parameter-efficient fine-tuning. The pretrained weight $W$ stays frozen; % LoRA learns a low-rank correction $\Delta W = BA$ (with $A, B$ thin) added in % parallel, so only a small fraction of the parameters are updated. \begin{tikzpicture}[>=stealth, font=\footnotesize, frozen/.style={draw, minimum width=16mm, minimum height=13mm, align=center, fill=black!6}, trained/.style={draw, draw=acc, text=acc, minimum width=11mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node (in) at (0,0) {input}; \node[frozen] (W) at (2.4,0) {W\\(frozen)}; \node[trained] (A) at (2.4,-1.9) {A}; \node[trained] (B) at (4.4,-1.9) {B}; \node[draw, circle, inner sep=1pt, font=\scriptsize] (plus) at (5.6,0) {+}; \node (out) at (7.0,0) {output}; \draw[->] (in) -- (W); \draw[->] (W) -- (plus); \draw[->, acc] (in |- A) ++(0,0) -- (A); \draw[->, acc] (0,0) |- (A); \draw[->, acc] (A) -- (B); \draw[->, acc] (B) -| (plus); \draw[->] (plus) -- (out); \node[acc, anchor=west, font=\scriptsize] at (2.9,-2.5) {low-rank correction BA (trained)}; \end{tikzpicture} $$ At inference the correction folds back into $W$, so PEFT adds no latency; only training and storage shrink. A single frozen base model can then serve many tasks, each with its own small adapter or LoRA module. ## Where fine-tuning stops and prompting starts Every form of fine-tuning changes the model. Even the parameter-efficient variants leave behind a small trained module that the base model needs in order to do the task. The second family of adaptation gives that up entirely — it specifies the task in the model's input and touches no weights at all. That family, together with the training stages that make a base model good at following instructions, is the subject of the next part. This continues in [Prompting and Alignment](/natural-language-processing/transformers/prompting-and-alignment). [^jm-ft]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), Ch. 11 — Fine-tuning and Masked Language Models: the pretrain-then-finetune paradigm, in which a self-supervised pretraining run produces a base model whose weights are the shared starting point for many cheap task-specific fine-tunes. [^jm-bert]: **Jurafsky & Martin**, Ch. 11 — Bidirectional Transformer Encoders: dropping the causal mask so every token attends to the whole sequence, yielding contextual (context-sensitive) token embeddings; the encoder-only BERT architecture. [^jm-mlm]: **Jurafsky & Martin**, Ch. 11 — Masked Language Modeling: corrupting a random ~15% of tokens with `[MASK]` and predicting the originals from two-sided context, and the sentence-level next-sentence-prediction (later contrastive) objective. [^jm-wp]: **Jurafsky & Martin**, §2 / Ch. 11 — sub-word tokenization (BPE / WordPiece) producing a fixed wordpiece vocabulary, and the special `[CLS]` (sequence summary) and `[SEP]` (segment separator) tokens. [^jm-ftclass]: **Jurafsky & Martin**, Ch. 11 — Fine-Tuning for Classification and Sequence Labeling: adding a task head (linear + softmax) on the `[CLS]` vector for sequence classification or on every token embedding for sequence labeling, and training head plus encoder together. [^jm-qa]: **Jurafsky & Martin**, §23.2 — reading-comprehension / span-based question answering: concatenating question and passage with `[SEP]`, learned span-start $S$ and span-end $E$ vectors, the softmax start/end probabilities (eqs. 23.16–23.17), the span score $S\cdot p'_i + E\cdot p'_j$, the fine-tuning loss (eq. 23.18), and pointing no-answer cases at `[CLS]`. [^jm-peft]: **Jurafsky & Martin**, Ch. 11 — parameter-efficient fine-tuning: freezing the pretrained weights and training only inserted adapter bottlenecks or a low-rank LoRA correction $\Delta W = BA$, recovering most of full fine-tuning's accuracy at a fraction of the trainable parameters.