Fine-Tuning and Prompting

A large language model is pretrained once, on web-scale text, to do one thing: predict tokens. That single objective forces it to absorb syntax, facts, and a great deal of world knowledge, all folded into its weights. But prediction is rarely the target task itself — the target is sentiment labels, named entities, answers to questions, a helpful reply. Adaptation turns the general-purpose predictor into a task-specific model. Pretraining is paid for once; adaptation is cheap and is done many different ways on the same frozen starting point.¹

There are two families of adaptation, and the difference between them is one bit: whether you change the weights.

• Fine-tuning updates the weights. Start from the pretrained model, add a small task-specific layer, and continue training on labelled examples for the target task. The result is a new, specialized model.
• Prompting leaves the weights frozen. Describe the task to the model in its input — an instruction, maybe a few worked examples — and read the answer off its next-token prediction. No gradient step, no new model.

These two parts build up both. This one covers fine-tuning, starting with the pretraining objective that makes it possible; the next covers prompting and the training stages that turn a base model into an assistant.

The pretrain-then-finetune paradigm

The scheme has two phases. In pretraining, a model learns from raw text with a self-supervised objective — a label mined from the text itself, no human annotation. In fine-tuning, that pretrained model is the starting point for supervised training on a specific labelled task.¹ The weights carry over; only the final task-specific piece is new.

The cost asymmetry is large. Pretraining a modern model costs millions of dollars and weeks of compute; fine-tuning it for a new task takes hours and a few thousand labelled examples, because the general representations are already learned. The fine-tune only has to learn the thin mapping from the model's existing representations to the task's outputs.

The feedforward and recurrent language models of earlier lessons were causal: they predict each word from the words to its left, because a language model that generates text must not peek at the future. That left-to-right constraint is right for generation but wrong for understanding. To classify a word — is bank a river bank or a financial one? — you want to see the words on both sides. The first family of adaptation is built on an encoder that does exactly that.

Bidirectional encoders and BERT

A bidirectional transformer encoder drops the causal mask. Every token attends to every other token, left and right, so each position's output vector is a contextual embedding that summarizes the whole sentence as seen from that word.² This is the architecture of BERT (Bidirectional Encoder Representations from Transformers) and its many descendants: a stack of transformer encoder blocks, no decoder, producing one context-sensitive vector per input token.

Because it sees the whole context, an encoder is not a text generator — you cannot sample the next word from it. Its job is to represent, producing embeddings that a downstream layer turns into labels. This poses a problem: a left-to-right model trains by predicting the next word, a signal available for free from any text. What is the self-supervised signal for a model that already sees every word? If BERT could see the whole sentence, predicting a word it already has as input would be trivial. The answer is to hide part of the input.

Masked language modeling

The pretraining objective for a bidirectional encoder is masked language modeling (MLM). Take a sentence, corrupt a random fraction of its tokens — in BERT, about $15\%$ — and train the model to reconstruct the originals from the surrounding context.³ The corruption is usually replacement with a special [MASK] token, and the loss is the cross-entropy of predicting the true token at each masked position, exactly the softmax-over-vocabulary loss of an ordinary language model, but applied only at the hidden positions.

Masking is a clean way to force bidirectional understanding. To fill in a blank correctly the model must use both the left context (the cat) and any right context, so it learns to build each position's embedding from the entire sentence. And the labels are free: any text is its own answer key once part of it is hidden — the same self-supervised principle from the taxonomy of learning.

BERT paired MLM with a second, sentence-level objective. In next-sentence prediction (NSP), the model is given two segments and asked to classify whether the second actually followed the first in the corpus or was randomly chosen.³ The goal was to teach the relationships between sentences that tasks like question answering and entailment need. Later encoders found NSP weak and often dropped or replaced it — for example with contrastive sentence objectives that pull the embeddings of related sentences together and push unrelated ones apart — but the pattern is the same: a cheap self-supervised signal invented from the raw text.

The tokens themselves are not words but wordpieces: sub-word units produced by an algorithm like BPE or WordPiece, so a rare word splits into known pieces and the vocabulary stays fixed and finite.⁴ Two special tokens matter for the tasks below: a [CLS] token prepended to every input, whose final embedding serves as a summary of the whole sequence, and a [SEP] token that separates two segments when the input is a pair.

How subword vocabularies are built: a BPE trace

The vocabulary is not hand-designed; it is learned from the corpus by byte-pair encoding (BPE; Sennrich et al., 2016), the algorithm SLP3 works through in Chapter 2.⁴ BPE starts with a vocabulary of individual characters and repeatedly merges the most frequent adjacent pair into a new symbol, growing longer tokens until it has performed $k$ merges. It runs inside words, so each word is first split into characters plus an end-of-word marker $\_$.

Take Jurafsky & Martin's tiny corpus of five distinct words with counts: low (5), lowest (2), newer (6), wider (3), new (2). The starting vocabulary is the eleven letters plus $\_$. The learner counts adjacent pairs across the whole corpus and merges the most frequent:

• Merge 1. The pair e r occurs in newer (6) and wider (3), total $9$ — the most frequent. Merge to er. Now newer is n e w er, wider is w i d er.
• Merge 2. er _ (word-final er) occurs $9$ times; merge to er_.
• Merge 3. n e occurs in newer (6) and new (2), total $8$; merge to ne.
• Merge 4. ne w totals $8$; merge to new.
• Merge 5. l o totals $7$ (low 5 + lowest 2); merge to lo.
• Merge 6. lo w totals $7$; merge to low.

After these merges the vocabulary contains the eleven characters plus er, er_, ne, new, lo, low. To tokenize new text, the learned merges are applied greedily in the order learned. The word newer collapses back to a single token; but an unseen word like lower tokenizes as low + er_ — two known pieces — so the model never meets a true unknown word, which is the entire point. The vocabulary size is fixed by $k$, yet any string is representable.

Fine-tuning: adding a task head

Pretraining gives one contextual embedding per token, plus the [CLS] summary. Fine-tuning turns those embeddings into task outputs by adding a task head — a small layer, often a single linear map plus softmax — on top of the frozen-then- unfrozen encoder, and training the whole assembly on labelled data.⁵ Three task shapes cover most of NLP, and each reads off a different part of the encoder's output.

Sequence classification (sentiment, topic, entailment) attaches a head to the [CLS] summary vector, mapping it to the class scores. Sequence labeling (part-of-speech tagging, named-entity recognition) attaches the same head to every token's embedding, producing one label per token. The third shape, span extraction, is used for extractive question answering; SLP3 gives it in full detail, so we work through it here.

Span-based question answering

In extractive (reading-comprehension) QA, the model is given a question and a passage, and must return the span of the passage that answers it.⁶ The input is built by concatenating the two segments — the question first, the passage second, separated by [SEP] — and running them through the encoder, which produces a contextual embedding $p_i^{\prime }$ for every passage token.

The head is two learned vectors: a span-start embedding $S$ and a span-end embedding $E$. For each passage token $p_i^{\prime }$, the probability that it begins the answer is a softmax of the dot product $S\cdot p_i^{\prime }$ over all passage tokens, and the end probability uses $E$ the same way:

A candidate span from token $i$ to token $j$ scores $S\cdot p_i^{\prime }+E\cdot p_j^{\prime }$, and the model predicts the highest-scoring span with $j\ge i$. Only $S$ and $E$ are new; everything below them is the pretrained encoder, nudged by the same gradient. The fine-tuning loss is the negative log-likelihood of the correct start and end positions,

summed over the training examples.⁶ When a dataset includes questions with no answer in the passage, the trick is to point the gold start and end at the [CLS] token, so the model can learn to abstain.

For example, take a four-token passage Ada wrote the program and a question who wrote it?. Suppose the encoder's contextual embeddings, dotted with the learned $S$ and $E$ vectors, give start scores $S\cdot p^{\prime }=(0.5,3.0,1.0,0.2)$ and end scores $E\cdot p^{\prime }=(0.3,0.8,2.5,0.4)$ over the four tokens. Softmax turns these into start probabilities $(0.064,0.782,0.106,0.048)$ — the model strongly prefers to start the answer at token $2$ (wrote) — and end probabilities $(0.078,0.129,0.706,0.086)$, peaking at token $3$ (the). The predicted span maximizes $S\cdot p_i^{\prime }+E\cdot p_j^{\prime }$ over $j\ge i$; the best pair is $i=2,j=3$ with score $3.0+2.5=5.5$, returning the span wrote the. The constraint $j\ge i$ rules out impossible spans where the end precedes the start, and the two independent softmaxes let a single forward pass score every start and every end at once.

The pattern generalizes: whatever the task, choose which embeddings to read, add a thin head, and let the gradient tune the head and the encoder together. Because the encoder starts from a rich pretrained state, a few thousand labelled examples usually suffice — the fine-tune is refining representations, not learning language from scratch.

Parameter-efficient fine-tuning

Full fine-tuning updates every weight, and produces a full-size copy of the model per task. For a billion-parameter encoder and a dozen tasks that is wasteful in both storage and compute. Parameter-efficient fine-tuning (PEFT) freezes the pretrained weights and trains only a small number of new ones, recovering most of the accuracy at a fraction of the cost.⁷

Two designs dominate. Adapters insert tiny bottleneck layers between the frozen transformer blocks — down-project to a small dimension, apply a non-linearity, up-project back — and train only those. LoRA (Low-Rank Adaptation) freezes each weight matrix $W$ and learns a low-rank correction $\Delta W=BA$, where $B$ and $A$ are thin matrices, so the effective weight is $W+BA$ with only $BA$ trainable.

At inference the correction folds back into $W$, so PEFT adds no latency; only training and storage shrink. A single frozen base model can then serve many tasks, each with its own small adapter or LoRA module.

Where fine-tuning stops and prompting starts

Every form of fine-tuning changes the model. Even the parameter-efficient variants leave behind a small trained module that the base model needs in order to do the task. The second family of adaptation gives that up entirely — it specifies the task in the model's input and touches no weights at all. That family, together with the training stages that make a base model good at following instructions, is the subject of the next part. This continues in Prompting and Alignment.