--- title: Prompting and Alignment module: Transformers moduleNumber: 5 lessonNumber: 6 order: 506 summary: > Fine-tuning adapts a model by changing its weights. The second family of adaptation changes nothing: a large frozen model performs a task from an instruction and a few examples placed in its context. This part covers prompting and in-context learning, chain-of-thought that elicits reasoning, and the two training stages — instruction tuning and RLHF — that turn a fluent base predictor into an aligned assistant, closing with the BERT, LoRA, chain-of-thought, InstructGPT, and retrieval-augmentation papers behind the modern adaptation pipeline. topics: [Transformers] sources: - book: Jurafsky ref: "Ch. 11 — Prompting, In-Context Learning, Chain-of-Thought, Instruction Tuning, and RLHF" --- This builds on [Fine-Tuning and Prompting](/natural-language-processing/transformers/fine-tuning-and-prompting), which covered the first family of adaptation — fine-tuning, which updates a pretrained model's weights with a small task head and a run of supervised training. Here we take up the second family, which changes no weights at all: the model is frozen and the task is written into its input. We then add the training stages that make a frozen model reliably promptable. ## Prompting and in-context learning Fine-tuning needs labelled data and a training run per task. The second family needs neither. A large pretrained **generative** language model performs a task from a description alone: the task is written into the input as a **prompt** $x$, and the model returns the continuation $y = \arg\max_{y} p_\theta(y \mid x)$ under frozen weights $\theta$. Specification is in-context, at inference, with no gradient step.[^jm-prompt] This works because a next-token predictor with parameters $\theta$ has, during pretraining, maximized $\sum_i \log p_\theta(w_i \mid w_{=stealth, font=\footnotesize, demo/.style={draw, minimum width=52mm, minimum height=7mm, align=left, font=\scriptsize, inner sep=3pt}, query/.style={draw, draw=acc, minimum width=52mm, minimum height=7mm, align=left, font=\scriptsize, inner sep=3pt}, ans/.style={draw, draw=acc, text=acc, minimum width=18mm, minimum height=7mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[demo] (d1) at (0,2.1) {Review: great f\/ilm. Sentiment: positive}; \node[demo] (d2) at (0,1.2) {Review: total waste. Sentiment: negative}; \node[query] (q) at (0,0.3) {Review: I loved every minute. Sentiment:}; \node[ans] (a) at (0,-0.9) {positive}; \node[black, anchor=west, font=\scriptsize] at (2.9,1.65) {demonstrations}; \node[acc, anchor=west, font=\scriptsize] at (2.9,0.3) {query}; \draw[->, acc, thick] (q) -- (a) node[midway, right, font=\scriptsize] {model completes}; \end{tikzpicture} $$ The demonstrations are not training data in the usual sense: they exist only in that one forward pass. The next prompt retains nothing from the previous one. In-context learning is adaptation carried entirely in the input. > **Definition (In-context learning).** A frozen language model performing a task > from instructions and/or examples placed in its input context, inferring the > task from the prompt with no update to its weights. **Zero-shot** gives only an > instruction; **few-shot** prepends a small number of solved examples. ### Chain-of-thought prompting Prompting directly for the answer $a$ to a multi-step problem $q$ often fails: the model must emit $p_\theta(a \mid q)$ in one commitment, before any intermediate computation. **Chain-of-thought** prompting inserts a reasoning trace $r$ between question and answer, factoring the prediction as $$ p_\theta(a \mid q) = \sum_r p_\theta(a \mid q, r)\, p_\theta(r \mid q), $$ and demonstrations of the form (question, worked steps, answer) steer the model onto a high-probability trace $r$ before it commits to $a$.[^jm-cot] $$ % caption: Chain-of-thought prompting. The demonstration shows intermediate % reasoning steps, not just the final answer, so the model generates its own steps % for the query before committing to an answer. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=62mm, minimum height=9mm, align=left, font=\scriptsize, inner sep=3pt}, qbox/.style={draw, draw=acc, minimum width=62mm, minimum height=8mm, align=left, font=\scriptsize, inner sep=3pt}, stepbox/.style={draw, draw=acc, text=acc, minimum width=62mm, minimum height=9mm, align=left, font=\scriptsize, inner sep=3pt}] \definecolor{acc}{HTML}{2348F2} \node[box] (d) at (0,2.0) {Q: 2 pens, 3 more bought. A: 2+3 = 5. Ans: 5}; \node[qbox] (q) at (0,0.9) {Q: 4 apples, eat 1, buy 2. A: ...}; \node[stepbox] (s) at (0,-0.4) {4-1 = 3, then 3+2 = 5. Ans: 5}; \node[black, anchor=west, font=\scriptsize] at (3.35,2.0) {shows reasoning}; \node[acc, anchor=west, font=\scriptsize] at (3.35,0.9) {query}; \draw[->, acc, thick] (q) -- (s) node[midway, right, font=\scriptsize] {generates steps then answer}; \end{tikzpicture} $$ The generated intermediate tokens become context the answer tokens condition on, giving the model extra sequential computation that a one-shot answer lacks. Writing out reasoning raises multi-step accuracy sharply. ## Instruction tuning and RLHF A raw pretrained model predicts likely continuations, which is not the same as following an instruction. Prompted with "Summarize this article," a base model may continue with another article-like paragraph — the likely continuation — rather than a summary. Two further training stages turn the base predictor into an assistant. **Instruction tuning** is supervised fine-tuning on a broad, diverse set of tasks, each phrased as a natural-language instruction paired with a correct response.[^jm-instr] Trained on thousands of (instruction, response) pairs, the model learns the general behaviour of _following an instruction_ and generalizes it to unseen instructions. It makes the model promptable: after instruction tuning, a plain zero-shot instruction succeeds far more often. The final stage aligns the model with preferences that are hard to specify as a loss — helpfulness, honesty, harmlessness. **Reinforcement learning from human feedback** (RLHF) proceeds in two steps.[^jm-rlhf] First, from human comparisons $(y_w \succ y_l)$ of two outputs for a prompt $x$, a **reward model** $r_\phi$ is fit under the **Bradley–Terry** preference model, minimizing $$ \mathcal{L}(\phi) = -\,\mathbb{E}_{(x,\,y_w,\,y_l)} \big[\log \sigma\!\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\big], $$ so $r_\phi$ scores the preferred output higher. Second, the policy $\pi_\theta$ is optimized against that reward with RL, under a KL penalty to the supervised reference $\pi_{\text{ref}}$ that prevents reward hacking and preserves fluency: $$ \max_{\theta}\ \mathbb{E}_{x,\,y \sim \pi_\theta} \big[r_\phi(x, y)\big] - \beta\, \mathrm{KL}\!\big(\pi_\theta(\cdot \mid x)\,\|\, \pi_{\text{ref}}(\cdot \mid x)\big). $$ The result is the base model turned into an aligned assistant: fluent, instruction-following, and steered toward responses people rate as good. RLHF is treated in full in the [reinforcement-learning course](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models) and the [deep-learning treatment of learning from human feedback](/deep-learning/reinforcement-learning/rl-from-human-feedback); here it is enough to place it as the last adaptation step in the stack — pretrain, instruction-tune, align. ## The papers behind the pipeline The instruction-tuning-then-RLHF stack and the prompting tricks above each trace to a specific public result. Naming them precisely keeps the claims honest. **BERT (Devlin et al., 2019).** The masked-language-modeling encoder of this lesson is the model of "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (NAACL 2019).[^devlin] It paired MLM (masking $\sim 15\%$ of tokens) with next-sentence prediction, pretrained on BooksCorpus plus English Wikipedia; a single pretrained model, fine-tuned with one added output layer, set new state-of-the-art on eleven understanding tasks. The pretrain-then-finetune scheme of the previous part is BERT's. **LoRA (Hu et al., 2021).** The low-rank correction $\Delta W = BA$ is "LoRA: Low-Rank Adaptation of Large Language Models."[^lora] On GPT-3 175B, LoRA trained roughly $10{,}000$ times fewer parameters than full fine-tuning while matching or beating its quality, and because the correction folds into $W$ at inference it adds no latency. That efficiency is why one frozen base model can host a whole library of task-specific adapters. **Chain-of-thought (Wei et al., 2022).** "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (NeurIPS 2022) showed that prompting a large model with worked reasoning steps, not just final answers, sharply raises accuracy on arithmetic and commonsense reasoning — and that the effect **emerges with scale**, appearing only above roughly $100$ billion parameters and being absent or harmful in small models.[^cot] Kojima et al. (2022) added the follow-up that merely appending "Let's think step by step" to a zero-shot prompt triggers much of the same reasoning without any worked examples at all.[^zerocot] **InstructGPT / RLHF (Ouyang et al., 2022).** The alignment stage is "Training Language Models to Follow Instructions with Human Feedback" (NeurIPS 2022), the InstructGPT paper.[^instructgpt] Its pipeline has three steps: supervised fine-tuning on human-written demonstrations (instruction tuning); training a **reward model** on human _rankings_ of candidate outputs; and optimizing the language model against that reward with the PPO reinforcement-learning algorithm, regularized by a KL penalty that keeps it near the supervised model. The headline result: human raters preferred outputs from the $1.3$-billion-parameter InstructGPT over the $175$-billion-parameter base GPT-3. A $100\times$ smaller model won on helpfulness because it was aligned; for a usable assistant, alignment mattered more than raw scale. $$ % caption: The InstructGPT / RLHF pipeline (Ouyang et al., 2022). A pretrained base model is % first supervised-fine-tuned on human demonstrations, then a reward model is trained on human % rankings of outputs, and finally the policy is optimized against that reward with PPO under % a KL penalty to the supervised model. \begin{tikzpicture}[>=stealth, font=\footnotesize, stg/.style={draw, minimum width=30mm, minimum height=12mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[stg] (base) at (0,0) {pretrained\\base model}; \node[stg] (sft) at (3.9,0) {supervised\\f\/ine-tune (SFT)}; \node[stg] (rm) at (7.8,0) {reward model\\(human rankings)}; \node[stg, draw=acc, text=acc] (rl) at (7.8,-2.3) {RL (PPO)\\+ KL penalty}; \node[stg, draw=acc, text=acc] (out) at (3.9,-2.3) {aligned\\assistant}; \draw[->, black] (base) -- (sft) node[midway,above,font=\scriptsize] {demos}; \draw[->, black] (sft) -- (rm) node[midway,above,font=\scriptsize] {outputs}; \draw[->, black] (rm) -- (rl) node[midway,right,font=\scriptsize] {reward}; \draw[->, acc] (rl) -- (out); \draw[->, black] (sft.south) to[bend right=18] node[midway,left,font=\scriptsize] {policy init} (rl.north west); \end{tikzpicture} $$ **Retrieval-augmented generation (Lewis et al., 2020).** Prompting and fine-tuning both leave a model's knowledge frozen in its weights at training time, which goes stale and cannot cite a source. **RAG** — "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020) — attaches a retriever that pulls relevant documents from an external corpus and concatenates them into the prompt, so the model conditions its generation on retrieved evidence rather than memory alone.[^rag] This is a third adaptation axis, orthogonal to the two in this lesson: new _context_ fetched at inference, rather than new weights or a longer instruction. It updates a model's knowledge by swapping the corpus, not retraining, and grounds answers in text that can be shown to a user. $$ % caption: Retrieval-augmented generation. A retriever encodes the query, finds the most % relevant documents in an external corpus, and prepends them to the prompt; the frozen % generator conditions on the retrieved passages rather than on parametric memory alone. \begin{tikzpicture}[>=stealth, font=\footnotesize, bx/.style={draw, minimum width=24mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[bx] (q) at (0,0) {query}; \node[bx] (ret) at (3.3,0) {retriever}; \node[bx] (corp) at (3.3,-1.9) {document\\corpus}; \node[bx] (docs) at (6.6,0) {top docs}; \node[bx, draw=acc, text=acc] (gen) at (9.9,0) {generator}; \node[bx, draw=acc, text=acc] (ans) at (9.9,-1.9) {grounded answer}; \draw[->, black] (q) -- (ret); \draw[->, black] (corp) -- (ret); \draw[->, black] (ret) -- (docs); \draw[->, acc] (docs) -- (gen); \draw[->, black] (q.north) to[bend left=22] (gen.north); \draw[->, acc] (gen) -- (ans); \end{tikzpicture} $$ ## Fine-tuning versus prompting The two families make opposite trade-offs, and the choice turns on how much labelled data you have and whether you can afford to store and serve a specialized model. | | Fine-tuning | Prompting | | --- | --- | --- | | Weights | updated (new specialized model) | frozen (one shared model) | | Adaptation signal | labelled training data | instruction + a few in-context examples | | When it happens | a training run, ahead of time | at inference, per request | | Cost per task | a gradient-descent run + a model copy | none beyond a longer prompt | | Data needed | hundreds–thousands of labels | zero to a handful of examples | | Best when | you have data and want peak accuracy | little/no data, or many tasks on one model | Neither wins outright. Fine-tuning (especially the parameter-efficient kind) tends to give the highest accuracy on a task with enough labelled data, and it bakes the task in so inference is cheap. Prompting needs no labels and no training, adapts a single frozen model to any number of tasks on the fly, and is the only option when the model is too large to fine-tune or reachable only through an API. The two also combine: instruction tuning is fine-tuning that makes a model better at prompting, and a fine-tuned model can still be prompted. Both rest on the same foundation. The expensive work of pretraining is done once and frozen into the weights. Everything after is cheap adaptation: a thin head and a gradient step, or a few sentences of context. [^jm-prompt]: **Jurafsky & Martin**, Prompting — specifying a task in the model's input as text and reading the answer off the frozen model's continuation, with zero-shot (instruction only) and few-shot (instruction plus demonstrations) variants. [^jm-icl]: **Jurafsky & Martin**, In-Context Learning — a frozen model inferring a task from demonstrations placed in its context with no gradient update, the demonstrations present only within the single forward pass. [^jm-cot]: **Jurafsky & Martin**, Chain-of-Thought Prompting — demonstrations that show intermediate reasoning steps before the answer, so the model generates its own reasoning path and answers multi-step problems more reliably. [^jm-instr]: **Jurafsky & Martin**, Instruction Tuning — fine-tuning on a broad set of tasks phrased as (instruction, response) pairs so the model learns to follow instructions and generalizes the behaviour to unseen instructions. [^jm-rlhf]: **Jurafsky & Martin**, Ch. 11 — aligning a base model with human preferences via reinforcement learning from human feedback: a reward model trained on human comparisons, then RL optimization of the language model against that reward, producing an aligned assistant. [^devlin]: **Devlin, Chang, Lee, Toutanova (2019)**, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," _NAACL 2019_. Pretrains a bidirectional transformer encoder with masked language modeling (~15% of tokens) plus next-sentence prediction on BooksCorpus and Wikipedia; fine-tuning the single pretrained model with one added output layer set state-of-the-art on eleven NLP benchmarks. [^lora]: **Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen (2021)**, "LoRA: Low-Rank Adaptation of Large Language Models," _ICLR 2022_. Freezes pretrained weights and learns a low-rank update $\Delta W = BA$; on GPT-3 175B it reduces trainable parameters by about four orders of magnitude versus full fine-tuning while matching quality, and folds into $W$ at inference so it adds no latency. [^cot]: **Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le, Zhou (2022)**, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," _NeurIPS 2022_. Few-shot prompts whose demonstrations include intermediate reasoning steps substantially raise accuracy on arithmetic, commonsense, and symbolic reasoning; the benefit emerges only in sufficiently large models (roughly 100B+ parameters). [^zerocot]: **Kojima, Gu, Reid, Matsuo, Iwasawa (2022)**, "Large Language Models Are Zero-Shot Reasoners," _NeurIPS 2022_. Shows that simply appending "Let's think step by step" to a zero-shot prompt elicits multi-step reasoning and improves accuracy without any worked in-context examples. [^instructgpt]: **Ouyang, Wu, Jiang, et al. (2022)**, "Training Language Models to Follow Instructions with Human Feedback" (InstructGPT), _NeurIPS 2022_. A three-stage alignment pipeline — supervised fine-tuning on demonstrations, a reward model trained on human output rankings, and PPO reinforcement learning against the reward with a KL penalty; human raters preferred the 1.3B InstructGPT over the 175B base GPT-3. [^rag]: **Lewis, Perez, Piktus, et al. (2020)**, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," _NeurIPS 2020_. Couples a neural retriever over an external document corpus with a sequence generator so that outputs are conditioned on retrieved passages; improves knowledge-intensive tasks and lets the model's knowledge be updated by changing the corpus rather than retraining.