Prompting and Alignment

This builds on Fine-Tuning and Prompting, which covered the first family of adaptation — fine-tuning, which updates a pretrained model's weights with a small task head and a run of supervised training. Here we take up the second family, which changes no weights at all: the model is frozen and the task is written into its input. We then add the training stages that make a frozen model reliably promptable.

Prompting and in-context learning

Fine-tuning needs labelled data and a training run per task. The second family needs neither. A large pretrained generative language model performs a task from a description alone: the task is written into the input as a prompt $x$, and the model returns the continuation $y=\arg {\max }_y{p}_{\theta }(y\mid x)$ under frozen weights $\theta$. Specification is in-context, at inference, with no gradient step.¹

This works because a next-token predictor with parameters $\theta$ has, during pretraining, maximized ${\sum }_i\log p_{\theta }(w_i\mid w_{<i})$ over corpora containing the requested pattern. Formatting a prompt as Review: … Sentiment: makes a sentiment word the high-probability continuation. The frozen model does not learn the task; it recognizes a distribution it already absorbed and completes it.

Zero-shot prompting gives only an instruction, conditioning on $x=[\text{instr},q]$. Few-shot prompting prepends $k$ solved demonstrations, conditioning on $x=[(x_1,y_1),\dots ,(x_k,y_k),q]$ so the model infers the task from the pattern. Picking up a task from context with no gradient step is in-context learning.²

The demonstrations are not training data in the usual sense: they exist only in that one forward pass. The next prompt retains nothing from the previous one. In-context learning is adaptation carried entirely in the input.

Chain-of-thought prompting

Prompting directly for the answer $a$ to a multi-step problem $q$ often fails: the model must emit $p_{\theta }(a\mid q)$ in one commitment, before any intermediate computation. Chain-of-thought prompting inserts a reasoning trace $r$ between question and answer, factoring the prediction as

and demonstrations of the form (question, worked steps, answer) steer the model onto a high-probability trace $r$ before it commits to $a$.³

The generated intermediate tokens become context the answer tokens condition on, giving the model extra sequential computation that a one-shot answer lacks. Writing out reasoning raises multi-step accuracy sharply.

Instruction tuning and RLHF

A raw pretrained model predicts likely continuations, which is not the same as following an instruction. Prompted with Summarize this article, a base model may continue with another article-like paragraph — the likely continuation — rather than a summary. Two further training stages turn the base predictor into an assistant.

Instruction tuning is supervised fine-tuning on a broad, diverse set of tasks, each phrased as a natural-language instruction paired with a correct response.⁴ Trained on thousands of (instruction, response) pairs, the model learns the general behaviour of following an instruction and generalizes it to unseen instructions. It makes the model promptable: after instruction tuning, a plain zero-shot instruction succeeds far more often.

The final stage aligns the model with preferences that are hard to specify as a loss — helpfulness, honesty, harmlessness. Reinforcement learning from human feedback (RLHF) proceeds in two steps.⁵ First, from human comparisons $(y_w\succ y_l)$ of two outputs for a prompt $x$, a reward model $r_{\varphi }$ is fit under the Bradley–Terry preference model, minimizing

so $r_{\varphi }$ scores the preferred output higher. Second, the policy ${\pi }_{\theta }$ is optimized against that reward with RL, under a KL penalty to the supervised reference ${\pi }_{\text{ref}}$ that prevents reward hacking and preserves fluency:

The result is the base model turned into an aligned assistant: fluent, instruction-following, and steered toward responses people rate as good. RLHF is treated in full in the reinforcement-learning course and the deep-learning treatment of learning from human feedback; here it is enough to place it as the last adaptation step in the stack — pretrain, instruction-tune, align.

The papers behind the pipeline

The instruction-tuning-then-RLHF stack and the prompting tricks above each trace to a specific public result. Naming them precisely keeps the claims honest.

BERT (Devlin et al., 2019). The masked-language-modeling encoder of this lesson is the model of BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (NAACL 2019).⁶ It paired MLM (masking $\sim 15\%$ of tokens) with next-sentence prediction, pretrained on BooksCorpus plus English Wikipedia; a single pretrained model, fine-tuned with one added output layer, set new state-of-the-art on eleven understanding tasks. The pretrain-then-finetune scheme of the previous part is BERT's.

LoRA (Hu et al., 2021). The low-rank correction $\Delta W=BA$ is LoRA: Low-Rank Adaptation of Large Language Models.⁷ On GPT-3 175B, LoRA trained roughly $10,000$ times fewer parameters than full fine-tuning while matching or beating its quality, and because the correction folds into $W$ at inference it adds no latency. That efficiency is why one frozen base model can host a whole library of task-specific adapters.

Chain-of-thought (Wei et al., 2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (NeurIPS 2022) showed that prompting a large model with worked reasoning steps, not just final answers, sharply raises accuracy on arithmetic and commonsense reasoning — and that the effect emerges with scale, appearing only above roughly $100$ billion parameters and being absent or harmful in small models.⁸ Kojima et al. (2022) added the follow-up that merely appending Let's think step by step to a zero-shot prompt triggers much of the same reasoning without any worked examples at all.⁹

InstructGPT / RLHF (Ouyang et al., 2022). The alignment stage is Training Language Models to Follow Instructions with Human Feedback (NeurIPS 2022), the InstructGPT paper.¹⁰ Its pipeline has three steps: supervised fine-tuning on human-written demonstrations (instruction tuning); training a reward model on human rankings of candidate outputs; and optimizing the language model against that reward with the PPO reinforcement-learning algorithm, regularized by a KL penalty that keeps it near the supervised model. The headline result: human raters preferred outputs from the $1.3$-billion-parameter InstructGPT over the $175$-billion-parameter base GPT-3. A $100\times$ smaller model won on helpfulness because it was aligned; for a usable assistant, alignment mattered more than raw scale.

Retrieval-augmented generation (Lewis et al., 2020). Prompting and fine-tuning both leave a model's knowledge frozen in its weights at training time, which goes stale and cannot cite a source. RAG — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020) — attaches a retriever that pulls relevant documents from an external corpus and concatenates them into the prompt, so the model conditions its generation on retrieved evidence rather than memory alone.¹¹ This is a third adaptation axis, orthogonal to the two in this lesson: new context fetched at inference, rather than new weights or a longer instruction. It updates a model's knowledge by swapping the corpus, not retraining, and grounds answers in text that can be shown to a user.

Fine-tuning versus prompting

The two families make opposite trade-offs, and the choice turns on how much labelled data you have and whether you can afford to store and serve a specialized model.

Neither wins outright. Fine-tuning (especially the parameter-efficient kind) tends to give the highest accuracy on a task with enough labelled data, and it bakes the task in so inference is cheap. Prompting needs no labels and no training, adapts a single frozen model to any number of tasks on the fly, and is the only option when the model is too large to fine-tune or reachable only through an API. The two also combine: instruction tuning is fine-tuning that makes a model better at prompting, and a fine-tuned model can still be prompted.

Both rest on the same foundation. The expensive work of pretraining is done once and frozen into the weights. Everything after is cheap adaptation: a thin head and a gradient step, or a few sentences of context.