---
title: "Dialogue Systems: LLM Assistants, Evaluation, and Design"
module: Applications
moduleNumber: 7
lessonNumber: 6
order: 706
summary: >
  Two dialogue traditions — chatbots built to chat and task-oriented frame
  systems built to get something done — met in the aligned LLM assistant.
  Instruction tuning plus RLHF fold chit-chat and task dialogue into one model;
  the LaMDA / InstructGPT / ChatGPT lineage fills in how. The lesson then turns
  to evaluation (human ratings and acute-eval for chatbots, task success and
  slot error rate for task systems), user-centered design with Wizard-of-Oz
  prototyping, and the ethical stakes of building agents people talk to.
topics: [Applications]
sources:
  - book: Jurafsky & Martin
    ref: "§24.4 The Dialogue-State Architecture (policy, NLG); instruction tuning & RLHF"
  - book: Jurafsky & Martin
    ref: "§24.5 Evaluating Dialogue Systems; §24.6 Dialogue System Design"
---

This builds on [Dialogue and Chatbots](/natural-language-processing/applications/dialogue-and-chatbots),
which laid out the two traditions in full: the chatbot built to chat, and the
task-oriented frame system — GUS and the modern NLU / state-tracker / policy / NLG
pipeline — built to get something done. Each tradition answered one narrow question,
and a system had to choose between them. This part shows how a single aligned
language model removes the choice, then takes up how such systems are evaluated,
designed, and kept safe.

## LLM assistants unify the two traditions

The two lineages above answer two different questions — how to chat, how to complete
a task — and until recently a system had to choose. A large language model removes
the choice. Pretrained on a web-scale corpus, an LLM already generates fluent,
contextual, wide-ranging text; the open problem was making it _follow instructions_
and _behave_ rather than merely continue text. Two steps of
[fine-tuning](/natural-language-processing/transformers/fine-tuning-and-prompting)
do that.

**Instruction tuning** fine-tunes the base model on a large collection of tasks
each phrased as a natural-language instruction with its response, teaching the model
to treat a prompt as a command to be carried out rather than a passage to be
continued. **Reinforcement learning from human feedback**
([RLHF](/deep-learning/reinforcement-learning/rl-from-human-feedback)) then aligns
the model with human preference: collect human comparisons of candidate outputs,
fit a **reward model** to those comparisons, and fine-tune the language model with
reinforcement learning to maximize that reward.[^jm-rlhf] The reward model is the
same dialogue-policy idea — a learned scorer over system responses — moved to the
level of the whole language model and driven by preference rather than task success.

Formally, RLHF is two optimizations. Given a prompt $x$ and a human-ranked pair of
completions $y_w \succ y_l$, the reward model $r_\phi$ is fit by maximum likelihood
under the **Bradley–Terry** preference model,

$$
\mathcal{L}(\phi) \;=\; -\,\mathbb{E}_{(x,\,y_w,\,y_l)}\!\left[\log \sigma\!\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\right],
$$

so widening the reward gap $r_\phi(x,y_w) - r_\phi(x,y_l)$ raises the preferred
completion's likelihood. The policy $\pi_\theta$ is then trained to maximize expected
reward while a **KL penalty** anchors it to the instruction-tuned reference
$\pi_{\text{ref}}$, so it cannot drift into high-reward gibberish,

$$
\max_{\theta}\;\; \mathbb{E}_{x\sim\mathcal{D},\; y\sim\pi_\theta(\cdot\mid x)}\!\big[r_\phi(x, y)\big] \;-\; \beta\,\mathrm{KL}\!\big[\pi_\theta(y\mid x)\,\big\|\,\pi_{\text{ref}}(y\mid x)\big].
$$

The coefficient $\beta$ trades alignment against fidelity to the base model, and
**reward hacking** is precisely $r_\phi$ ceasing to track true preference once
$\pi_\theta$ optimizes it hard.

The result folds both traditions into one network. Open-ended chit-chat is just
generation, the corpus-based chatbot's job, now done by a far larger model that no
longer collapses into blandness because the reward model penalizes dull, unhelpful
replies. Task completion — booking, looking up, calculating — becomes a matter of
prompting or tool use rather than a hand-built frame pipeline, with slot-filling and
policy folded into the model's learned behavior. ELIZA matched a sentence with a
regular expression; the aligned LLM scores its output with a reward model fit to
human preference. The interface is the same; the machinery sits at the opposite
end of the design space.

## LaMDA, InstructGPT, and the aligned assistant

The claim that instruction tuning plus RLHF turns a base language model into a usable
conversational agent is supported by three public systems referenced in Jurafsky &
Martin's treatment.

**LaMDA — a language model specialized for dialogue.** Thoppilan and colleagues,
"LaMDA: Language Models for Dialog Applications" (Google, 2022), pretrained a decoder
model on dialogue and then fine-tuned it for three dialogue-specific objectives:
**quality** (sensibleness, specificity, interestingness — the "SSI" metrics, a direct
descendant of the human-quality dimensions the evaluation section describes),
**safety** (against a set of harm objectives), and **groundedness** (backing factual
claims with external sources). LaMDA's groundedness fine-tuning let it call an external
information-retrieval tool mid-conversation, which is the same retrieve-then-generate
idea from the sibling [question answering](/natural-language-processing/applications/question-answering)
lesson applied to keep a chatbot factual — a concrete answer to the blandness and
hallucination failures the chatbot section raises.[^bb-lamda]

**InstructGPT — RLHF made the recipe explicit.** Ouyang, Wu, Jiang, and colleagues,
"Training Language Models to Follow Instructions with Human Feedback" (**InstructGPT**,
NeurIPS 2022), give the three-step procedure this lesson's RLHF paragraph compresses:
(1) collect human demonstrations of desired behavior and **supervised-fine-tune** the
base model on them; (2) collect human **rankings** of several model outputs per prompt
and fit a **reward model** to those comparisons; (3) optimize the language model
against the reward model with the PPO reinforcement-learning algorithm. Their headline
finding is that labelers preferred the outputs of a $1.3$-billion-parameter InstructGPT
model over those of the $175$-billion-parameter base GPT-3 — a hundredfold smaller model
winning on helpfulness because it was aligned. This is the concrete evidence that
alignment, not scale alone, is what makes an assistant follow instructions.[^bb-instructgpt]

$$
% caption: The three-step RLHF pipeline behind an aligned assistant (InstructGPT).
% Step 1 supervised-fine-tunes on human demonstrations; step 2 fits a reward model to
% human rankings of outputs; step 3 optimizes the language model against that reward
% with reinforcement learning (PPO). The reward model is the dialogue-policy scorer
% moved to the whole language model.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  stg/.style={draw, minimum width=33mm, minimum height=13mm, align=center, font=\scriptsize},
  core/.style={draw=acc, text=acc, minimum width=33mm, minimum height=13mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[stg] (sft) at (0,0)      {1. supervised\\f\/ine-tune on\\demonstrations};
  \node[core] (rm) at (4.6,0)    {2. reward model\\from human\\rankings};
  \node[stg] (ppo) at (9.2,0)    {3. RL (PPO)\\against the\\reward model};
  \node[stg, draw=black] (out) at (9.2,-2.4) {aligned\\assistant};
  \draw[->, acc, thick] (sft) -- (rm);
  \draw[->, acc, thick] (rm) -- (ppo);
  \draw[->, black] (ppo) -- (out);
\end{tikzpicture}
$$

**ChatGPT and the collapse of the two traditions.** ChatGPT (OpenAI, released November
2022) applied this same instruction-tuning-plus-RLHF recipe to a conversational model
and made the aligned assistant a mass-market interface. Publicly, OpenAI describes it as
trained with the InstructGPT method adapted to dialogue, with human trainers providing
both sides of sample conversations. In practice it delivers the unification this
section argues for: one model does open-ended chit-chat (the chatbot tradition) and
carries out instructions and looks things up (the task tradition) without a hand-built
frame pipeline or a separate chit-chat module. The GUS frame and the ELIZA rule did not
disappear so much as get absorbed — slot-filling and policy became learned behavior of a
single generation model.[^bb-chatgpt]

**The failures did not vanish.** These systems still stumble on the §1 properties. An
aligned LLM hallucinates facts (a groundedness failure LaMDA's tool-use only partly
fixes), mis-grounds — confidently proceeding on a misunderstanding it never signals —
and loses track of constraints across a long exchange, the very common-ground and
state-tracking problems the frame architecture was built to make explicit. RLHF also
introduces its own issues: reward models can be gamed (**reward hacking**), and aligning
to average labeler preference bakes in whose preferences those are. The aligned assistant
is a wider answer to the opening question, not a final one.[^bb-instructgpt]

## Evaluating dialogue systems

Evaluation drives dialogue design, and because chatbots and task systems have
different goals, they are evaluated in entirely different ways.[^jm-eval] The split
covers ground truth, primary metric, and whether automatic scoring works at all:

| | **Chatbot** | **Task-based system** |
| --- | --- | --- |
| Ground truth | none | task outcome |
| Primary metric | human quality rating | task success rate |
| Automatic metrics | fail (BLEU/ROUGE uncorrelated) | slot / concept error rate |
| Who rates | participant or observer | user survey + interaction logs |
| Cost measures | — | efficiency cost, quality cost |

### Evaluating chatbots

A chatbot has no task to complete, so there is no ground truth to score against;
chatbots are evaluated by **humans** who assign a score. The rater is either a
**participant** (the person who just chatted with the model) or an **observer** (a
third party reading a transcript of someone else's conversation). Participant
evaluation asks the rater to score the bot along several **dimensions of
conversational quality**: in one standard protocol the evaluator chats for six turns
and rates the bot on eight axes — avoiding repetition, interestingness, making sense,
fluency, listening, inquisitiveness, humanness, and engagingness. The questions are
concrete ("How much did you enjoy talking to this user?"; "How often did this user
say something which did NOT make sense?") so that different raters agree.

Observer evaluation reads a finished conversation instead. Sometimes a rater scores
each system turn (for coherence, say); more often we want a single comparative
verdict. The **acute-eval** metric is an observer method that shows a rater two
whole conversations side by side and asks four preference questions phrased for high
agreement — _who would you rather talk to for a long conversation?_ (engagingness),
_who is more interesting?_, _who sounds more human?_, _who is more knowledgeable?_ —
and reports which system won.

$$
% caption: Human evaluation of chatbots. Participant evaluation has the person who
% chatted rate the bot on quality dimensions; observer evaluation (acute-eval) shows
% a third party two conversations and asks which system they would rather talk to.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=34mm, minimum height=11mm, align=center, font=\scriptsize},
  dim/.style={draw, fill=black!3, minimum width=30mm, minimum height=7mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % participant branch
  \node[box, draw=acc, text=acc] (part) at (0,1.4) {participant\\(rater chatted)};
  \node[dim] (d1) at (4.6,1.4) {engagingness,\\sense, humanness ...};
  \draw[->, acc, thick] (part) -- (d1);
  % observer branch
  \node[box] (obs) at (0,-1.4) {observer\\(reads transcript)};
  \node[dim] (aa) at (4.0,-0.7) {conversation A};
  \node[dim] (bb) at (4.0,-2.1) {conversation B};
  \node[box, draw=acc, text=acc] (pick) at (8.4,-1.4) {acute-eval:\\which is better?};
  \draw[->, black] (obs) -- (aa);
  \draw[->, black] (obs) -- (bb);
  \draw[->, acc, thick] (aa) -- (pick);
  \draw[->, acc, thick] (bb) -- (pick);
\end{tikzpicture}
$$

**Automatic metrics are not used** for chatbots, for a specific reason.
Word-overlap metrics like BLEU and ROUGE, and even embedding dot products between the
bot's response and a human response, correlate _very poorly_ with human judgment. The
cause is the same one that makes generation hard: there are too many acceptable
responses to any turn, so overlap with a single reference means little. Overlap
metrics work when the response space is small and lexically constrained — as
in machine translation — and dialogue is the opposite. Research does pursue automatic
proxies that go beyond overlap. The most promising is **adversarial evaluation**,
borrowed from the Turing test: train a classifier $D$ to tell human-written turns from
machine-written ones, and score a generator by its **fool rate**, the probability the
classifier labels a machine turn as human,

$$
\text{fool rate} = \Pr\big[\,D(y) = \text{human} \mid y \sim \pi_\theta\,\big],
$$

which approaches the human turn's own pass rate as the bot becomes indistinguishable.

### Evaluating task-based dialogue

A task system, by contrast, has a definite job, so **task success** is the primary
metric: did it book the right flight, add the right calendar event? When the task is
unambiguous this is a clean yes/no; the extrinsic form is the **task error rate**,
how often the correct outcome was actually achieved end to end.

A finer-grained proxy, useful before the task even finishes, is the **slot error
rate** — the fraction of slots filled with the wrong value:

$$
\text{Slot Error Rate} = \frac{\text{\# inserted, deleted, or substituted slots}}{\text{\# reference slots in the sentence}}.
$$

For _Make an appointment with Chris at 10:30 in Gates 104_, a system that extracts
`PERSON=Chris`, `ROOM=Gates 104`, but `TIME=11:30` has one wrong slot of three, a
slot error rate of $1/3$. (Slot precision, recall, and F-score serve the same role;
the metric is also called **concept error rate**.)

Beyond correctness, task systems are rated on two families of cost. **Efficiency
cost** measures how much work the interaction took — elapsed time, system turns,
query count, and the **turn correction ratio**, the share of turns spent only fixing
errors:

$$
\text{TCR} = \frac{\text{\# turns spent solely on correction}}{\text{\# total turns}}.
$$

**Quality cost** measures what sours the experience without failing the task: ASR
rejection rate (how often recognition returned nothing), barge-in rate, and the rate
of verbose or ambiguous system questions. These roll up into a **user satisfaction**
rating collected by questionnaire (ease of understanding, pace, whether the system
behaved as expected). One finding recurs: the user's
_perception_ of whether the task succeeded often predicts satisfaction better than
whether it actually did.

## Dialogue system design and ethics

The user matters more in dialogue systems than almost anywhere else in NLP, which is
why building one is as much **human-computer interaction** as machine learning. The
craft of designing prompts, dialogue strategies, and error messages is **voice user
interface design**, and it follows a **user-centered design** loop.[^jm-design]

The first step is to **study the user and the task** — interview prospective users,
examine similar systems, and study recordings of the human-human version of the
conversation. The second is to **build prototypes**, and the standard tool is
the **Wizard-of-Oz system**: the user talks to what they believe is a program, but a
hidden human "wizard," disguised by a software interface, is actually reading the
input and choosing the responses (the name is the man behind the curtain in Baum's
book). A wizard study lets designers test an architecture before it exists — only the
interface and database need to be real — and its transcripts can seed a pilot model.
It is not a perfect simulation: a human wizard cannot faithfully reproduce a real
system's recognition errors, latency, or limits, so wizard results are idealized. The
third step is to **iterate with real users**. An oft-cited lesson: an early system
required users to press a key to interrupt it, but testing showed users simply _barged
in_ with speech, forcing a redesign to recognize overlapping talk. Iterative testing
also shapes prompts so that users answer in the ways the system can handle.

$$
% caption: The user-centered design loop for a dialogue system. Study users and
% task, build a Wizard-of-Oz prototype (a hidden human behind the interface), test on
% real users, and cycle; evaluation feeds each pass back into design.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  stg/.style={draw, minimum width=30mm, minimum height=12mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[stg, draw=acc, text=acc] (study) at (0,2.4)    {1. study user\\and task};
  \node[stg] (proto) at (6.0,2.4)  {2. Wizard-of-Oz\\prototype};
  \node[stg] (test)  at (6.0,-0.4) {3. test on\\real users};
  \node[stg] (eval)  at (0,-0.4)   {evaluate\\and ref\/ine};
  \draw[->, acc, thick] (study) -- (proto);
  \draw[->, acc, thick] (proto) -- (test);
  \draw[->, acc, thick] (test)  -- (eval);
  \draw[->, acc, thick] (eval)  -- (study) node[midway, left, font=\scriptsize] {iterate};
  % wizard note
  \node[anchor=west, font=\scriptsize, text=black] at (7.7,2.4) {hidden human};
  \node[anchor=west, font=\scriptsize, text=black] at (7.7,2.0) {behind interface};
\end{tikzpicture}
$$

The design process carries ethical weight, because a conversational agent can harm the
people who talk to it — a worry as old as _Frankenstein_, whose lesson is the danger
of building an agent without considering its consequences.[^jm-ethics] Four issues
recur. **Safety**: users ask agents for medical advice or in emergencies, and one
study found that acting on the responses of commercial assistants to medical prompts
could, in many cases, have led to harm or death. **Abuse and representational harm**:
a system can verbally attack users or generate demeaning stereotypes — Microsoft's
Tay chatbot was pulled offline sixteen hours after launch when it began posting slurs
and conspiracy theories it had absorbed from users deliberately teaching it, and
dialogue corpora scraped from social media are known to carry toxic language that
models then reproduce and even amplify. **Privacy**: in-home agents overhear private
speech, and a human-seeming bot makes users likelier to disclose sensitive
information, so training transcripts must be anonymized. **Persona and gender**:
commercial assistants are overwhelmingly given female names and voices, reinforcing
the stereotype of a subservient female servant, and most respond to sexual harassment
by evading or even flirting rather than clearly declining.

The standing response is **value-sensitive design** — weighing benefits, harms, and
stakeholders during design rather than after deployment — together with concrete
mitigations (detecting and defusing toxic contexts) and, because human subjects are
involved by definition, review by an Institutional Review Board.

From ELIZA to the LLM assistant, the problem has stayed the same: map the
grounded, intention-laden signal of human conversation onto something a machine can
respond to. Pattern rules handled one narrow pose, frames one narrow task, and the
aligned language model a wide slice of both — while still failing on the same
properties (grounding a genuine misunderstanding, tracking common ground across a
long exchange) that make conversation hard.


[^jm-eval]: **Jurafsky & Martin**, §24.5 — Evaluating Dialogue Systems: §24.5.1 chatbot evaluation by human participant (eight quality dimensions, See et al. 2019) and observer (acute-eval, Li et al. 2019a) ratings, the failure of automatic overlap metrics (BLEU/ROUGE) for dialogue, and adversarial evaluation; §24.5.2 task-based evaluation by task success, slot/concept error rate, user satisfaction survey (Walker et al. 2001), and efficiency and quality costs (turn correction ratio, barge-ins).
[^jm-design]: **Jurafsky & Martin**, §24.6 — Dialogue System Design: dialogue as human-computer interaction; voice user interface design under user-centered design (Gould and Lewis 1985): study the user and task, build Wizard-of-Oz simulations and prototypes, and iteratively test on real users (the barge-in redesign, Stifelman et al. 1993).
[^jm-ethics]: **Jurafsky & Martin**, §24.6.1 — Ethical Issues in Dialogue System Design: user safety (dangerous medical advice, Bickmore et al. 2018), abuse and representational harm (the Tay chatbot; toxic training corpora that models amplify), privacy (in-home agents, disclosure to human-like bots), gender/persona bias, and value-sensitive design plus IRB oversight.
[^jm-rlhf]: **Jurafsky & Martin**, Ch. 24 / Ch. 11 — instruction tuning and reinforcement learning from human feedback: fine-tuning a base model to follow instructions, then fitting a reward model to human comparisons and optimizing the language model against it to produce an aligned assistant.
[^bb-lamda]: **Thoppilan et al.**, "LaMDA: Language Models for Dialog Applications," Google, 2022. A decoder model pretrained on dialogue and fine-tuned for quality (sensibleness, specificity, interestingness), safety, and groundedness, including calling an external information-retrieval tool during a conversation to back factual claims — a dialogue-specialized application of the retrieve-then-generate idea.
[^bb-instructgpt]: **Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, et al.**, "Training Language Models to Follow Instructions with Human Feedback," _NeurIPS_ 2022. The three-step RLHF recipe (supervised fine-tuning on demonstrations, a reward model fit to human output rankings, PPO optimization against it); labelers preferred a 1.3B-parameter InstructGPT model's outputs over the 175B-parameter base GPT-3, evidence that alignment rather than scale alone produces instruction-following. The paper also discusses residual issues including reward-model gaming and whose preferences the alignment reflects.
[^bb-chatgpt]: **OpenAI**, "Introducing ChatGPT," 2022 — a conversational model trained with the InstructGPT instruction-tuning-plus-RLHF method adapted to dialogue (human trainers supplying both sides of sample conversations), which brought the aligned assistant to mass use and folded open-ended chat and task completion into one generation model.