Dialogue Systems: LLM Assistants, Evaluation, and Design

This builds on Dialogue and Chatbots, which laid out the two traditions in full: the chatbot built to chat, and the task-oriented frame system — GUS and the modern NLU / state-tracker / policy / NLG pipeline — built to get something done. Each tradition answered one narrow question, and a system had to choose between them. This part shows how a single aligned language model removes the choice, then takes up how such systems are evaluated, designed, and kept safe.

LLM assistants unify the two traditions

The two lineages above answer two different questions — how to chat, how to complete a task — and until recently a system had to choose. A large language model removes the choice. Pretrained on a web-scale corpus, an LLM already generates fluent, contextual, wide-ranging text; the open problem was making it follow instructions and behave rather than merely continue text. Two steps of fine-tuning do that.

Instruction tuning fine-tunes the base model on a large collection of tasks each phrased as a natural-language instruction with its response, teaching the model to treat a prompt as a command to be carried out rather than a passage to be continued. Reinforcement learning from human feedback (RLHF) then aligns the model with human preference: collect human comparisons of candidate outputs, fit a reward model to those comparisons, and fine-tune the language model with reinforcement learning to maximize that reward.¹ The reward model is the same dialogue-policy idea — a learned scorer over system responses — moved to the level of the whole language model and driven by preference rather than task success.

Formally, RLHF is two optimizations. Given a prompt $x$ and a human-ranked pair of completions $y_w\succ y_l$, the reward model $r_{\varphi }$ is fit by maximum likelihood under the Bradley–Terry preference model,

so widening the reward gap $r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)$ raises the preferred completion's likelihood. The policy ${\pi }_{\theta }$ is then trained to maximize expected reward while a KL penalty anchors it to the instruction-tuned reference ${\pi }_{\text{ref}}$, so it cannot drift into high-reward gibberish,

The coefficient $\beta$ trades alignment against fidelity to the base model, and reward hacking is precisely $r_{\varphi }$ ceasing to track true preference once ${\pi }_{\theta }$ optimizes it hard.

The result folds both traditions into one network. Open-ended chit-chat is just generation, the corpus-based chatbot's job, now done by a far larger model that no longer collapses into blandness because the reward model penalizes dull, unhelpful replies. Task completion — booking, looking up, calculating — becomes a matter of prompting or tool use rather than a hand-built frame pipeline, with slot-filling and policy folded into the model's learned behavior. ELIZA matched a sentence with a regular expression; the aligned LLM scores its output with a reward model fit to human preference. The interface is the same; the machinery sits at the opposite end of the design space.

LaMDA, InstructGPT, and the aligned assistant

The claim that instruction tuning plus RLHF turns a base language model into a usable conversational agent is supported by three public systems referenced in Jurafsky & Martin's treatment.

LaMDA — a language model specialized for dialogue. Thoppilan and colleagues, LaMDA: Language Models for Dialog Applications (Google, 2022), pretrained a decoder model on dialogue and then fine-tuned it for three dialogue-specific objectives: quality (sensibleness, specificity, interestingness — the SSI metrics, a direct descendant of the human-quality dimensions the evaluation section describes), safety (against a set of harm objectives), and groundedness (backing factual claims with external sources). LaMDA's groundedness fine-tuning let it call an external information-retrieval tool mid-conversation, which is the same retrieve-then-generate idea from the sibling question answering lesson applied to keep a chatbot factual — a concrete answer to the blandness and hallucination failures the chatbot section raises.²

InstructGPT — RLHF made the recipe explicit. Ouyang, Wu, Jiang, and colleagues, Training Language Models to Follow Instructions with Human Feedback (InstructGPT, NeurIPS 2022), give the three-step procedure this lesson's RLHF paragraph compresses: (1) collect human demonstrations of desired behavior and supervised-fine-tune the base model on them; (2) collect human rankings of several model outputs per prompt and fit a reward model to those comparisons; (3) optimize the language model against the reward model with the PPO reinforcement-learning algorithm. Their headline finding is that labelers preferred the outputs of a $1.3$-billion-parameter InstructGPT model over those of the $175$-billion-parameter base GPT-3 — a hundredfold smaller model winning on helpfulness because it was aligned. This is the concrete evidence that alignment, not scale alone, is what makes an assistant follow instructions.³

ChatGPT and the collapse of the two traditions. ChatGPT (OpenAI, released November 2022) applied this same instruction-tuning-plus-RLHF recipe to a conversational model and made the aligned assistant a mass-market interface. Publicly, OpenAI describes it as trained with the InstructGPT method adapted to dialogue, with human trainers providing both sides of sample conversations. In practice it delivers the unification this section argues for: one model does open-ended chit-chat (the chatbot tradition) and carries out instructions and looks things up (the task tradition) without a hand-built frame pipeline or a separate chit-chat module. The GUS frame and the ELIZA rule did not disappear so much as get absorbed — slot-filling and policy became learned behavior of a single generation model.⁴

The failures did not vanish. These systems still stumble on the §1 properties. An aligned LLM hallucinates facts (a groundedness failure LaMDA's tool-use only partly fixes), mis-grounds — confidently proceeding on a misunderstanding it never signals — and loses track of constraints across a long exchange, the very common-ground and state-tracking problems the frame architecture was built to make explicit. RLHF also introduces its own issues: reward models can be gamed (reward hacking), and aligning to average labeler preference bakes in whose preferences those are. The aligned assistant is a wider answer to the opening question, not a final one.³

Evaluating dialogue systems

Evaluation drives dialogue design, and because chatbots and task systems have different goals, they are evaluated in entirely different ways.⁵ The split covers ground truth, primary metric, and whether automatic scoring works at all:

Evaluating chatbots

A chatbot has no task to complete, so there is no ground truth to score against; chatbots are evaluated by humans who assign a score. The rater is either a participant (the person who just chatted with the model) or an observer (a third party reading a transcript of someone else's conversation). Participant evaluation asks the rater to score the bot along several dimensions of conversational quality: in one standard protocol the evaluator chats for six turns and rates the bot on eight axes — avoiding repetition, interestingness, making sense, fluency, listening, inquisitiveness, humanness, and engagingness. The questions are concrete (How much did you enjoy talking to this user?; How often did this user say something which did NOT make sense?) so that different raters agree.

Observer evaluation reads a finished conversation instead. Sometimes a rater scores each system turn (for coherence, say); more often we want a single comparative verdict. The acute-eval metric is an observer method that shows a rater two whole conversations side by side and asks four preference questions phrased for high agreement — who would you rather talk to for a long conversation? (engagingness), who is more interesting?, who sounds more human?, who is more knowledgeable? — and reports which system won.

Automatic metrics are not used for chatbots, for a specific reason. Word-overlap metrics like BLEU and ROUGE, and even embedding dot products between the bot's response and a human response, correlate very poorly with human judgment. The cause is the same one that makes generation hard: there are too many acceptable responses to any turn, so overlap with a single reference means little. Overlap metrics work when the response space is small and lexically constrained — as in machine translation — and dialogue is the opposite. Research does pursue automatic proxies that go beyond overlap. The most promising is adversarial evaluation, borrowed from the Turing test: train a classifier $D$ to tell human-written turns from machine-written ones, and score a generator by its fool rate, the probability the classifier labels a machine turn as human,

which approaches the human turn's own pass rate as the bot becomes indistinguishable.

Evaluating task-based dialogue

A task system, by contrast, has a definite job, so task success is the primary metric: did it book the right flight, add the right calendar event? When the task is unambiguous this is a clean yes/no; the extrinsic form is the task error rate, how often the correct outcome was actually achieved end to end.

A finer-grained proxy, useful before the task even finishes, is the slot error rate — the fraction of slots filled with the wrong value:

For Make an appointment with Chris at 10:30 in Gates 104, a system that extracts PERSON=Chris, ROOM=Gates 104, but TIME=11:30 has one wrong slot of three, a slot error rate of $1/3$. (Slot precision, recall, and F-score serve the same role; the metric is also called concept error rate.)

Beyond correctness, task systems are rated on two families of cost. Efficiency cost measures how much work the interaction took — elapsed time, system turns, query count, and the turn correction ratio, the share of turns spent only fixing errors:

Quality cost measures what sours the experience without failing the task: ASR rejection rate (how often recognition returned nothing), barge-in rate, and the rate of verbose or ambiguous system questions. These roll up into a user satisfaction rating collected by questionnaire (ease of understanding, pace, whether the system behaved as expected). One finding recurs: the user's perception of whether the task succeeded often predicts satisfaction better than whether it actually did.

Dialogue system design and ethics

The user matters more in dialogue systems than almost anywhere else in NLP, which is why building one is as much human-computer interaction as machine learning. The craft of designing prompts, dialogue strategies, and error messages is voice user interface design, and it follows a user-centered design loop.⁶

The first step is to study the user and the task — interview prospective users, examine similar systems, and study recordings of the human-human version of the conversation. The second is to build prototypes, and the standard tool is the Wizard-of-Oz system: the user talks to what they believe is a program, but a hidden human wizard, disguised by a software interface, is actually reading the input and choosing the responses (the name is the man behind the curtain in Baum's book). A wizard study lets designers test an architecture before it exists — only the interface and database need to be real — and its transcripts can seed a pilot model. It is not a perfect simulation: a human wizard cannot faithfully reproduce a real system's recognition errors, latency, or limits, so wizard results are idealized. The third step is to iterate with real users. An oft-cited lesson: an early system required users to press a key to interrupt it, but testing showed users simply barged in with speech, forcing a redesign to recognize overlapping talk. Iterative testing also shapes prompts so that users answer in the ways the system can handle.

The design process carries ethical weight, because a conversational agent can harm the people who talk to it — a worry as old as Frankenstein, whose lesson is the danger of building an agent without considering its consequences.⁷ Four issues recur. Safety: users ask agents for medical advice or in emergencies, and one study found that acting on the responses of commercial assistants to medical prompts could, in many cases, have led to harm or death. Abuse and representational harm: a system can verbally attack users or generate demeaning stereotypes — Microsoft's Tay chatbot was pulled offline sixteen hours after launch when it began posting slurs and conspiracy theories it had absorbed from users deliberately teaching it, and dialogue corpora scraped from social media are known to carry toxic language that models then reproduce and even amplify. Privacy: in-home agents overhear private speech, and a human-seeming bot makes users likelier to disclose sensitive information, so training transcripts must be anonymized. Persona and gender: commercial assistants are overwhelmingly given female names and voices, reinforcing the stereotype of a subservient female servant, and most respond to sexual harassment by evading or even flirting rather than clearly declining.

The standing response is value-sensitive design — weighing benefits, harms, and stakeholders during design rather than after deployment — together with concrete mitigations (detecting and defusing toxic contexts) and, because human subjects are involved by definition, review by an Institutional Review Board.

From ELIZA to the LLM assistant, the problem has stayed the same: map the grounded, intention-laden signal of human conversation onto something a machine can respond to. Pattern rules handled one narrow pose, frames one narrow task, and the aligned language model a wide slice of both — while still failing on the same properties (grounding a genuine misunderstanding, tracking common ground across a long exchange) that make conversation hard.