Agent Architectures

This builds on Intelligent Agents, which fixed what an agent is (a percept-to-action loop, captured by an agent function), what makes one good (rationality — maximizing expected performance), and how to specify the problem it faces (PEAS and the six environment axes). This lesson takes up the remaining question: how do you actually build a program that computes a good agent function? The answer is not one design but a ladder of four, each adding machinery the one below it lacked.

The structure of agents

The job of AI is to design an agent program that implements the agent function, running on an architecture — a device with physical sensors and actuators. In short, $\text{agent}=\text{architecture}+\text{program}$. The agent program takes just the current percept as input (that is all the sensors offer at any instant) and returns an action; if the agent's behavior must depend on the whole history, the program has to remember the percepts itself.

Why not implement the agent function as a literal lookup table? Because it is astronomically too big. Let $\mathcal{P}$ be the set of percepts and $T$ the agent's lifetime; the table needs ${\sum }_{t=1}^T|\mathcal{P}{|}^t$ entries. For an automated taxi with a single camera, that is on the order of $10^{250,000,000,000}$ entries for an hour's driving — no physical agent could store it, no designer could fill it, no learner could ever populate it. The challenge of AI is to produce rational behavior from a small program rather than a vast table, just as a five-line Newton's-method routine replaced the printed tables of square roots. Four program skeletons underlie almost every intelligent system, in order of increasing sophistication.

Simple reflex agents

The simplest agent selects an action from the current percept alone, ignoring the rest of the history. It works by condition–action rules (also called situation–action rules or if–then rules): if car-in-front-is-braking then initiate-braking. For the vacuum world the entire program is three rules — if dirty then suck; else if in A then go right; else go left — a dramatic reduction from the $4^T$ table.

Simple reflex agents are admirably compact but of limited intelligence: they work only if the correct action can be decided from the current percept alone — only if the environment is fully observable. A little unobservability causes trouble. A vacuum agent stripped of its location sensor, perceiving only $[\text{Dirty}]$ or $[\text{Clean}]$, has no rule that can reliably progress — moving Left loops forever if it starts in $A$. Escaping such loops sometimes requires randomization: perceiving $[\text{Clean}]$, flip a coin between Left and Right. A randomized simple reflex agent can outperform a deterministic one in partially observable worlds — though in single-agent settings, a better deterministic agent with memory usually does better still.

Model-based reflex agents

The way to handle partial observability is to keep track of the part of the world the agent can't currently see — to maintain internal state that depends on the percept history and reflects the unobserved aspects of the current state. Updating that state requires two pieces of knowledge encoded in the program: how the world evolves independently of the agent (an overtaking car will be closer a moment later), and how the agent's own actions affect the world (turning the wheel clockwise turns the car right). This knowledge — how the world works — is a model of the world, and an agent that uses one is a model-based agent.

Even a model-based agent seldom knows the current state exactly in a partially observable world; the box what the world is like now holds its best guess. The taxi may not see around the truck stalled ahead of it and can only guess at the hold-up — but it must still decide. Internal state need not describe the world literally, either: driving home is really a fact about the taxi's destination, an aspect of the agent's own intended state, not the world's.

Goal-based agents

Knowing the current state is not always enough to decide what to do. At a junction the taxi can turn left, turn right, or go straight — and the right choice depends on where the taxi is trying to get to. Beyond a current-state description, the agent needs goal information describing desirable situations, and it combines the goal with its model to choose actions that (eventually) achieve the goal.

This differs sharply from reflex rules, because it involves consideration of the future — what will happen if I do this? and will that make me happy? — information a reflex agent never represents explicitly. When goal satisfaction takes a single action it is easy; when it needs a long sequence of twists and turns, finding one is the province of search and planning. The goal-based agent looks less efficient than a reflex agent, but it is far more flexible: its knowledge is explicit and modifiable. Tell it a new destination and its whole behavior adapts; the reflex agent would need every rule rewritten.

Utility-based agents

Goals are only a crude binary — happy or unhappy. Most environments offer many action sequences that reach the goal, some quicker, safer, or cheaper than others, and goals alone cannot choose among them. A utility function maps a state (or sequence of states) to a real number capturing how desirable it is; it is essentially an internalization of the performance measure. When the internal utility function agrees with the external performance measure, an agent that chooses actions to maximize utility is rational by the external standard.

A utility function is indispensable in two cases goals cannot handle. When goals conflict — speed against safety — utility specifies the right tradeoff. When several goals are all uncertain, utility weighs the likelihood of success against the importance of each goal. Because partial observability and stochasticity are everywhere, a rational utility-based agent chooses the action that maximizes expected utility: the utility it expects on average, given the probabilities and utilities of each outcome.

Any rational agent can be shown to behave as if it possesses a utility function it is trying to maximize; making that function explicit lets the agent make rational decisions with a general-purpose algorithm. This is the thread the uncertainty chapters pick up. Building such an agent is not simple — it must model and track its environment, tasks that draw on perception, representation, reasoning, and learning — but the principle is clean: maximize expected utility.

Learning agents

None of the four architectures explains how an agent's program comes into being. Turing's answer, now the state of the art, was to build a learning agent and teach it. Learning also lets an agent operate in an initially unknown environment and become more competent than its starting knowledge allows. A learning agent has four conceptual components.

The performance element is what we previously called the whole agent — it takes percepts and chooses actions. The learning element improves it. The critic tells the learning element how well the agent is doing against a fixed performance standard; the standard must be fixed and external, or the agent could satisfy it by lowering its own bar. The problem generator suggests exploratory actions: the performance element left alone would keep doing what it already knows is best, but a suboptimal exploratory action now may reveal a much better one later. This is what a scientist does — Galileo dropped rocks not to break them but to improve his own theory.

The performance element can be any of the four architectures, and learning can improve any of its knowledge components — how the world evolves, what actions do, the utility of states. In this sense the performance standard supplies a reward (or penalty) about the quality of the agent's behavior, the seed of the framework that reinforcement learning develops in full. Learning in intelligent agents is, in one line, the process of modifying each component to bring it into closer agreement with the available feedback, improving overall performance.

Representing the world

Every box in the diagrams above — what the world is like now, goals, utility — holds a description of the world, and how detailed that description is turns out to matter as much as the architecture wrapped around it. That question of how a component represents the world runs across all these designs, along a scale of increasing expressive power. An atomic representation treats each state as an indivisible black box, distinguishable only as same-or-different from another — enough for the route-finding of search and the states of a Markov decision process. A factored representation splits each state into a fixed set of variables or attributes, each with a value, so two states can share some attributes and differ on others — the substrate of constraint satisfaction, propositional logic, and Bayesian networks. A structured representation describes objects and the relationships among them, the expressiveness underlying first-order logic. Each step up the scale can say, at least as concisely, everything the level below can, and more — at the cost of harder reasoning and learning.

The language-model agent

The four architectures were written for agents whose program a human designs component by component: the model, the goals, the utility function are explicit data structures. A different construction became prominent after AIMA's third edition, and it fits the same skeleton without changing it. A large language model trained on text can be placed inside the perceive-act loop directly: its percept is a text context (the user's request plus any results it has seen), and its action is the next text it emits. Wrapped in a loop that feeds its output back as new input, an off-the-shelf model becomes an agent in the exact sense of this lesson.

The four internal components reappear, but as learned or prompted structure rather than hand-coded modules. The ReAct pattern (Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, ICLR 2023) interleaves a reasoning trace with actions: the model writes a thought, takes an action (often a tool call), observes the result, and repeats — a goal-based agent whose what will happen if I do A step is the model's own predicted continuation. Tool use gives it actuators beyond text: Toolformer (Schick et al., NeurIPS 2023) showed a model can learn to call external tools — a calculator, a search engine, a database — and splice the results back into its reasoning, exactly the actuator/sensor extension the PEAS frame anticipates. Memory supplies the internal state a model-based agent needs, since a raw model is stateless past its context window: retrieval over a store of past interactions (the retrieval-augmented pattern of Lewis et al., Retrieval-Augmented Generation, NeurIPS 2020) reintroduces persistence across a long task.

The honest reading keeps the frame in charge, not the technique. A language-model agent is a new and capable way of computing the agent function, but it is subject to the same analysis: it has a performance measure (often implicit and mis-specified, which is the source of much of its unreliability), it operates in a partially observable, non-deterministic, sequential environment, and it is rational only to the extent that its emitted action maximizes expected performance given its context. The concerns this lesson raised — designing the measure around what you want in the environment rather than what you imagine the agent should do, and distinguishing the agent function from the program that realizes it — apply to it unchanged. It is the four architectures seen again, with a learned program in the box.¹

The ladder of architectures

Each architecture answers the same question — what action should I take now? — with more machinery than the last. The reflex agent maps percept straight to action; the model-based agent adds memory; the goal-based agent adds foresight; the utility-based agent adds a scale of preference; and the learning agent improves any of them from experience.