---
title: Agent Architectures
module: Foundations
moduleNumber: 1
lessonNumber: 4
order: 104
summary: >
  How to build a program that computes a good agent function without storing an
  astronomically large lookup table. Four skeleton architectures in order of
  increasing power — simple reflex, model-based, goal-based, and utility-based —
  plus the learning agent that improves any of them, the scale of world
  representations (atomic, factored, structured) they rest on, and how a modern
  language-model agent fits the same frame.
topics: [Foundations]
sources:
  - book: AIMA
    ref: "Ch. 2 — Intelligent Agents; §2.4 The Structure of Agents"
  - book: AIMA
    ref: "§2.4.6 How the Components of Agent Programs Work"
---

This builds on [Intelligent Agents](/artificial-intelligence/foundations/intelligent-agents),
which fixed what an agent _is_ (a percept-to-action loop, captured by an agent
function), what makes one _good_ (rationality — maximizing expected performance),
and how to _specify_ the problem it faces (PEAS and the six environment axes). This
lesson takes up the remaining question: how do you actually _build_ a program that
computes a good agent function? The answer is not one design but a ladder of four,
each adding machinery the one below it lacked.

## The structure of agents

The job of AI is to design an **agent program** that implements the agent
function, running on an **architecture** — a device with physical sensors and
actuators. In short, $\textit{agent} = \textit{architecture} + \textit{program}$.
The agent program takes just the _current_ percept as input (that is all the
sensors offer at any instant) and returns an action; if the agent's behavior must
depend on the whole history, the program has to remember the percepts itself.

Why not implement the agent function as a literal lookup table? Because it is
astronomically too big. Let $\mathcal{P}$ be the set of percepts and $T$ the
agent's lifetime; the table needs $\sum_{t=1}^{T} |\mathcal{P}|^t$ entries. For an
automated taxi with a single camera, that is on the order of $10^{250{,}000{,}000{,}000}$
entries for an hour's driving — no physical agent could store it, no designer
could fill it, no learner could ever populate it. The challenge of AI is to
produce rational behavior from a _small program_ rather than a vast table, just
as a five-line Newton's-method routine replaced the printed tables of square
roots. Four program skeletons underlie almost every intelligent system, in order
of increasing sophistication.

### Simple reflex agents

The simplest agent selects an action from the _current_ percept alone, ignoring
the rest of the history. It works by **condition–action rules** (also called
situation–action rules or if–then rules): _if car-in-front-is-braking then
initiate-braking_. For the vacuum world the entire program is three rules — _if
dirty then suck; else if in A then go right; else go left_ — a dramatic reduction
from the $4^T$ table.

$$
% caption: A simple reflex agent. The current percept is abstracted into a
% description of the world now; a matching condition-action rule selects the
% action. There is no memory of past percepts.
\begin{tikzpicture}[>=stealth, font=\small,
  ag/.style={draw, thick, minimum width=72mm, minimum height=42mm},
  env/.style={draw, minimum width=16mm, minimum height=42mm, align=center, fill=black!5},
  bx/.style={draw, minimum width=26mm, minimum height=10mm, align=center, font=\footnotesize},
  ov/.style={draw, ellipse, minimum width=32mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[ag] (agent) at (0,0) {};
  \node[anchor=north west, font=\footnotesize] at (agent.north west) {Agent};
  \node[env] (env) at (5.6,0) {Environment};
  \node[font=\scriptsize, anchor=south east] at (2.55,2.18) {Sensors};
  \node[font=\scriptsize, anchor=north east] at (2.55,-2.18) {Actuators};
  \node[bx] (world) at (1.1,1.1) {what the world\\is like now};
  \node[ov] (rules) at (-2.15,-1.1) {condition-action rules};
  \node[bx] (act)   at (1.1,-1.1) {what action I\\should do now};
  % percept in
  \draw[->, acc, thick] (env.north west) ++(0,-0.4) -| (world.north);
  \draw[->, acc, thick] (world) -- (act);
  \draw[->, acc, thick] (rules) -- (act);
  \draw[->, acc, thick] (act.south) |- ++(0,-0.5) -| (env.south west);
\end{tikzpicture}
$$

Simple reflex agents are admirably compact but of limited intelligence: they work
_only if the correct action can be decided from the current percept alone — only
if the environment is fully observable_. A little unobservability causes trouble.
A vacuum agent stripped of its location sensor, perceiving only $[\textit{Dirty}]$
or $[\textit{Clean}]$, has no rule that can reliably progress — moving `Left`
loops forever if it starts in $A$. Escaping such loops sometimes requires
**randomization**: perceiving $[\textit{Clean}]$, flip a coin between `Left` and
`Right`. A randomized simple reflex agent can outperform a deterministic one in
partially observable worlds — though in single-agent settings, a better
deterministic agent with memory usually does better still.

### Model-based reflex agents

The way to handle partial observability is to _keep track of the part of the
world the agent can't currently see_ — to maintain **internal state** that depends
on the percept history and reflects the unobserved aspects of the current state.
Updating that state requires two pieces of knowledge encoded in the program: how
the world evolves independently of the agent (an overtaking car will be closer a
moment later), and how the agent's own actions affect the world (turning the
wheel clockwise turns the car right). This knowledge — "how the world works" — is
a **model** of the world, and an agent that uses one is a **model-based agent**.

$$
% caption: A model-based reflex agent. It combines the current percept with the
% previous internal state, using its model of how the world evolves and what its
% actions do, to update its best estimate of the world now, then applies
% condition-action rules as before.
\begin{tikzpicture}[>=stealth, font=\small,
  ag/.style={draw, thick, minimum width=80mm, minimum height=46mm},
  env/.style={draw, minimum width=16mm, minimum height=46mm, align=center, fill=black!5},
  bx/.style={draw, minimum width=24mm, minimum height=10mm, align=center, font=\footnotesize},
  ov/.style={draw, ellipse, minimum width=30mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[ag] (agent) at (0,0) {};
  \node[anchor=south west, font=\footnotesize] at (agent.south west) {Agent};
  \node[env] (env) at (6.0,0) {Environment};
  \node[font=\scriptsize, anchor=south east] at (2.95,2.38) {Sensors};
  \node[font=\scriptsize, anchor=north east] at (2.95,-2.38) {Actuators};
  \node[bx] (world)  at (1.4,1.2) {what the world\\is like now};
  \node[ov] (evolve) at (-2.4,0.7) {how the world evolves};
  \node[ov] (actions) at (-2.4,-0.35) {what my actions do};
  \node[ov] (rules)  at (-2.4,-1.4) {condition-action rules};
  \node[bx] (act)    at (1.4,-1.4) {what action I\\should do now};
  % percept in
  \draw[->, acc, thick] (env.north west) ++(0,-0.4) -| (world.north);
  \draw[->, acc, thick] (evolve.east) -- (world.west);
  \draw[->, acc, thick] (actions.east) -- ++(0.5,0) |- (world.west);
  \draw[->, acc, thick] (world) -- (act);
  \draw[->, acc, thick] (rules.east) -- (act.west);
  \draw[->, acc, thick] (act.south) |- ++(0,-0.5) -| (env.south west);
\end{tikzpicture}
$$

Even a model-based agent seldom knows the current state _exactly_ in a
partially observable world; the box "what the world is like now" holds its best
guess. The taxi may not see around the truck stalled ahead of it and can only
guess at the hold-up — but it must still decide. Internal state need not describe
the world literally, either: "driving home" is really a fact about the taxi's
_destination_, an aspect of the agent's own intended state, not the world's.

### Goal-based agents

Knowing the current state is not always enough to decide what to do. At a junction
the taxi can turn left, turn right, or go straight — and the right choice depends
on _where the taxi is trying to get to_. Beyond a current-state description, the
agent needs **goal** information describing desirable situations, and it combines
the goal with its model to choose actions that (eventually) achieve the goal.

$$
% caption: A goal-based agent. On top of the model it keeps a goal, and it
% predicts what the world will be like if it takes an action A, choosing the
% action that leads toward the goal. This requires reasoning about the future.
\begin{tikzpicture}[>=stealth, font=\small,
  ag/.style={draw, thick, minimum width=82mm, minimum height=50mm},
  env/.style={draw, minimum width=16mm, minimum height=50mm, align=center, fill=black!5},
  bx/.style={draw, minimum width=26mm, minimum height=10mm, align=center, font=\footnotesize},
  ov/.style={draw, ellipse, minimum width=30mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[ag] (agent) at (0,0) {};
  \node[anchor=south west, font=\footnotesize] at (agent.south west) {Agent};
  \node[env] (env) at (6.1,0) {Environment};
  \node[font=\scriptsize, anchor=south east] at (3.05,2.58) {Sensors};
  \node[font=\scriptsize, anchor=north east] at (3.05,-2.58) {Actuators};
  \node[bx] (world)  at (1.5,1.6) {what the world\\is like now};
  \node[bx] (pred)   at (1.5,0.05) {what it will be like\\if I do action A};
  \node[ov] (evolve) at (-2.5,1.35) {how the world evolves};
  \node[ov] (actions) at (-2.5,0.3) {what my actions do};
  \node[ov] (goals)  at (-2.5,-1.4) {goals};
  \node[bx] (act)    at (1.5,-1.6) {what action I\\should do now};
  \draw[->, acc, thick] (env.north west) ++(0,-0.4) -| (world.north);
  \draw[->, acc, thick] (evolve.east) -- (world.west);
  \draw[->, acc, thick] (actions.east) -- (pred.west);
  \draw[->, acc, thick] (world) -- (pred);
  \draw[->, acc, thick] (pred) -- (act);
  \draw[->, acc, thick] (goals.east) -- (act.west);
  \draw[->, acc, thick] (act.south) |- ++(0,-0.5) -| (env.south west);
\end{tikzpicture}
$$

This differs sharply from reflex rules, because it involves
consideration of the future — "what will happen if I do this?" and "will that
make me happy?" — information a reflex agent never represents explicitly. When
goal satisfaction takes a single action it is easy; when it needs a long sequence
of twists and turns, finding one is the province of
[search](/artificial-intelligence/search/uninformed-search) and
[planning](/artificial-intelligence/logic-and-planning/classical-planning). The
goal-based agent looks less efficient than a reflex agent, but it is far more
flexible: its knowledge is explicit and modifiable. Tell it a new destination and
its whole behavior adapts; the reflex agent would need every rule rewritten.

### Utility-based agents

Goals are only a crude binary — "happy" or "unhappy." Most environments offer
many action sequences that reach the goal, some quicker, safer, or cheaper than
others, and goals alone cannot choose among them. A **utility function** maps a
state (or sequence of states) to a real number capturing how desirable it is; it
is essentially an internalization of the performance measure. When the internal
utility function agrees with the external performance measure, an agent that
chooses actions to maximize utility is rational by the external standard.

$$
% caption: A utility-based agent. It predicts the outcome of each action and
% scores how desirable the resulting state is by a utility function, choosing the
% action of highest expected utility — the average utility over possible outcomes,
% weighted by their probability.
\begin{tikzpicture}[>=stealth, font=\small,
  ag/.style={draw, thick, minimum width=82mm, minimum height=56mm},
  env/.style={draw, minimum width=16mm, minimum height=56mm, align=center, fill=black!5},
  bx/.style={draw, minimum width=26mm, minimum height=10mm, align=center, font=\footnotesize},
  ov/.style={draw, ellipse, minimum width=30mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[ag] (agent) at (0,0) {};
  \node[anchor=south west, font=\footnotesize] at (agent.south west) {Agent};
  \node[env] (env) at (6.1,0) {Environment};
  \node[font=\scriptsize, anchor=south east] at (3.05,2.88) {Sensors};
  \node[font=\scriptsize, anchor=east] at (3.95,-2.5) {Actuators};
  \node[bx] (world)  at (1.5,1.95) {what the world\\is like now};
  \node[bx] (pred)   at (1.5,0.55) {what it will be like\\if I do action A};
  \node[bx] (happy)  at (1.5,-0.85) {how happy I will be\\in such a state};
  \node[ov] (evolve) at (-2.5,1.7) {how the world evolves};
  \node[ov] (actions) at (-2.5,0.65) {what my actions do};
  \node[ov] (util)   at (-2.5,-0.9) {utility};
  \node[bx] (act)    at (1.5,-2.1) {what action I\\should do now};
  \draw[->, acc, thick] (env.north west) ++(0,-0.4) -| (world.north);
  \draw[->, acc, thick] (evolve.east) -- (world.west);
  \draw[->, acc, thick] (actions.east) -- (pred.west);
  \draw[->, acc, thick] (world) -- (pred);
  \draw[->, acc, thick] (pred) -- (happy);
  \draw[->, acc, thick] (util.east) -- (happy.west);
  \draw[->, acc, thick] (happy) -- (act);
  \draw[->, acc, thick] (act.south) |- ++(0,-0.5) -| (env.south west);
\end{tikzpicture}
$$

A utility function is indispensable in two cases goals cannot handle. When goals
_conflict_ — speed against safety — utility specifies the right tradeoff. When
several goals are all _uncertain_, utility weighs the likelihood of success
against the importance of each goal. Because partial observability and
stochasticity are everywhere, a rational utility-based agent chooses the action
that maximizes **expected utility**: the utility it expects on average, given the
probabilities and utilities of each outcome.

> **Definition (Expected utility).** Given a probability distribution over the
> possible outcomes of an action, its expected utility is $\mathbb{E}[U] = \sum_{s'} P(s') \, U(s')$,
> the utility of each outcome state weighted by its probability. A rational agent
> in an uncertain environment chooses the action of maximum expected utility.

Any rational agent can be shown to behave _as if_ it possesses a utility function
it is trying to maximize; making that function explicit lets the agent make
rational decisions with a general-purpose algorithm. This is the thread the
[uncertainty](/artificial-intelligence/uncertainty/making-decisions) chapters pick
up. Building such an agent is not simple — it must model and track its
environment, tasks that draw on perception, representation, reasoning, and
learning — but the _principle_ is clean: maximize expected utility.

### Learning agents

None of the four architectures explains how an agent's program _comes into being_.
Turing's answer, now the state of the art, was to build a **learning agent** and
teach it. Learning also lets an agent operate in an initially unknown environment
and become more competent than its starting knowledge allows. A learning agent has
four conceptual components.

$$
% caption: A general learning agent. The performance element (a whole agent of
% one of the earlier kinds) selects actions; the critic judges results against a
% fixed performance standard; the learning element uses that feedback to improve
% the performance element; the problem generator suggests exploratory actions.
\begin{tikzpicture}[>=stealth, font=\small,
  ag/.style={draw, thick, minimum width=84mm, minimum height=56mm},
  env/.style={draw, minimum width=15mm, minimum height=56mm, align=center, fill=black!5},
  bx/.style={draw, minimum width=24mm, minimum height=11mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[ag] (agent) at (0,0) {};
  \node[anchor=south west, font=\footnotesize] at (agent.south west) {Agent};
  \node[env] (env) at (6.2,0) {Environment};
  \node[font=\scriptsize] at (-0.5,2.95) {Performance standard};
  \node[font=\scriptsize, anchor=east] at (3.05,1.6) {Sensors};
  \node[font=\scriptsize, anchor=east] at (3.05,-2.45) {Actuators};
  \node[bx] (critic)  at (-0.5,1.5) {Critic};
  \node[bx] (learn)   at (-2.0,-0.45) {Learning\\element};
  \node[bx] (perf)    at (1.6,-0.45) {Performance\\element};
  \node[bx] (probgen) at (-2.0,-2.0) {Problem\\generator};
  % percept in to critic
  \draw[->, acc, thick] (env.north west) ++(0,-0.5) -- (critic.east);
  % performance standard in to critic
  \draw[->, acc, thick] (-0.5,2.7) -- (critic.north);
  \draw[->, acc, thick] (critic) -- node[right, font=\scriptsize, text=black] {feedback} (learn);
  \draw[->, acc, thick] (learn) -- node[above, font=\scriptsize, text=black] {changes} (perf);
  \draw[->, acc, thick] (perf) -- node[below, font=\scriptsize, text=black] {knowledge} (learn);
  \draw[->, acc, thick] (learn) -- node[left, font=\scriptsize, text=black] {goals} (probgen);
  \draw[->, acc, thick] (probgen.east) -- (perf.south west);
  \draw[->, acc, thick] (perf.south) |- ++(0,-0.6) -| (env.south west);
\end{tikzpicture}
$$

The **performance element** is what we previously called the whole agent — it
takes percepts and chooses actions. The **learning element** improves it. The
**critic** tells the learning element how well the agent is doing against a
_fixed_ performance standard; the standard must be fixed and external, or the
agent could satisfy it by lowering its own bar. The **problem generator** suggests
exploratory actions: the performance element left alone would keep doing what it
already knows is best, but a suboptimal exploratory action now may reveal a much
better one later. This is what a scientist does — Galileo dropped rocks not to
break them but to improve his own theory.

The performance element can be any of the four architectures, and learning can
improve any of its knowledge components — how the world evolves, what actions do,
the utility of states. In this sense the performance standard supplies a **reward**
(or penalty) about the quality of the agent's behavior, the seed of the framework
that [reinforcement learning](/artificial-intelligence/learning/reinforcement-learning)
develops in full. Learning in intelligent agents is, in one line, the process of
modifying each component to bring it into closer agreement with the available
feedback, improving overall performance.

## Representing the world

Every box in the diagrams above — "what the world is like now," "goals," "utility" —
holds a description of the world, and how detailed that description is turns out to
matter as much as the architecture wrapped around it. That question of _how_ a
component represents the world runs across all these designs, along a scale of
increasing expressive power. An **atomic**
representation treats each state as an indivisible black box, distinguishable only
as same-or-different from another — enough for the route-finding of
[search](/artificial-intelligence/search/uninformed-search) and the states of a
Markov decision process. A **factored** representation splits each state into a
fixed set of variables or attributes, each with a value, so two states can share
some attributes and differ on others — the substrate of
[constraint satisfaction](/artificial-intelligence/search/constraint-satisfaction),
[propositional logic](/artificial-intelligence/logic-and-planning/propositional-logic),
and [Bayesian networks](/artificial-intelligence/uncertainty/bayesian-networks). A
**structured** representation describes objects and the relationships among them,
the expressiveness underlying
[first-order logic](/artificial-intelligence/logic-and-planning/first-order-logic).
Each step up the scale can say, at least as concisely, everything the level below
can, and more — at the cost of harder reasoning and learning.

$$
% caption: Three ways to represent a state, in order of increasing expressive
% power: atomic (an indivisible label), factored (a vector of attribute values),
% and structured (objects with attributes and relations among them).
\begin{tikzpicture}[>=stealth, font=\small,
  cell/.style={draw, minimum width=8mm, minimum height=8mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  % atomic
  \node[cell] (b) at (0,0) {B};
  \node[cell] (c) at (1.4,0) {C};
  \draw[->, acc, thick] (b) -- (c);
  \node[font=\scriptsize, anchor=north] at (0.7,-0.9) {atomic};
  % factored
  \begin{scope}[xshift=42mm]
    \draw (0,-0.55) rectangle (1.5,0.95);
    \node[font=\scriptsize] at (0.35,0.65) {oil};
    \node[font=\scriptsize] at (0.4,0.25) {gas};
    \node[font=\scriptsize] at (0.5,-0.15) {city};
    \node[draw, minimum width=6mm, minimum height=3mm, font=\tiny, anchor=west] at (0.75,-0.15) {B};
    \node[font=\scriptsize, anchor=north] at (0.75,-0.9) {factored};
  \end{scope}
  % structured
  \begin{scope}[xshift=82mm]
    \node[cell, minimum width=6mm, minimum height=6mm] (o1) at (0,0.35) {};
    \node[cell, minimum width=6mm, minimum height=6mm] (o2) at (1.5,0.35) {};
    \node[cell, minimum width=6mm, minimum height=6mm] (o3) at (0.75,-0.5) {};
    \draw[->, acc] (o1) -- (o2);
    \draw[->, acc] (o1) -- (o3);
    \draw[->, acc] (o3) -- (o2);
    \node[font=\scriptsize, anchor=north] at (0.75,-1.25) {structured};
  \end{scope}
\end{tikzpicture}
$$

## The language-model agent

The four architectures were written for agents whose program a human designs
component by component: the model, the goals, the utility function are explicit
data structures. A different construction became prominent after _AIMA_'s third
edition, and it fits the same skeleton without changing it. A large language model
trained on text can be placed inside the perceive-act loop directly: its **percept**
is a text context (the user's request plus any results it has seen), and its
**action** is the next text it emits. Wrapped in a loop that feeds its output back
as new input, an off-the-shelf model becomes an agent in the exact sense of this
lesson.

The four internal components reappear, but as
_learned or prompted_ structure rather than hand-coded modules. The **ReAct**
pattern (Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models,"
ICLR 2023) interleaves a reasoning trace with actions: the model writes a thought,
takes an action (often a tool call), observes the result, and repeats — a
goal-based agent whose "what will happen if I do A" step is the model's own
predicted continuation. **Tool use** gives it actuators beyond text: Toolformer
(Schick et al., NeurIPS 2023) showed a model can learn to call external tools —
a calculator, a search engine, a database — and splice the results back into its
reasoning, exactly the actuator/sensor extension the PEAS frame anticipates.
**Memory** supplies the internal state a model-based agent needs, since a raw model
is stateless past its context window: retrieval over a store of past interactions
(the retrieval-augmented pattern of Lewis et al., "Retrieval-Augmented Generation,"
NeurIPS 2020) reintroduces persistence across a long task.

$$
% caption: A language-model agent in the ReAct loop, mapped onto the goal-based
% skeleton. The model reasons (predicts an outcome), acts (emits text or a tool
% call), and observes the result, which becomes the next percept. Tools are
% actuators and sensors; a memory store supplies internal state.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, minimum width=26mm, minimum height=10mm, align=center},
  bxa/.style={draw=acc, text=acc, thick, minimum width=26mm, minimum height=10mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[bxa] (think) at (0,1.4)   {reason (thought)};
  \node[bxa] (act)   at (0,-0.4)  {act (text / tool call)};
  \node[bx]  (obs)   at (5.6,-0.4) {observe result};
  \node[bx]  (mem)   at (5.6,1.4)  {memory store};
  \draw[->, acc, thick] (think) -- (act);
  \draw[->, thick] (act) -- (obs);
  \draw[->, thick] (obs) -- ++(0,1.0) -| (think.east);
  \node[font=\scriptsize, anchor=south] at (2.8,-0.35) {tool = actuator};
  \draw[->, black] (mem) -- (think);
  \node[font=\scriptsize, text=black, anchor=west] at (6.9,1.4) {internal state};
\end{tikzpicture}
$$

The honest reading keeps the frame in charge, not the technique. A language-model
agent is a new and capable way of _computing the agent function_, but it is subject
to the same analysis: it has a performance measure (often implicit and
mis-specified, which is the source of much of its unreliability), it operates in a
partially observable, non-deterministic, sequential environment, and it is rational
only to the extent that its emitted action maximizes expected performance given its
context. The concerns this lesson raised — designing the measure around what you
want in the environment rather than what you imagine the agent should do, and
distinguishing the agent function from the program that realizes it — apply to it
unchanged. It is the four architectures seen again, with a learned program in the
box.[^llm-agents]

## The ladder of architectures

Each architecture answers the same question — what action should I take now? — with
more machinery than the last. The reflex agent maps percept straight to action; the
model-based agent adds memory; the goal-based agent adds foresight; the
utility-based agent adds a scale of preference; and the learning agent improves any
of them from experience.

$$
% caption: The four architectures as a ladder of added machinery, each keeping
% everything below it. Reflex maps percept to action; model-based adds memory of
% the unseen world; goal-based adds prediction toward a goal; utility-based adds a
% preference scale; and a learning agent can wrap any of them.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  rung/.style={draw, minimum width=44mm, minimum height=10mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[rung] (r1) at (0,0)    {simple re\/f\/lex\\percept to action};
  \node[rung] (r2) at (0,1.3)  {model-based\\add internal state};
  \node[rung] (r3) at (0,2.6)  {goal-based\\add foresight to a goal};
  \node[rung, draw=acc, text=acc, thick] (r4) at (0,3.9) {utility-based\\add a preference scale};
  \draw[->, black] (r1) -- (r2);
  \draw[->, black] (r2) -- (r3);
  \draw[->, black] (r3) -- (r4);
  \node[rung, minimum height=52mm, minimum width=6mm] (learn) at (3.6,1.95) {};
  \node[text=acc, rotate=90, font=\scriptsize] at (3.6,1.95) {learning wraps any};
\end{tikzpicture}
$$

[^llm-agents]: **S. Yao et al.**, "ReAct: Synergizing Reasoning and Acting in Language Models," ICLR 2023 (interleaved reasoning and action); **T. Schick et al.**, "Toolformer: Language Models Can Teach Themselves to Use Tools," NeurIPS 2023 (learned tool use as actuators); and **P. Lewis et al.**, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020 (retrieval as external memory / internal state). These place a language model inside the perceive-act loop of §2.1 without altering the agent framework.