Intelligent Agents

An agent is anything that perceives its environment through sensors and acts on that environment through actuators. A human agent has eyes and ears for sensors, hands and legs for actuators; a robot has cameras and range finders for sensors and motors for actuators; a software agent receives keystrokes and network packets and acts by writing files and sending packets. The definition is deliberately thin, because it is meant to be a lens for analyzing systems, not a line that divides the world into agents and non-agents. A pocket calculator, viewed this way, is an agent that outputs 4 on the percept 2 + 2 = — an unhelpful reading, but not a wrong one. AI reserves its attention for the interesting end of that spectrum: agents with real computational resources whose task environment demands nontrivial decisions.¹

This lesson works through three ideas in order. What an agent is (the percept-to-action loop). What it means for an agent to be good (rationality). And how to specify the problem an agent faces (PEAS and the properties of task environments). The companion lesson, agent architectures, then shows how to build one. Every later chapter — search, logic, probability, learning — is a way of computing the action a rational agent should take in some class of environment.

Agents and the agent function

At any instant an agent has a percept: whatever its sensors report. Its percept sequence is the complete history of everything it has perceived. The central premise: an agent's action at any instant can depend on the entire percept sequence to date, but on nothing it has not perceived. So an agent's behavior is captured in full by an agent function mapping every percept sequence to an action.²

Equivalently: given the full sensing history, the next action is determined. This map is the agent function

where $\mathcal{P}$ is the set of percepts, ${\mathcal{P}}^{\ast }$ the set of finite percept sequences, and $\mathcal{A}$ the set of actions. A percept sequence is $p_{1:t}=⟨p_1,\dots ,p_t⟩$, and the action at step $t$ is $a_t=f(p_{1:t})$. The function $f$ is an external, mathematical description — a (usually infinite) table pairing each percept history with a response. Internally, $f$ is realized by an agent program running on a physical device. Keep the two distinct: $f$ is the abstract specification, the program is the concrete implementation. The distinction returns when we ask how much memory a program needs to reproduce a given $f$.

Take the vacuum-cleaner world. Two squares $A$ and $B$, each either clean or dirty; the agent perceives its square and whether it is dirty, and acts in $\mathcal{A}=\{\text{Left},\text{Right},\text{Suck},\text{NoOp}\}$. A simple agent function is: if the current square is dirty, suck; otherwise move to the other square. The design question of the whole field, in miniature, is: what is the right way to fill out that table? That is what rationality answers.

Writing out the tabulation shows $f$ is a finite object here even though it is infinite in general. Because this function depends only on the current percept, $f(p_{1:t})=f(p_t)$, so only the four one-step percepts matter and $f$ is four rows.

Rationality

A rational agent does the right thing — it fills out every entry of the agent function correctly. But right cannot mean whatever the agent believes is right, or an agent could be rational by simply deluding itself. Rightness is judged from outside, by a performance measure that scores sequences of environment states (not agent states). For the vacuum agent, a good measure might award one point per clean square at each time step over a lifetime of 1000 steps, perhaps with a penalty for wasted motion. As a design rule, it is better to specify performance in terms of what one actually wants in the environment than in terms of how one imagines the agent should behave — otherwise the agent will game the letter of the measure. (Reward amount of dirt cleaned, and a rational agent will dump dirt back on the floor so it can clean it again.)³

What is rational at a given moment depends on four things: the performance measure, the agent's prior knowledge of the environment, the actions available to it, and its percept sequence to date. Assembling them gives the definition the rest of the course rests on.

Written as a choice rule: with performance measure $U$ over environment-state sequences, prior knowledge $K$, and percepts $p_{1:t}$, the rational action is

the action whose expected performance is greatest given everything known so far.

Two words in that definition do heavy lifting. Expected: rationality maximizes expected performance $\mathbb{E}[U]$, not actual performance $U$. This is the difference between rationality and omniscience. An omniscient agent knows the actual outcome of its actions; no real agent does. Crossing an empty street is rational even if, freakishly, a cargo door falls from a passing airliner and flattens you — the percept sequence gave no reason to expect it, so the choice was correct even though the outcome was catastrophic. Rationality is about making the best decision given what you could know, not about being right after the fact.

To see what maximize the performance measure means, score the simple vacuum agent above. Fix $U={\sum }_{t=0}^{T-1}(\text{clean squares at }t)$ with $T=1000$, the geography and initial dirt known, the start square random. Trace the worst start ($A$ dirty, $B$ dirty, agent in $A$): $t=0$ suck ($A$ clean), $t=1$ move right, $t=2$ suck ($B$ clean), both clean thereafter. Score: $1$ at $t=0$, $1$ at $t=1$, $2$ for each remaining step — $1+1+2\times 998=1998$ of a possible $2000$. Against this measure, in this fully observable environment, the agent is rational: no $f$ does better, since two squares cannot both be clean before the agent has visited each, so the two lost points are unavoidable. Change the measure — penalize each move by one point to model energy — and the same agent is no longer rational; the rational agent now halts once both squares are clean rather than oscillate. Rationality is relative to the four inputs; changing any one can change which $f$ is rational.

Given the percept sequence: rationality is relative to the information available. If an agent does not look before crossing a busy road, its percept sequence will not warn it of the approaching truck — but that does not license the crossing. A rational agent should first take the looking action, because looking maximizes expected performance. Acting to improve future percepts is called information gathering, and it is a genuine part of rational behavior. A rational agent should also learn: use its percepts to correct and extend whatever prior knowledge it started with. An agent that relies wholly on its designer's prior knowledge, rather than its own percepts, lacks autonomy; a truly rational agent is autonomous, learning enough to compensate for partial or incorrect priors.⁴

Specifying the task: PEAS

Before designing an agent, specify the problem it must solve. Rationality is defined relative to a performance measure, an environment, and the agent's sensors and actuators — group these four under the heading task environment, and remember them by the acronym PEAS: Performance, Environment, Actuators, Sensors. The first step in building any agent is to specify the PEAS as fully as possible.⁵

Take an automated taxi driver — a task still somewhat beyond current technology, and open-ended enough to exercise every part of the framework.

The performance measure already exposes real design tension: safety, speed, and profit conflict, so the agent needs a way to trade them off — a hint that goals alone will not be enough. The same template describes agents in purely software environments, where sensors are keyboard input and actuators are screen output; what matters is not whether the environment is physical but how complex the coupling between behavior, percepts, and performance is.

Work one all the way through to see how the PEAS specification drives the design. Take a spam filter, an agent almost everyone runs. Its PEAS is compact but every entry carries a design consequence.

Two design facts fall straight out of the P and the E. The asymmetric penalty in Performance tells the agent to bias toward the inbox when unsure — a false spam is worse than a false inbox — which is a threshold choice, not a new mechanism. And the Environment contains an adversary: spammers change their messages precisely to defeat the filter, so the sender population is non-stationary and the agent must keep learning. That single observation is why spam filtering is a machine-learning problem and not a fixed rulebook, and it is visible in the PEAS before any algorithm is chosen. Specifying PEAS well is most of the analysis.

Properties of task environments

Task environments vary along a handful of dimensions, and those dimensions — more than anything else — determine which agent design and which family of techniques is appropriate. The definitions here are informal; later chapters sharpen each one.⁶

• Fully vs. partially observable. Write the current percept as $p_t$ and the environment state as $s_t$. The environment is fully observable if $p_t$ determines the action-relevant part of $s_t$ at each instant; then the agent needs no internal state. It is partially observable if sensors are noisy, inaccurate, or miss part of $s_t$ — a vacuum agent with only a local dirt sensor cannot tell whether the other square is dirty. An environment with no sensors is unobservable.
• Single-agent vs. multi-agent. Solving a crossword is single-agent; chess is two-agent. What makes an entity $B$ count as an agent (rather than as stochastic scenery) is whether $B$'s behavior is best described as maximizing a performance measure that depends on agent $A$'s behavior. Chess is competitive (the opponent's gain is your loss); taxi driving is partly cooperative (everyone wants to avoid collisions) and partly competitive (only one car fits the parking space).
• Deterministic vs. stochastic. Deterministic if the next state is a function of the current state and action, $s_{t+1}=T(s_t,a_t)$; stochastic if instead governed by a distribution $P(s_{t+1}\mid s_t,a_t)$. Taxi driving is stochastic — traffic is never exactly predictable, tires blow out. An environment that is not fully observable or not deterministic is uncertain. Nondeterministic is subtly different: outcomes are listed as possible values with no probabilities attached.
• Episodic vs. sequential. In an episodic environment, experience divides into atomic episodes — one percept, one action — and the next episode does not depend on actions taken in previous ones. An assembly-line defect spotter is episodic. In a sequential environment, a current decision can affect all future ones; chess and taxi driving are sequential, and demand looking ahead.
• Static vs. dynamic. If the environment can change while the agent deliberates, it is dynamic; otherwise static. Taxi driving is dynamic — the world does not wait while the agent decides. Crossword puzzles are static. If the environment itself holds still but the agent's performance score keeps ticking (chess with a clock), the environment is semidynamic.
• Discrete vs. continuous. This applies to the state, to time, and to the agent's percepts and actions. Chess has a finite set of states and a discrete set of moves; taxi driving sweeps continuously through speeds, positions, and steering angles.

A seventh axis, known vs. unknown, refers not to the environment itself but to the agent's knowledge of its laws of physics — whether it is given the outcomes (or outcome probabilities) of its actions. This is not the same as observability: solitaire is known but partially observable (you know the rules, but not the face-down cards), while a new video game can be fully observable yet unknown (the screen shows everything, but you don't yet know what the buttons do).

The hardest case is partially observable, multi-agent, stochastic, sequential, dynamic, continuous, and unknown — which is roughly the real world, and roughly what taxi driving is. Almost every technique in the course is a way of coping with one or more of these axes.