---
title: "Making Decisions: Utility Theory"
module: Uncertainty
moduleNumber: 4
lessonNumber: 7
order: 407
summary: >
  A rational agent chooses the action that maximizes expected utility — the
  probability of each outcome weighted by how much the agent wants it. We derive
  the utility function from six axioms on preferences, so maximizing expected utility
  is forced by consistency rather than assumed; look at risk aversion in the
  utility-of-money curve; package one-shot choices into decision networks; and quantify
  what an observation is worth with the value of information.
topics: [Uncertainty]
sources:
  - book: AIMA
    ref: "Ch. 16 — Making Simple Decisions; §16.1 Combining Beliefs and Desires; §16.2 The Basis of Utility Theory"
  - book: AIMA
    ref: "§16.3 Utility Functions; §16.5 Decision Networks; §16.6 The Value of Information"
---

A logic agent divides the world in two: goal states and everything else. That is
enough to decide what to do only when the agent knows exactly what each action
will bring about and cares about nothing but reaching the goal. Neither holds in
an uncertain world. Actions have stochastic outcomes, and outcomes differ not
just in whether they are the goal but in _how good_ they are. **Decision
theory** addresses both: it replaces the binary goal test with a real-valued
**utility**, replaces certainty about outcomes with a **probability**
distribution over them, and picks the action whose outcomes are best _on
average_.[^aima-16intro]

Decision theory = probability theory + utility theory. Probability says what the
world will do; utility says how much the agent likes each thing the world might
do; and the two combine into a single rule for acting.

## Expected utility and the MEU principle

The core idea is a weighted average: list an action's possible outcomes, score
each one, and weight the scores by probability. The action with the best
weighted-average score is the one to take. This section makes that precise.

An agent about to act does not know the current state exactly (it has only
evidence $\mathbf{e}$) and does not know which state an action will produce. Write
$\text{Result}(a)$ for the random outcome state of action $a$, so
$P(\text{Result}(a) = s' \mid a, \mathbf{e})$ is the probability that doing $a$
lands the agent in $s'$. A **utility function** $U(s)$ assigns each state a single
number measuring how desirable it is. The **expected utility** of an action is the
average utility of its outcomes, weighted by their probability:[^aima-meu]

$$
EU(a \mid \mathbf{e}) \;=\; \sum_{s'} P\big(\text{Result}(a) = s' \mid a, \mathbf{e}\big)\, U(s').
$$

> **Definition (Maximum Expected Utility).** A rational agent should choose the
> action that maximizes its expected utility,
> $$a^\ast \;=\; \argmax_a\; EU(a \mid \mathbf{e}).$$
> This is the **MEU principle**: do the thing that is best on average, given what
> you believe and what you want.

MEU comes close to defining the whole subject: an intelligent agent has only to
estimate its state, compute these two quantities, maximize, and act. In practice
each step is hard. Estimating the state needs
perception, learning, and inference; computing $P(\text{Result}(a) \mid a,
\mathbf{e})$ needs a full causal model of the world and, in general, NP-hard
[Bayesian-network inference](/artificial-intelligence/uncertainty/bayesian-networks);
computing $U(s')$ may itself require search or planning, because the agent often
cannot tell how good a state is until it knows what it can reach from there. MEU
formalizes "do the right thing" without operationalizing it — that is the rest of
AI.[^aima-meu]

$$
% caption: A decision under uncertainty. Action $a$ induces a distribution over
% outcome states; each outcome carries a utility; the expected utility is the
% probability-weighted average, and MEU picks the action with the largest one.
\begin{tikzpicture}[>=stealth, font=\small,
  act/.style={draw, minimum width=15mm, minimum height=8mm},
  st/.style={draw, circle, minimum size=7mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \node[act, draw=acc, text=acc] (a) at (0,0) {action a};
  \node[st] (s1) at (3.4,1.6) {s1};
  \node[st] (s2) at (3.4,0)   {s2};
  \node[st] (s3) at (3.4,-1.6){s3};
  \draw[->, acc] (a) -- (s1) node[midway, above, font=\scriptsize] {0.7};
  \draw[->, acc] (a) -- (s2) node[midway, above, font=\scriptsize] {0.2};
  \draw[->, acc] (a) -- (s3) node[midway, below, font=\scriptsize] {0.1};
  \node[anchor=west, font=\scriptsize] at (4.1,1.6) {U = 10};
  \node[anchor=west, font=\scriptsize] at (4.1,0)   {U = 4};
  \node[anchor=west, font=\scriptsize] at (4.1,-1.6){U = -6};
  \node[anchor=west, align=left, font=\footnotesize] at (7.0,0)
    {EU(a) = 0.7(10)\\ \hphantom{EU(a) =} + 0.2(4)\\ \hphantom{EU(a) =} + 0.1(-6)\\ \hphantom{EU(a)} = 7.2};
\end{tikzpicture}
$$

### A worked decision under uncertainty

Consider a startup that must decide whether to `Launch` a product now, run a `Test` market first (which
costs 2 utility units but reveals demand), or `Abandon` the idea. Demand is either
_high_ (prior $0.6$) or _low_ (prior $0.4$). Launching into high demand is worth
$+20$, launching into low demand $-10$; abandoning is worth $0$ regardless. The
test is imperfect: it reports `up` or `down`, with $P(\text{up} \mid \text{high}) =
0.8$ and $P(\text{up} \mid \text{low}) = 0.3$.

Take the two simple actions first. Launching blind has expected utility
$EU(\text{Launch}) = 0.6(20) + 0.4(-10) = 12 - 4 = 8$, and abandoning has
$EU(\text{Abandon}) = 0$, so absent any test the MEU action is `Launch`, worth $8$.

Now value the test. First fold the priors and the sensor model through Bayes' rule
to get the posterior after each report. The marginal probability of `up` is
$P(\text{up}) = 0.6(0.8) + 0.4(0.3) = 0.48 + 0.12 = 0.60$, so `down` has
probability $0.40$. The posteriors are

$$
P(\text{high} \mid \text{up}) = \frac{0.48}{0.60} = 0.80, \qquad
P(\text{high} \mid \text{down}) = \frac{0.6(0.2)}{0.40} = \frac{0.12}{0.40} = 0.30.
$$

After `up`, launching is worth $0.8(20) + 0.2(-10) = 16 - 2 = 14 > 0$, so launch.
After `down`, launching is worth $0.3(20) + 0.7(-10) = 6 - 7 = -1 < 0$, so abandon
(worth $0$). The expected utility of the `Test` branch, _before_ paying its cost,
averages these best responses over the two reports:

$$
0.60 \cdot \underbrace{14}_{\text{launch}} + 0.40 \cdot \underbrace{0}_{\text{abandon}}
  = 8.4.
$$

Subtracting the test's cost of $2$ leaves $6.4$, which is _less_ than launching
blind at $8$. So the MEU decision is to skip the test and launch. The test would be
worth running only if its cost fell below $8.4 - 8 = 0.4$ — the point where the plan
it enables beats the plan you would follow without it. That gap measures the value
of information, quantified later in this lesson.

$$
% caption: A decision tree for the launch problem. Squares are decision nodes,
% circles are chance nodes; leaves hold utilities. Folding back from the leaves --
% expectation at chance nodes, max at decision nodes -- gives EU(Launch) = 8, and
% the Test branch nets 6.4 after its cost, so the root chooses Launch.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  dec/.style={draw, fill=black!5, minimum size=4mm, inner sep=1pt},
  ch/.style={draw, circle, minimum size=4mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[dec, draw=acc] (root) at (0,0) {};
  % Launch branch (blind)
  \node[ch] (L) at (2.6,2.4) {};
  \draw[->, acc] (root) -- (L) node[midway, above, sloped, font=\scriptsize] {go now};
  \node[anchor=west, font=\scriptsize] at (3.3,3.0) {high 0.6: +20};
  \node[anchor=west, font=\scriptsize] at (3.3,2.4) {low 0.4: -10};
  \draw[->, black] (L) -- (3.2,3.0);
  \draw[->, black] (L) -- (3.2,2.4);
  \node[anchor=west, font=\scriptsize, acc] at (6.6,2.7) {EU = 8};
  % Abandon branch
  \draw[->, acc] (root) -- (2.6,0) node[midway, above, font=\scriptsize] {Abandon};
  \node[anchor=west, font=\scriptsize] at (2.7,0) {0};
  % Test branch
  \node[ch] (T) at (2.6,-2.4) {};
  \draw[->, acc] (root) -- (T) node[midway, below, sloped, font=\scriptsize] {Test};
  \node[dec, draw=acc] (Tu) at (4.6,-1.5) {};
  \node[dec, draw=acc] (Td) at (4.6,-3.3) {};
  \draw[->, black] (T) -- (Tu) node[midway, above, font=\scriptsize] {up .6};
  \draw[->, black] (T) -- (Td) node[midway, below=1.5pt, sloped, font=\scriptsize] {down .4};
  \node[anchor=west, font=\scriptsize] at (5.1,-1.5) {go: +14};
  \node[anchor=west, font=\scriptsize] at (5.1,-3.3) {abandon: 0};
  \node[anchor=west, font=\scriptsize, acc] at (7.8,-2.4) {6.4 net};
\end{tikzpicture}
$$

## Why a utility function exists

MEU looks arbitrary — why _average_ utility, rather than the worst case, or the
sum of cubes? The justification runs the other way. Start not from a utility
function but from an agent's **preferences** over outcomes, and impose only that
those preferences be _consistent_. Consistency alone forces the existence of a
utility function that the agent behaves as if it were maximizing.

Because actions have uncertain outcomes, the objects of preference are not states
but **lotteries**: a lottery $L = [\,p_1, S_1;\; \dots;\; p_n, S_n\,]$ yields
outcome $S_i$ with probability $p_i$. Write $A \succ B$ for "$A$ is preferred to
$B$", $A \sim B$ for indifference. A rational agent's preferences must obey six
axioms:[^aima-axioms]

> **Definition (Axioms of utility theory).** For all lotteries $A, B, C$:
> **Orderability** — exactly one of $A \succ B$, $B \succ A$, $A \sim B$ holds.
> **Transitivity** — $A \succ B$ and $B \succ C$ imply $A \succ C$.
> **Continuity** — if $A \succ B \succ C$, some probability $p$ makes
> $[\,p, A;\; 1{-}p, C\,] \sim B$. **Substitutability** — indifferent lotteries
> can be swapped inside any larger lottery. **Monotonicity** — if $A \succ B$, the
> lottery giving $A$ the higher probability is preferred. **Decomposability** —
> a compound lottery equals the simple lottery with the same net outcome
> probabilities.

Each axiom rules out a way of being pumped. Violate transitivity, holding
$A \succ B \succ C \succ A$, and someone can walk you around the cycle — trade you
$C$ for $A$ for a cent, $B$ for $C$ for a cent, $A$ for $B$ for a cent — draining
your money one loop at a time while you end where you started.

$$
% caption: The money pump. Nontransitive preferences $A \succ B \succ C \succ A$
% let an adversary charge a cent at each swap and cycle forever; the agent pays to
% return exactly where it began, so nontransitivity is irrational.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \node[circle, draw, minimum size=8mm] (A) at (90:1.7) {A};
  \node[circle, draw, minimum size=8mm] (B) at (210:1.7) {B};
  \node[circle, draw, minimum size=8mm] (C) at (330:1.7) {C};
  \draw[->, acc, thick] (A) to[bend right=22] node[left, font=\scriptsize] {pay 1c} (B);
  \draw[->, acc, thick] (B) to[bend right=22] node[below, font=\scriptsize] {pay 1c} (C);
  \draw[->, acc, thick] (C) to[bend right=22] node[right, font=\scriptsize] {pay 1c} (A);
\end{tikzpicture}
$$

The result is a representation theorem of von Neumann and Morgenstern: an agent
whose preferences obey the axioms behaves _exactly_ as if it had a utility
function $U$ and were maximizing its expectation.[^aima-existence]

> **Theorem (Existence of utility).** If an agent's preferences satisfy the six
> axioms, there exists a real-valued function $U$ with
> $$U(A) > U(B) \iff A \succ B, \qquad U(A) = U(B) \iff A \sim B,$$
> and the utility of a lottery is the expectation of its outcome utilities,
> $$U([\,p_1, S_1; \dots; p_n, S_n\,]) = \sum_i p_i\, U(S_i).$$

So MEU follows from the axioms rather than being an extra assumption. The second line is why the average, and not some other functional of
the outcomes, is the right thing to maximize.

The theorem determines $U$ only up to an **affine transformation**: replacing
$U(s)$ by $U'(s) = aU(s) + b$ with $a > 0$ leaves every preference, and hence
every decision, unchanged.[^aima-affine] Utilities are like temperatures — the
Fahrenheit and Celsius scales rank warmth identically. A purely deterministic
agent needs even less: only an ordinal ranking of states, called a **value
function**, since with no lotteries the numeric gaps never matter.

## Utility of money and risk aversion

Money is the obvious candidate for a utility scale, and agents usually show a
**monotonic preference** for more of it. But utility is _not_ linear in money.
Offered a certain \$1{,}000{,}000 or a coin flip between \$0 and \$2{,}500{,}000,
most people take the sure million — even though the gamble's **expected monetary
value**, $\tfrac{1}{2}(0) + \tfrac{1}{2}(2{,}500{,}000) = 1{,}250{,}000$, is
larger.[^aima-money] That is not irrational; it reflects a utility curve that is
_concave_ in wealth. An additional million is worth less the wealthier the agent
already is. Bernoulli's suggestion, borne out
empirically by Grayson, is that utility is roughly _logarithmic_ in total assets.

$$
% caption: A concave utility-of-money curve. The certainty equivalent CE — the
% sure amount the agent will accept in place of a gamble — sits below the gamble's
% expected monetary value EMV; the gap EMV minus CE is the insurance premium, and
% a concave curve is the signature of risk aversion.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (-0.3,0) -- (7.2,0) node[right, font=\footnotesize] {wealth};
  \draw[->, black] (0,-0.3) -- (0,4.4) node[above, font=\footnotesize] {U};
  % concave (log-like) utility curve
  \draw[acc, very thick] (0.15,0.2) .. controls (2.2,2.7) and (4.5,3.5) .. (6.8,3.95)
    node[anchor=west, font=\footnotesize, text=acc] {};
  \node[acc, anchor=south west, font=\footnotesize] at (5.4,3.75) {U(wealth)};
  % two gamble endpoints on the curve and their chord
  \coordinate (lo) at (1.0,1.55);
  \coordinate (hi) at (5.5,3.72);
  \fill[black] (lo) circle (1.6pt);
  \fill[black] (hi) circle (1.6pt);
  \draw[black, dashed] (lo) -- (hi);
  % midpoint of chord = expected utility of the 50/50 gamble
  \coordinate (mid) at (3.25,2.635);
  \fill[red] (mid) circle (1.8pt);
  \node[red, anchor=west, font=\scriptsize] at (3.38,2.60) {E[U]};
  % point on curve directly above the same wealth (EMV) 
  \coordinate (onEMV) at (3.25,3.30);
  \fill[acc] (onEMV) circle (1.8pt);
  % drop lines
  \draw[black, dotted] (3.25,0) -- (onEMV);
  \node[anchor=north, font=\scriptsize] at (3.25,-0.05) {EMV};
  % certainty equivalent: wealth whose utility equals E[U]
  \coordinate (ce) at (2.15,2.635);
  \fill[acc] (ce) circle (1.8pt);
  \draw[red!70, dotted] (ce) -- (2.15,0);
  \node[anchor=north, font=\scriptsize] at (2.15,-0.05) {CE};
  \draw[black, dotted] (mid) -- (ce);
  % premium span label
  \draw[black, |-|] (2.15,-0.5) -- (3.25,-0.5);
  \node[anchor=north, font=\scriptsize] at (2.7,-0.55) {premium};
\end{tikzpicture}
$$

For any lottery $L$ over the concave region, the utility of the gamble is less
than the utility of its expected value paid for sure:
$U(L) < U(S_{EMV(L)})$. An agent with this curve is **risk-averse**. The sure
amount it will accept in lieu of the gamble is the **certainty equivalent**; the
gap between the gamble's EMV and its certainty equivalent is the **insurance
premium**, and that is why the insurance industry works — people pay a
little more than the expected loss to shed the variance.[^aima-riskaverse] Over
large negative wealth the curve turns convex, producing **risk-seeking** behavior
(a desperate agent gambles); and for small stakes relative to current wealth any
smooth curve is locally linear, so the agent is approximately **risk-neutral** and
maximizes EMV directly.

## Decision networks

For a one-shot decision with several uncertain variables, the machinery is
packaged into a **decision network** (or **influence diagram**): a
[Bayesian network](/artificial-intelligence/uncertainty/bayesian-networks)
extended with two new node types. **Chance nodes** (ovals) are random variables
as before; **decision nodes** (rectangles) are the actions the agent controls; and
a single **utility node** (diamond) holds $U$ as a function of its parents.
Evaluating it is the MEU rule made mechanical: for each setting of the decision
node, set it as evidence, run standard inference to get the posterior over the
utility node's parents, and read off the expected utility — then return the
action with the highest one.[^aima-decnet]

$$
% caption: A decision network for siting an airport. The rectangle is the
% decision (which site); ovals are chance nodes for the uncertain outcomes; the
% diamond is the utility node, a function of the outcome attributes. Setting the
% decision and running inference gives each action's expected utility.
\begin{tikzpicture}[>=stealth, font=\small,
  chance/.style={draw, ellipse, minimum width=18mm, minimum height=8mm, inner sep=1pt, font=\footnotesize},
  dec/.style={draw, minimum width=16mm, minimum height=8mm, font=\footnotesize},
  util/.style={draw, diamond, aspect=1.4, inner sep=1pt, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[dec, draw=acc, text=acc] (site) at (0,1.4) {Airport site};
  \node[chance] (traffic) at (-0.2,-0.1) {Air tra\/f\/f\/ic};
  \node[chance] (litig)   at (-0.2,-1.4) {Litigation};
  \node[chance] (deaths)  at (3.4,0.9)  {Deaths};
  \node[chance] (noise)   at (3.4,-0.4) {Noise};
  \node[chance] (cost)    at (3.4,-1.7) {Cost};
  \node[util, draw=acc, text=acc] (u) at (6.6,-0.4) {U};
  \draw[->, black] (site) -- (deaths);
  \draw[->, black] (site) -- (noise);
  \draw[->, black] (site) to[out=-40,in=150] (cost);
  \draw[->, black] (traffic) -- (deaths);
  \draw[->, black] (litig) to[out=0,in=200] (cost);
  \draw[->, acc] (deaths) -- (u);
  \draw[->, acc] (noise) -- (u);
  \draw[->, acc] (cost) -- (u);
\end{tikzpicture}
$$

## The value of information

A decision network selects an action given the evidence in hand. It says nothing
about which observations are worth acquiring first. Real
agents rarely start with every relevant fact. A doctor does not receive the results
of all possible tests before the patient sits down; tests cost money, time, and
sometimes risk. **Information value theory** decides which observations are worth
acquiring, by treating each observation as a miniature decision whose only effect is
to change the agent's belief state before it commits to a real action.[^aima-voi]

The value of an observation comes entirely from its potential to _change the plan_.
If learning $E_j$ would leave the agent doing exactly what it would have done
anyway, the information is worthless, however interesting. Concretely: with initial
evidence $\mathbf{e}$, the current best action $\alpha$ has value

$$
EU(\alpha \mid \mathbf{e}) = \max_a \sum_{s'} P(\text{Result}(a) = s' \mid a, \mathbf{e})\, U(s'),
$$

and after learning $E_j = e_{jk}$ the agent would instead pick the best action
$\alpha_{e_{jk}}$ for that new evidence. Since $E_j$ is not yet observed, average the
post-observation value over the possible findings, weighted by their current
probabilities, and subtract the value of acting now:

> **Definition (Value of perfect information).** The expected value of learning the
> exact value of a chance variable $E_j$ before acting is
> $$VPI_{\mathbf{e}}(E_j) = \Big(\sum_k P(E_j = e_{jk} \mid \mathbf{e})\; EU(\alpha_{e_{jk}} \mid \mathbf{e}, E_j = e_{jk})\Big) - EU(\alpha \mid \mathbf{e}).$$

Run it on the launch example. Acting now gives $EU(\alpha) = 8$ (`Launch`). The test
is a noisy sensor of demand, but suppose instead a market analyst could reveal
_Demand_ exactly. With probability $0.6$ the report is _high_, and the agent then
launches for $20$; with probability $0.4$ it is _low_, and the agent abandons for
$0$. So the value of acting under perfect knowledge is $0.6(20) + 0.4(0) = 12$, and

$$
VPI(\textit{Demand}) = 12 - 8 = 4.
$$

Perfect information about demand is worth $4$ units; the noisy market test earlier
was worth only $0.4$, because it distinguishes the two states imperfectly. An agent
should pay for the test only if its cost is below what it is worth.

$$
% caption: Three regimes for the value of information, plotting the belief over the
% two actions' utilities. (a) a1 dominates and information will not change the
% choice, so VPI is near zero; (b) the choice is close and the utilities spread
% wide, so information often flips the decision and pays off; (c) the choice is
% close but the stakes are tiny, so information rarely helps enough to matter.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \foreach \dx/\lab in {0/(a) no value, 4.4/(b) high value, 8.8/(c) low value} {
    \begin{scope}[xshift=\dx cm]
      \draw[->, black] (-0.1,0) -- (3.4,0) node[right, font=\scriptsize] {U};
      \node[font=\scriptsize, anchor=north] at (1.6,-0.15) {\lab};
    \end{scope}
  }
  % (a): narrow, well separated
  \draw[acc, thick] (0.2,0) .. controls (0.6,1.9) and (1.0,1.9) .. (1.4,0);
  \node[acc, font=\scriptsize, anchor=south] at (0.8,1.6) {U2};
  \draw[red, thick] (2.0,0) .. controls (2.4,1.9) and (2.8,1.9) .. (3.2,0);
  \node[red, font=\scriptsize, anchor=south] at (2.6,1.6) {U1};
  % (b): broad, overlapping
  \draw[acc, thick] (4.5,0) .. controls (5.6,1.2) and (6.2,1.2) .. (7.3,0);
  \node[acc, font=\scriptsize, anchor=south] at (5.4,0.95) {U2};
  \draw[red, thick] (5.3,0) .. controls (6.4,1.2) and (7.0,1.2) .. (8.1,0);
  \node[red, font=\scriptsize, anchor=south] at (7.2,0.95) {U1};
  % (c): overlapping but one very peaked (small difference)
  \draw[acc, thick] (9.4,0) .. controls (9.9,0.9) and (10.4,0.9) .. (10.9,0);
  \node[acc, font=\scriptsize, anchor=south] at (9.7,0.75) {U2};
  \draw[red, thick] (10.2,0) .. controls (10.45,2.1) and (10.75,2.1) .. (11.0,0);
  \node[red, font=\scriptsize, anchor=south] at (11.35,1.4) {U1};
\end{tikzpicture}
$$

VPI has two general properties. First, information is never harmful in
expectation: $VPI_{\mathbf{e}}(E_j) \ge 0$, because the agent can always ignore an
unhelpful finding and fall back on its original plan.[^aima-voipos] (A specific
finding can still mislead — a false-positive test can trigger needless surgery — but
that is bad luck, not negative _expected_ value.) Second, VPI is not additive: two
observations together can be worth more than the sum of their individual values (one
enables a plan only the other completes) or less (they are redundant). It is,
however, order-independent, which is what lets an agent plan a _sequence_ of queries.

An agent that repeatedly asks the observation with the highest $VPI(E_j)/Cost(E_j)$,
stopping once no observation is worth its cost, is a **myopic** information gatherer
— myopic because it values each observation as if it were the last one it would
make. Myopia is the same greedy shortcut as one-step lookahead, and it is often good
enough; it has been shown to select diagnostic tests better than expert physicians.
Its blind spot is a pair of observations that are worthless alone but decisive
together, where a non-myopic conditional plan would do better.[^aima-myopic]

```algorithm
caption: $\textsc{Information-Gathering-Agent}(percept)$ — acquire observations by value, then act
input: a percept; a persistent decision network $D$
integrate $percept$ into $D$
$j \gets \argmax_k VPI(E_k) / Cost(E_k)$
if $VPI(E_j) > Cost(E_j)$ then
  return $\textsc{Request}(E_j)$
else
  return the best action from $D$
```

That handles a single decision, even one padded out with information gathering. The
real difficulty is what happens when the agent must decide _again and again_, each
choice changing the situation the next one faces — a shift from one-shot utility
maximization to the machinery of sequential decisions.

This continues in [Markov Decision Processes](/artificial-intelligence/uncertainty/markov-decision-processes),
which formalizes acting repeatedly in a stochastic world and solves it with the
Bellman equation, value iteration, and policy iteration.

[^aima-16intro]: **AIMA**, Ch. 16 — Making Simple Decisions: a decision-theoretic agent replaces the goal-based agent's binary good/bad distinction with a continuous measure of outcome quality, combining beliefs (probability) and desires (utility).
[^aima-meu]: **AIMA**, §16.1 — Combining Beliefs and Desires under Uncertainty: expected utility $EU(a \mid \mathbf{e}) = \sum_{s'} P(\text{Result}(a){=}s' \mid a, \mathbf{e})\,U(s')$, and the MEU principle $a^\ast = \argmax_a EU(a \mid \mathbf{e})$, which formalizes but does not operationalize rational action.
[^aima-axioms]: **AIMA**, §16.2.1 — Constraints on rational preferences: the six axioms (orderability, transitivity, continuity, substitutability, monotonicity, decomposability) that any rational preference relation over lotteries must satisfy; violating them exposes the agent to a money pump.
[^aima-existence]: **AIMA**, §16.2.2 — Preferences lead to utility: the von Neumann–Morgenstern result that preferences obeying the axioms are represented by a utility function whose expectation the agent maximizes, with the utility of a lottery equal to the expected utility of its outcomes.
[^aima-affine]: **AIMA**, §16.2.2 — utility is defined only up to an affine transformation $U'(s) = aU(s) + b$, $a > 0$; a deterministic agent needs only an ordinal value function.
[^aima-money]: **AIMA**, §16.3.2 — The utility of money: expected monetary value versus expected utility; the game-show gamble, the concavity of the utility curve, and Grayson's finding that utility is roughly logarithmic in total assets.
[^aima-riskaverse]: **AIMA**, §16.3.2 — risk aversion: for a concave curve $U(L) < U(S_{EMV(L)})$; the certainty equivalent, the insurance premium, and the risk-seeking and risk-neutral regimes.
[^aima-decnet]: **AIMA**, §16.5 — Decision Networks: influence diagrams extending Bayesian networks with decision (rectangle) and utility (diamond) nodes; the evaluation procedure that sets the decision node, runs inference, and returns the maximum-expected-utility action.
[^aima-voi]: **AIMA**, §16.6 — The Value of Information: information value theory as a simplified sequential decision whose observation actions affect only the belief state; the oil-block example showing perfect information about a block is worth as much as the block itself.
[^aima-voipos]: **AIMA**, §16.6.3 — Properties of the value of information: $VPI_{\mathbf{e}}(E_j) \ge 0$ (an agent can ignore misleading information), non-additivity, and order-independence of a sequence of sensing actions.
[^aima-myopic]: **AIMA**, §16.6.4 — the myopic information-gathering agent that requests the observation maximizing $VPI(E_j)/Cost(E_j)$ until no observation beats its cost; myopia's failure on observations useful only in combination.