Making Decisions: Utility Theory

A logic agent divides the world in two: goal states and everything else. That is enough to decide what to do only when the agent knows exactly what each action will bring about and cares about nothing but reaching the goal. Neither holds in an uncertain world. Actions have stochastic outcomes, and outcomes differ not just in whether they are the goal but in how good they are. Decision theory addresses both: it replaces the binary goal test with a real-valued utility, replaces certainty about outcomes with a probability distribution over them, and picks the action whose outcomes are best on average.¹

Decision theory = probability theory + utility theory. Probability says what the world will do; utility says how much the agent likes each thing the world might do; and the two combine into a single rule for acting.

Expected utility and the MEU principle

The core idea is a weighted average: list an action's possible outcomes, score each one, and weight the scores by probability. The action with the best weighted-average score is the one to take. This section makes that precise.

An agent about to act does not know the current state exactly (it has only evidence $\mathbf{e}$) and does not know which state an action will produce. Write $\text{Result}(a)$ for the random outcome state of action $a$, so $P(\text{Result}(a)=s^{\prime }\mid a,\mathbf{e})$ is the probability that doing $a$ lands the agent in $s^{\prime }$. A utility function $U(s)$ assigns each state a single number measuring how desirable it is. The expected utility of an action is the average utility of its outcomes, weighted by their probability:²

MEU comes close to defining the whole subject: an intelligent agent has only to estimate its state, compute these two quantities, maximize, and act. In practice each step is hard. Estimating the state needs perception, learning, and inference; computing $P(\text{Result}(a)\mid a,\mathbf{e})$ needs a full causal model of the world and, in general, NP-hard Bayesian-network inference; computing $U(s^{\prime })$ may itself require search or planning, because the agent often cannot tell how good a state is until it knows what it can reach from there. MEU formalizes do the right thing without operationalizing it — that is the rest of AI.²

A worked decision under uncertainty

Consider a startup that must decide whether to Launch a product now, run a Test market first (which costs 2 utility units but reveals demand), or Abandon the idea. Demand is either high (prior $0.6$) or low (prior $0.4$). Launching into high demand is worth $+20$, launching into low demand $-10$; abandoning is worth $0$ regardless. The test is imperfect: it reports up or down, with $P(\text{up}\mid \text{high})=0.8$ and $P(\text{up}\mid \text{low})=0.3$.

Take the two simple actions first. Launching blind has expected utility $EU(\text{Launch})=0.6(20)+0.4(-10)=12-4=8$, and abandoning has $EU(\text{Abandon})=0$, so absent any test the MEU action is Launch, worth $8$.

Now value the test. First fold the priors and the sensor model through Bayes' rule to get the posterior after each report. The marginal probability of up is $P(\text{up})=0.6(0.8)+0.4(0.3)=0.48+0.12=0.60$, so down has probability $0.40$. The posteriors are

After up, launching is worth $0.8(20)+0.2(-10)=16-2=14>0$, so launch. After down, launching is worth $0.3(20)+0.7(-10)=6-7=-1<0$, so abandon (worth $0$). The expected utility of the Test branch, before paying its cost, averages these best responses over the two reports:

Subtracting the test's cost of $2$ leaves $6.4$, which is less than launching blind at $8$. So the MEU decision is to skip the test and launch. The test would be worth running only if its cost fell below $8.4-8=0.4$ — the point where the plan it enables beats the plan you would follow without it. That gap measures the value of information, quantified later in this lesson.

Why a utility function exists

MEU looks arbitrary — why average utility, rather than the worst case, or the sum of cubes? The justification runs the other way. Start not from a utility function but from an agent's preferences over outcomes, and impose only that those preferences be consistent. Consistency alone forces the existence of a utility function that the agent behaves as if it were maximizing.

Because actions have uncertain outcomes, the objects of preference are not states but lotteries: a lottery $L=[p_1,S_1;\dots ;p_n,S_n]$ yields outcome $S_i$ with probability $p_i$. Write $A\succ B$ for $A$ is preferred to $B$, $A\sim B$ for indifference. A rational agent's preferences must obey six axioms:³

Each axiom rules out a way of being pumped. Violate transitivity, holding $A\succ B\succ C\succ A$, and someone can walk you around the cycle — trade you $C$ for $A$ for a cent, $B$ for $C$ for a cent, $A$ for $B$ for a cent — draining your money one loop at a time while you end where you started.

The result is a representation theorem of von Neumann and Morgenstern: an agent whose preferences obey the axioms behaves exactly as if it had a utility function $U$ and were maximizing its expectation.⁴

So MEU follows from the axioms rather than being an extra assumption. The second line is why the average, and not some other functional of the outcomes, is the right thing to maximize.

The theorem determines $U$ only up to an affine transformation: replacing $U(s)$ by $U^{\prime }(s)=aU(s)+b$ with $a>0$ leaves every preference, and hence every decision, unchanged.⁵ Utilities are like temperatures — the Fahrenheit and Celsius scales rank warmth identically. A purely deterministic agent needs even less: only an ordinal ranking of states, called a value function, since with no lotteries the numeric gaps never matter.

Utility of money and risk aversion

Money is the obvious candidate for a utility scale, and agents usually show a monotonic preference for more of it. But utility is not linear in money. Offered a certain $1{,}000{,}000 or a coin flip between $0 and $2{,}500{,}000, most people take the sure million — even though the gamble's expected monetary value, $\frac{1}{2}(0)+\frac{1}{2}(2,500,000)=1,250,000$, is larger.⁶ That is not irrational; it reflects a utility curve that is concave in wealth. An additional million is worth less the wealthier the agent already is. Bernoulli's suggestion, borne out empirically by Grayson, is that utility is roughly logarithmic in total assets.

For any lottery $L$ over the concave region, the utility of the gamble is less than the utility of its expected value paid for sure: $U(L)<U(S_{EMV(L)})$. An agent with this curve is risk-averse. The sure amount it will accept in lieu of the gamble is the certainty equivalent; the gap between the gamble's EMV and its certainty equivalent is the insurance premium, and that is why the insurance industry works — people pay a little more than the expected loss to shed the variance.⁷ Over large negative wealth the curve turns convex, producing risk-seeking behavior (a desperate agent gambles); and for small stakes relative to current wealth any smooth curve is locally linear, so the agent is approximately risk-neutral and maximizes EMV directly.

Decision networks

For a one-shot decision with several uncertain variables, the machinery is packaged into a decision network (or influence diagram): a Bayesian network extended with two new node types. Chance nodes (ovals) are random variables as before; decision nodes (rectangles) are the actions the agent controls; and a single utility node (diamond) holds $U$ as a function of its parents. Evaluating it is the MEU rule made mechanical: for each setting of the decision node, set it as evidence, run standard inference to get the posterior over the utility node's parents, and read off the expected utility — then return the action with the highest one.⁸

The value of information

A decision network selects an action given the evidence in hand. It says nothing about which observations are worth acquiring first. Real agents rarely start with every relevant fact. A doctor does not receive the results of all possible tests before the patient sits down; tests cost money, time, and sometimes risk. Information value theory decides which observations are worth acquiring, by treating each observation as a miniature decision whose only effect is to change the agent's belief state before it commits to a real action.⁹

The value of an observation comes entirely from its potential to change the plan. If learning $E_j$ would leave the agent doing exactly what it would have done anyway, the information is worthless, however interesting. Concretely: with initial evidence $\mathbf{e}$, the current best action $\alpha$ has value

and after learning $E_j=e_{jk}$ the agent would instead pick the best action ${\alpha }_{e_{jk}}$ for that new evidence. Since $E_j$ is not yet observed, average the post-observation value over the possible findings, weighted by their current probabilities, and subtract the value of acting now:

Run it on the launch example. Acting now gives $EU(\alpha )=8$ (Launch). The test is a noisy sensor of demand, but suppose instead a market analyst could reveal Demand exactly. With probability $0.6$ the report is high, and the agent then launches for $20$; with probability $0.4$ it is low, and the agent abandons for $0$. So the value of acting under perfect knowledge is $0.6(20)+0.4(0)=12$, and

Perfect information about demand is worth $4$ units; the noisy market test earlier was worth only $0.4$, because it distinguishes the two states imperfectly. An agent should pay for the test only if its cost is below what it is worth.

VPI has two general properties. First, information is never harmful in expectation: $VP{I}_{\mathbf{e}}(E_j)\ge 0$, because the agent can always ignore an unhelpful finding and fall back on its original plan.¹⁰ (A specific finding can still mislead — a false-positive test can trigger needless surgery — but that is bad luck, not negative expected value.) Second, VPI is not additive: two observations together can be worth more than the sum of their individual values (one enables a plan only the other completes) or less (they are redundant). It is, however, order-independent, which is what lets an agent plan a sequence of queries.

An agent that repeatedly asks the observation with the highest $VPI(E_j)/Cost(E_j)$, stopping once no observation is worth its cost, is a myopic information gatherer — myopic because it values each observation as if it were the last one it would make. Myopia is the same greedy shortcut as one-step lookahead, and it is often good enough; it has been shown to select diagnostic tests better than expert physicians. Its blind spot is a pair of observations that are worthless alone but decisive together, where a non-myopic conditional plan would do better.¹¹

That handles a single decision, even one padded out with information gathering. The real difficulty is what happens when the agent must decide again and again, each choice changing the situation the next one faces — a shift from one-shot utility maximization to the machinery of sequential decisions.

This continues in Markov Decision Processes, which formalizes acting repeatedly in a stochastic world and solves it with the Bellman equation, value iteration, and policy iteration.