---
title: Decision-Time Planning
module: Tabular Solution Methods
moduleNumber: 2
lessonNumber: 11
order: 211
summary: >
  Planning need not build a global policy. Decision-time planning runs a fresh
  lookahead every time a state arrives and returns just one action, then throws
  the work away. We start from real-time dynamic programming — asynchronous value
  iteration on the states the agent actually visits — then move through heuristic
  search and rollout algorithms, each a one-step policy improvement applied on the
  fly to the current state.
topics: [Tabular Methods]
sources:
  - book: Sutton & Barto
    ref: "Ch. 8 — Planning and Learning with Tabular Methods; §8.7 Real-time Dynamic Programming; §8.8 Planning at Decision Time"
  - book: Sutton & Barto
    ref: "§8.9 Heuristic Search; §8.10 Rollout Algorithms"
---

The [planning-and-learning](/reinforcement-learning/tabular-methods/planning-and-learning)
lesson built Dyna: a model generates simulated experience, and the same backups
that learn from real experience improve a global value function state by state.
That is **background planning**. When a state $S_t$ finally arrives, its action is
already stored in the table; reading it off is a lookup. All the computation
happened earlier, spread across the whole state space.

There is a second way to use a model. Wait until $S_t$ arrives, then run a
computation whose _only_ output is the single action $A_t$ — a deep, focused
lookahead from _this_ state — and discard everything the moment you have played.
The next state $S_{t+1}$ starts the computation over. This is **decision-time
planning**, and it is how a chess engine spends its clock: not on a policy for all
of chess, but on the best move in the position on the board right now. This lesson
takes the compressed treatment from the planning lesson and works it out in full —
real-time DP, heuristic search, and rollout algorithms — building toward Monte Carlo
Tree Search, the search behind AlphaZero, which the next lesson develops.[^sb-decision]

## Background versus decision-time planning

The distinction is _when_ the planning happens relative to the moment of action,
and _what it produces_.

**Background planning** improves the value function or policy over many states,
ahead of demand, so that action selection is cheap.[^sb-decision] Dyna and
prioritized sweeping are background planners: their updates are not aimed at the
current state in particular. By the time $S_t$ is encountered, planning has already
shaped the table entries used to choose $A_t$, and possibly the entries for a great
many other states too.

**Decision-time planning** begins and completes _after_ encountering each new state,
as a computation whose output is one action. The values and policy it produces are
specific to $S_t$ and its likely successors, and they are usually discarded once the
action is chosen — you rarely return to the exact same state soon enough for saved
values to help. In exchange, every unit of computation
is spent on the decision immediately at hand.

$$
% caption: Two ways to spend a model. Background planning (top) improves a global
% policy over many states ahead of time, so selecting $A_t$ is a table lookup.
% Decision-time planning (bottom) runs a fresh lookahead from $S_t$ and returns
% one action, then discards the work.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=26mm, minimum height=10mm, align=center, font=\footnotesize},
  small/.style={circle, draw, minimum size=4mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- background planning (top row) ---
  \node[font=\footnotesize, anchor=west] at (-4.6,3.5) {background planning};
  \node[box] (model) at (0,3.0) {model};
  \node[box] (table) at (3.6,3.0) {global policy\\(table)};
  \node[box, draw=acc, text=acc] (act1) at (7.6,3.0) {select $A_t$\\(lookup)};
  \draw[->, acc, thick] (model) -- (table) node[midway, above, font=\scriptsize] {sim.};
  \draw[->, acc, thick] (table) -- (act1) node[midway, above, font=\scriptsize] {cheap};
  \node[font=\scriptsize, black, align=center] at (1.8,2.2) {ahead of time,\\for all states};
  % --- decision-time planning (bottom row) ---
  \node[font=\footnotesize, anchor=west] at (-4.6,0.4) {decision-time planning};
  \node[box] (st) at (0,-0.1) {state $S_t$};
  \node[box, draw=red, text=red] (look) at (3.6,-0.1) {lookahead\\from $S_t$};
  \node[box, draw=acc, text=acc] (act2) at (7.6,-0.1) {select $A_t$};
  \draw[->, red, thick] (st) -- (look) node[midway, above, font=\scriptsize] {on demand};
  \draw[->, acc, thick] (look) -- (act2);
  \draw[->, black, dashed] (act2.south) .. controls (5.5,-1.6) and (2.0,-1.6) .. (look.south)
    node[midway, below, font=\scriptsize] {discard, repeat at $S_{t+1}$};
\end{tikzpicture}
$$

> **Definition (Background vs. decision-time planning).** **Background** planning
> uses simulated experience to improve a global value function or policy ahead of
> time, so choosing an action is a cheap table lookup. **Decision-time** planning
> runs on demand for the current state $S_t$, producing one action $A_t$ from a
> focused, often deep lookahead, then throwing the intermediate values and policy
> away.

The two blend in practice — you can focus planning on the current state _and_ save
a little of it so a later return is cheaper — but they are cleanest understood
apart. The trade is latency against depth: background planning suits control that
must respond in milliseconds; decision-time planning suits settings where you can
afford seconds or minutes per move, and a deep lookahead buys a much better action.

## Real-time dynamic programming

Before decision-time planning proper, there is a method between the
two categories, and it illustrates
why _focusing on the states the agent actually visits_ helps. **Real-time
dynamic programming** (RTDP) is an on-policy trajectory-sampling version of the
[value-iteration](/reinforcement-learning/tabular-methods/dynamic-programming)
algorithm of dynamic programming.[^sb-rtdp]

Conventional value iteration sweeps the entire state set, applying the expected
value-iteration update

$$
V(s) \gets \max_{a} \sum_{s', r} p(s', r \mid s, a)\bigl[\,r + \gamma\,V(s')\,\bigr]
$$

to every state, over and over, until the values stop changing. RTDP applies the
_same_ update, but not in systematic sweeps. It is an instance of **asynchronous
dynamic programming**: the update order is dictated by the order in which states are
visited along real or simulated trajectories. At each step of a trajectory, the
agent is in some state $S$; it applies the value-iteration update to $S$ (and
optionally to a few other states, such as those in a limited look-ahead search),
takes a greedy action, and moves on. States that the agent never visits are never
updated.

$$
% caption: RTDP updates only states the agent reaches. From the start states, an
% optimal policy visits the relevant states; irrelevant states are unreachable
% under any optimal policy from any start, and RTDP can skip them entirely.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % outer ellipse: all states
  \draw[black] (0,0) ellipse (5.0 and 2.7);
  \node[black, anchor=north east, font=\footnotesize] at (4.9,2.55) {all states};
  % inner ellipse: relevant states
  \draw[acc, thick] (-1.0,-0.1) ellipse (2.7 and 1.7);
  \node[acc, anchor=south, font=\footnotesize] at (-1.0,1.55) {relevant states};
  % start states blob
  \fill[acc!18] (-2.1,-0.1) circle (0.55);
  \draw[acc] (-2.1,-0.1) circle (0.55);
  \node[acc, font=\scriptsize] at (-2.1,-0.1) {start};
  % an optimal trajectory through relevant states
  \draw[acc, thick, ->] (-1.7,0.2) .. controls (-0.6,0.9) and (0.4,0.4) .. (1.0,-0.5);
  \node[acc, anchor=west, font=\scriptsize] at (1.05,-0.6) {optimal path};
  % irrelevant region marker
  \node[red, font=\scriptsize, align=center] at (3.4,0.4) {irrelevant\\(never visited)};
  \fill[black] (3.4,-0.5) circle (2.2pt);
  \fill[black] (2.9,-0.9) circle (2.2pt);
  \fill[black] (3.9,-1.0) circle (2.2pt);
\end{tikzpicture}
$$

The close tie to conventional DP lets convergence results carry over. If
trajectories can start only from a designated set of **start states**, then for a
prediction problem, on-policy trajectory sampling lets RTDP skip every state that
the given policy cannot reach from any start — those states are **irrelevant** to
the prediction problem. For control, the goal is an **optimal partial policy**: one
that is optimal on the **relevant** states (reachable from some start under _some_
optimal policy) but may be arbitrary, or even undefined, on the irrelevant ones.
There is no need to specify optimal actions where an optimal policy never goes.

> **Definition (Relevant and irrelevant states).** A state is **relevant** if it
> is reachable from some start state under some optimal policy. A state is
> **irrelevant** if no optimal policy reaches it from any start state. An
> **optimal partial policy** is optimal on the relevant states and unconstrained
> on the irrelevant ones.

The key result: for a class of problems, RTDP converges to a policy optimal on
the relevant states _without_ visiting every state infinitely often — sometimes
without visiting some states at all. When only a small fraction of a very large
state set is relevant, that is decisive; even a single full sweep may be infeasible,
yet RTDP never needs one.

In plain terms: as long as the agent can always reach the goal, every step costs
something, and the values start out optimistic, RTDP will settle on the right action
for every state that matters — even though it never updates the states that do
not. The formal statement lists exactly those conditions.

> **Theorem (RTDP convergence, Barto–Bradtke–Singh 1995).** For undiscounted
> episodic tasks with absorbing goal states of zero reward, RTDP converges to a
> policy optimal on all the relevant states — with probability one — provided:
> (1) the initial value of every goal state is zero; (2) at least one policy
> reaches a goal with probability one from every start state; (3) all rewards for
> transitions from non-goal states are strictly negative; and (4) all initial
> values are $\ge$ their optimal values (satisfied by initializing every state to
> zero). At each step RTDP selects a greedy action, breaking ties randomly.

These are the **stochastic optimal-path problems** — cost-minimization tasks such
as minimum-time control, where each step costs $-1$ and the objective is to reach
the goal as cheaply as possible. The convergence proof combines asynchronous-DP
results with results on _learning real-time A\*_ (Korf, 1990), the heuristic-search
algorithm RTDP generalizes.

### RTDP on the racetrack

Sutton and Barto compare RTDP against conventional (sweep-based, Gauss–Seidel)
value iteration on a racetrack: an agent must drive around a turn and cross the
finish line in as few steps as possible, $-1$ per step, restarting from a random
start state on a crash. A small racetrack has $9{,}115$ states reachable from the
start states, but only about $599$ of them are _relevant_ — reachable under some
optimal policy. That gap is where RTDP wins.

| Measure | conventional DP | RTDP |
| --- | --- | --- |
| Computation to convergence | 28 sweeps | 4000 episodes |
| Total value updates | 252{,}784 | 127{,}600 |
| Updates per episode | — | 31.9 |
| % of states updated $\le 100$ times | — | 98.45 |
| % of states updated 0 times | — | 3.18 |

Both methods reach policies averaging 14–15 steps to the finish, but RTDP does it
with roughly _half_ the updates. Conventional value iteration updates every state on
every sweep; RTDP focuses updates on the states relevant to the objective, and that
focus narrows as learning continues. There is a second, subtler advantage: because
RTDP is always greedy with respect to its current values, the policy it uses to
generate trajectories approaches an optimal policy as the values approach $v_\ast$ —
whereas conventional value iteration keeps updating _every_ state right up to
termination and only checks the policy afterward.

## Planning at decision time

RTDP still improves values it might revisit; it leans toward background. Genuine
decision-time planning discards its work. On encountering $S_t$, it begins and
completes a planning computation whose output is the single action $A_t$; on the
next step it begins anew with $S_{t+1}$ to produce $A_{t+1}$, and so on.[^sb-decision]

$$
% caption: The decision-time loop. Encounter a state, plan a lookahead from it,
% emit one action, act, and discard the plan; the next state restarts the
% computation. Planning output is an action, not a stored policy.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=22mm, minimum height=10mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box] (s) at (0,0) {encounter $S_t$};
  \node[box, draw=acc, text=acc] (p) at (3.4,0) {plan from $S_t$};
  \node[box] (a) at (6.8,0) {emit $A_t$};
  \node[box] (e) at (10.2,0) {act,\\observe $S_{t+1}$};
  \draw[->, acc, thick] (s) -- (p);
  \draw[->, acc, thick] (p) -- (a);
  \draw[->, black, thick] (a) -- (e);
  \draw[->, red, thick] (e.south) .. controls (10.2,-1.7) and (0,-1.7) .. (s.south)
    node[midway, below, font=\scriptsize] {discard plan, restart with $S_{t+1}$};
\end{tikzpicture}
$$

Even done only at decision time, planning is still what it always was — computing
updates and values from simulated experience, and ultimately a policy — except the
values and the policy are now specific to $S_t$ and thrown away after selecting the
action. In many applications that is no great loss: the state space is huge and you
are unlikely to return to the same state soon. You can mix the two (focus on the
current state _and_ store some of the result), but decision-time planning is most
useful when fast responses are _not_ required. A chess engine may take minutes per
move and plan dozens of moves ahead in that time; if low-latency selection is the
priority, background planning is the better choice.

## Heuristic search

The classical state-space planners of artificial intelligence — game-tree search
and its relatives — are decision-time planning methods, collectively **heuristic
search**.[^sb-heuristic] For each state encountered, heuristic search considers a
large tree of possible continuations. It applies an approximate value function to
the **leaf** nodes and backs those values up toward the current state at the root.
The backing-up reuses the expected updates with maxes (the ones for $v_\ast$ and
$q_\ast$) from throughout this course — nothing new. The backup stops at the
state–action nodes for the current state; the best of those is chosen as $A_t$, and
all the backed-up values are then discarded.

$$
% caption: Heuristic search as decision-time planning. A lookahead tree is grown
% from the root ($S_t$); an approximate value $\tilde v$ is applied at the leaves
% and backed up with expected max updates toward the root, where the best action
% is chosen. Backed-up values are then discarded.
\begin{tikzpicture}[>=stealth, font=\small,
  nd/.style={circle, draw, fill=white, minimum size=5mm, inner sep=0pt},
  lf/.style={circle, draw, fill=black!8, minimum size=4.5mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % root
  \node[nd, draw=acc, thick] (r) at (0,3.0) {};
  \node[acc, anchor=east, font=\scriptsize] at (-0.25,3.0) {root $S_t$};
  % level 1
  \node[nd] (a) at (-2.4,1.6) {};
  \node[nd] (b) at (0,1.6) {};
  \node[nd] (c) at (2.4,1.6) {};
  \draw[acc, thick] (r) -- (a);
  \draw[black] (r) -- (b);
  \draw[black] (r) -- (c);
  % level 2 leaves
  \node[lf] (la1) at (-3.2,0.1) {};
  \node[lf] (la2) at (-1.6,0.1) {};
  \node[lf] (lb1) at (-0.7,0.1) {};
  \node[lf] (lb2) at (0.7,0.1) {};
  \node[lf] (lc1) at (1.6,0.1) {};
  \node[lf] (lc2) at (3.2,0.1) {};
  \draw[black] (a) -- (la1); \draw[black] (a) -- (la2);
  \draw[black] (b) -- (lb1); \draw[black] (b) -- (lb2);
  \draw[black] (c) -- (lc1); \draw[black] (c) -- (lc2);
  % leaf value label
  \node[black, font=\scriptsize, anchor=north] at (0,-0.35) {leaves: apply approximate value $v$-tilde};
  % backup arrow (red, upward)
  \draw[red, thick, ->] (-1.7,1.75) -- (-0.7,2.6);
  \node[red, anchor=west, font=\scriptsize] at (0.65,2.55) {back up (max)};
  % chosen action marker
  \node[acc, anchor=west, font=\scriptsize] at (-2.3,1.15) {best action};
\end{tikzpicture}
$$

In conventional heuristic search no effort is made to _save_ the backed-up values;
the value function is designed by people and left unchanged. (One can allow it to
learn — that is essentially what we have been doing all along.) The one-step greedy,
$\varepsilon$-greedy, and UCB action-selection rules are heuristic search on a
smaller scale: to pick the greedy action given a model and a state-value function,
you look ahead one step to each possible next state, add rewards and estimated
values, and take the best. Heuristic search is that greedy computation extended
**beyond a single step**.

Why does deeper search help? With a perfect model and an _imperfect_ value function,
deeper search yields better policies. Search all the way to the end of the episode
and the imperfect value function is eliminated entirely — the action is optimal.
Search to depth $k$ with $\gamma^k$ small and the actions are near-optimal, because
the leftover value-function error is discounted away. The cost is compute: deeper
search means a slower response. Tesauro's grandmaster-level backgammon player,
TD-Gammon, illustrated the trade — the deeper its heuristic search, the better its
moves, but the longer each took.

The most important effect of heuristic search is not depth but **focus**. Its search
tree is tightly concentrated on the states and actions that might immediately follow
$S_t$. Those states are the ones where you most want the value function to be
accurate, and where updates matter most for the imminent decision. In chess there
are far too many positions to store a value for each, but a program can easily store
distinct estimates for the millions it encounters while looking ahead from a single
position. Any state-space search can be viewed as piecing together a large number of
individual one-step updates, ordered to concentrate on states downstream of the
current one — which is where the performance gain of deeper search actually comes
from.

## Rollout algorithms

**Rollout algorithms** are decision-time planners built on
[Monte Carlo](/reinforcement-learning/tabular-methods/monte-carlo-methods) control,
applied to simulated trajectories that all begin at the current state.[^sb-rollout]
They estimate action values for a given policy by averaging the returns of many
simulated trajectories that start with each possible action and thereafter follow
that policy — the **rollout policy**. When the estimates are good enough, the action
with the highest value is played, and the whole process repeats from the resulting
next state. The name comes from backgammon: "rolling out" a position means playing
it out to the end many times with random dice.

Unlike the Monte Carlo control of the
[Monte Carlo lesson](/reinforcement-learning/tabular-methods/monte-carlo-methods),
a rollout algorithm does **not** aim to estimate a complete optimal action-value
function $q_\ast$, nor even a complete $q_\pi$. It produces Monte Carlo estimates of
action values only for the current state and a given rollout policy $\pi$. That is
what makes rollouts simple: no need to sample every state–action pair, and no need
to approximate a function over the whole space.

$$
% caption: A rollout at state $s$. For each candidate action $a$, simulate many
% trajectories that take $a$ then follow the rollout policy to termination;
% average their returns to estimate $Q(s,a)$; play the best. Estimates are then
% discarded.
\begin{tikzpicture}[>=stealth, font=\small,
  nd/.style={circle, draw, fill=white, minimum size=6mm, inner sep=0pt},
  term/.style={draw, fill=black!12, minimum size=4mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[nd, draw=acc, thick] (s) at (0,2.4) {};
  \node[acc, anchor=east, font=\scriptsize] at (-0.25,2.4) {state $s$};
  % candidate actions
  \node[nd] (a1) at (-2.6,1.2) {};
  \node[nd] (a2) at (0,1.2) {};
  \node[nd] (a3) at (2.6,1.2) {};
  \draw[acc, thick] (s) -- (a1);
  \draw[black] (s) -- (a2);
  \draw[black] (s) -- (a3);
  \node[font=\scriptsize, anchor=south east] at (-1.5,1.9) {a1};
  \node[font=\scriptsize, anchor=west] at (0.15,2.0) {a2};
  \node[font=\scriptsize, anchor=south west] at (1.5,1.9) {a3};
  % Q estimates beside each action node
  \node[font=\scriptsize, anchor=east] at (-2.95,1.2) {Q=0.7};
  \node[acc, font=\scriptsize, anchor=east] at (-0.35,1.2) {Q=0.9};
  \node[font=\scriptsize, anchor=west] at (2.95,1.2) {Q=0.5};
  % rollouts to terminal (dashed) under rollout policy
  \node[term] (ta1) at (-2.6,-0.9) {};
  \node[term] (ta2) at (0,-0.9) {};
  \node[term] (ta3) at (2.6,-0.9) {};
  \draw[black, dashed, ->] (a1) -- (ta1);
  \draw[black, dashed, ->] (a2) -- (ta2);
  \draw[black, dashed, ->] (a3) -- (ta3);
  \node[black, anchor=west, font=\scriptsize] at (0.35,-0.1) {follow rollout policy};
  \node[anchor=north, font=\scriptsize] at (0,-1.25) {terminal};
  \node[acc, anchor=west, font=\scriptsize] at (0.35,2.4) {pick best action};
\end{tikzpicture}
$$

What does that accomplish? The
[policy-improvement theorem](/reinforcement-learning/tabular-methods/dynamic-programming)
tells us that if two policies $\pi$ and $\pi'$ are identical except that
$\pi'(s) = a \ne \pi(s)$ for one state $s$, and $q_\pi(s,a) \ge v_\pi(s)$, then
$\pi'$ is at least as good as $\pi$ — strictly better if the inequality is strict.
Take $s$ to be the current state and $\pi$ the rollout policy. Averaging the
simulated returns estimates $q_\pi(s,a')$ for each $a' \in \mathcal{A}(s)$; the
policy that plays the best of these and then follows $\pi$ improves on $\pi$. A
rollout is therefore one step of policy improvement, applied on the fly to the
current state and then discarded — like one step of
[asynchronous](/reinforcement-learning/tabular-methods/planning-and-learning) value
iteration that changes the action for a single state.

> **Definition (Rollout algorithm).** A decision-time planner that, at the current
> state $s$, uses Monte Carlo simulations under a fixed **rollout policy** $\pi$ to
> estimate $q_\pi(s,a)$ for each action $a$, plays $\arg\max_a q_\pi(s,a)$, and
> discards the estimates. It performs one step of policy improvement over $\pi$ at
> $s$; it does not try to learn $q_\ast$.

The aim is to improve _over_ the rollout policy, not to find an optimal policy. The
better the rollout policy and the more accurate the value estimates, the better the
resulting decision (though better rollout policies also take longer to simulate).
Because the Monte Carlo trials are independent, they parallelize trivially across
processors; and one can truncate trajectories short of termination, correcting the
truncated return with a stored evaluation function — which brings the whole
apparatus of truncated returns back into play.

```algorithm
caption: $\textsc{Rollout}(s, \pi, m)$ — choose an action at state $s$ by one step of policy improvement over $\pi$
input: current state $s$, rollout policy $\pi$, simulations per action $m$
for each action $a \in \mathcal{A}(s)$ do
  $G_a \gets 0$
  repeat $m$ times
    simulate a trajectory from $s$: take $a$, then follow $\pi$ to termination
    $G \gets$ return of that trajectory
    $G_a \gets G_a + G$
  $Q(s,a) \gets G_a / m$ // Monte Carlo estimate of $q_\pi(s,a)$
return $\arg\max_a Q(s,a)$ // best over the candidates; then discard $Q$
```

This continues in [Monte Carlo Tree Search](/reinforcement-learning/tabular-methods/monte-carlo-tree-search), which turns the rollout algorithm into a planner with memory: it accumulates value estimates across simulations, steers later ones with the UCT rule, and — combined with a deep network — becomes the search behind AlphaGo, AlphaZero, and MuZero.

[^sb-decision]: **Sutton & Barto**, _Reinforcement Learning: An Introduction_ (2nd ed.), §8.8 — Planning at Decision Time: the background-vs.-decision-time distinction, planning as a computation whose output is a single action for $S_t$, and the latency/depth trade-off (chess programs planning dozens of moves ahead per move).
[^sb-rtdp]: **Sutton & Barto**, §8.7 — Real-time Dynamic Programming: RTDP as on-policy trajectory-sampling / asynchronous value iteration, relevant vs. irrelevant states, the optimal-partial-policy convergence result of Barto, Bradtke, and Singh (1995) for stochastic optimal-path problems (its four conditions), and the racetrack comparison (Example 8.6) showing ~half the updates of sweep-based value iteration.
[^sb-heuristic]: **Sutton & Barto**, §8.9 — Heuristic Search: classical state-space search as decision-time planning, applying an approximate value function at the leaves and backing up with expected max updates toward the root, deeper search reducing the effect of value-function error ($\gamma^k$), and the focusing of updates on states downstream of the current one (Figure 8.9).
[^sb-rollout]: **Sutton & Barto**, §8.10 — Rollout Algorithms: Monte Carlo estimates of $q_\pi(s,a)$ under a rollout policy at the current state, rollout as one step of the policy-improvement theorem, the backgammon origin (Tesauro & Galperin, 1997), and the time trade-offs of simulation count and rollout-policy quality.