--- title: "Planning: Focusing Updates and Decision-Time Search" module: Tabular Solution Methods moduleNumber: 2 lessonNumber: 10 order: 210 summary: > Dyna plans by replaying remembered transitions, but sampling them uniformly wastes most of the effort. This lesson sharpens planning: prioritized sweeping works backward from states whose value just changed, expected versus sample updates weigh thoroughness against cost, and trajectory sampling and real-time DP focus updates on the states the policy actually visits. We trace Dyna forward to model-based deep RL, then turn to decision-time planning — heuristic search, rollouts, and Monte Carlo Tree Search. topics: [Tabular Methods] sources: - book: Sutton & Barto ref: "§8.4 Prioritized Sweeping; §8.5 Expected vs. Sample Updates; §8.6 Trajectory Sampling; §8.7 Real-time DP" - book: Sutton & Barto ref: "§8.8 Planning at Decision Time; §8.9 Heuristic Search; §8.10 Rollout Algorithms; §8.11 Monte Carlo Tree Search" --- This builds on [Planning and Learning](/reinforcement-learning/tabular-methods/planning-and-learning), which showed that planning and learning are the same backup applied to simulated versus real experience, and built the Dyna architecture around that idea. Dyna replays remembered transitions chosen uniformly at random — and most of those replays do no useful work. This lesson makes planning efficient, then turns from background planning to planning done afresh at each decision. ## Prioritized sweeping Dyna-Q starts its simulated transitions at pairs chosen uniformly at random, wasting effort. In the second maze episode only the pair leading into the goal has nonzero value, so an update anywhere else backs up zero onto zero and changes nothing. Under uniform sampling the agent performs many null updates before finding a useful one. To address this, work **backward** from states whose value just changed. When $V(s')$ changes, the only one-step updates that can do useful work are those on actions leading _into_ $s'$; updating them may change _their_ predecessors, so the frontier of useful updates propagates backward from the change. This is **backward focusing**. The frontier grows fast, and not every pair matters equally — some values would change by a lot, others barely. **Prioritized sweeping** orders the work by urgency: keep a queue of state–action pairs keyed by the magnitude of the Bellman error they would incur, and always process the top first. $$ % caption: Prioritized sweeping. A value change at a state (right) makes its % predecessors candidates for update; each is pushed onto a priority queue keyed % by the size of its expected change, and the queue is processed most-urgent % first, so the effect of a change propagates backward until it dies out. \begin{tikzpicture}[>=stealth, font=\small, st/.style={circle, draw, fill=white, minimum size=6mm, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[st] (p1) at (0,1.1) {}; \node[st] (p2) at (0,-1.1) {}; \node[st] (m) at (3.0,0) {}; \node[st, draw=acc, text=acc, thick] (c) at (6.0,0) {}; \draw[->, black] (p1) -- (m); \draw[->, black] (p2) -- (m); \draw[->, red, thick] (m) -- (c) node[midway, above, font=\footnotesize] {change}; \node[acc, anchor=west, font=\footnotesize] at (6.25,0) {value changed}; \node[anchor=east, font=\footnotesize, align=right] at (-0.25,1.1) {predecessor\\(high priority)}; \node[anchor=east, font=\footnotesize, align=right] at (-0.25,-1.1) {predecessor\\(low priority)}; % backward-propagation arrows (queue order) \draw[->, acc, thick] (m) to[bend left=22] (p1); \draw[->, acc, thick] (m) to[bend right=22] (p2); \end{tikzpicture} $$ ```algorithm caption: $\textsc{Prioritized-Sweeping}$ — deterministic environment $Q(s,a), \mathit{Model}(s,a) \gets$ arbitrary; $PQueue \gets$ empty loop // forever $S \gets$ current (nonterminal) state $A \gets \varepsilon\text{-greedy}(S, Q)$ take action $A$, observe $R$, $S'$ $\mathit{Model}(S,A) \gets R, S'$ $P \gets \lvert R + \gamma \max_a Q(S',a) - Q(S,A) \rvert$ if $P > \theta$ then insert $S,A$ into $PQueue$ with priority $P$ repeat $n$ times while $PQueue$ is nonempty $S,A \gets \first(PQueue)$ $R, S' \gets \mathit{Model}(S,A)$ $Q(S,A) \gets Q(S,A) + \alpha\big[R + \gamma \max_a Q(S',a) - Q(S,A)\big]$ for each $\bar S, \bar A$ predicted to lead to $S$ do $\bar R \gets$ predicted reward for $\bar S, \bar A, S$ $P \gets \lvert \bar R + \gamma \max_a Q(S,a) - Q(\bar S, \bar A) \rvert$ if $P > \theta$ then insert $\bar S, \bar A$ into $PQueue$ with priority $P$ ``` On maze tasks prioritized sweeping reaches the optimal policy with 5-to-10 times fewer updates than unprioritized Dyna-Q, and the advantage widens with the grid size. It extends to stochastic environments by keeping transition counts and using **expected** updates over all possible successors — which is also its main limitation, since expected updates can waste computation on low-probability transitions. That tension is the next topic. ## Expected vs. sample updates A backup can be computed two ways. An **expected update** considers _all_ possible next states and rewards, weighting each by its probability; a **sample update** considers a single sampled successor. For the action-value $q_\ast$ with a distribution model $\hat p(s',r \mid s,a)$, the expected update is $$ Q(s,a) \;\gets\; \sum_{s',r} \hat p(s',r \mid s,a)\Big[\, r + \gamma \max_{a'} Q(s',a') \,\Big], $$ while the sample update, given one sampled $S', R$, is the Q-learning form $$ Q(s,a) \;\gets\; Q(s,a) + \alpha\Big[\, R + \gamma \max_{a'} Q(S',a') - Q(s,a) \,\Big]. $$ The expected update is thorough but slow (it visits every successor); the sample update is cheap but noisy (it visits one). Under a fixed compute budget, which wins? They coincide when the environment is deterministic (one successor, $\alpha = 1$). With many successors they diverge: the expected update is exact — its new $Q(s,a)$ is limited in error only by the errors at the successors — while the sample update carries sampling error. So an expected update yields the better estimate but costs more. Let $b$ be the **branching factor**, $b = |\{s' : \hat p(s'\mid s,a) > 0\}|$. One expected update costs about $b$ times as much computation as one sample update. $$ % caption: Backup trees for the two updates. The expected update (left) fans out % to every possible successor and averages exactly; the sample update (right) % follows one sampled transition. An expected update costs about b sample updates, % where b is the branching factor. \begin{tikzpicture}[>=stealth, font=\small, st/.style={circle, draw, fill=white, minimum size=5mm, inner sep=0pt}, ac/.style={circle, draw, fill=black, minimum size=2.4mm, inner sep=0pt}, lf/.style={circle, draw, fill=white, minimum size=4mm, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} % ---- left: expected ---- \begin{scope} \node[ac, label={[font=\footnotesize]above:(s a)}] (sa) at (0,2.4) {}; \node[lf] (l1) at (-1.5,0.9) {}; \node[lf] (l2) at (-0.5,0.9) {}; \node[lf] (l3) at (0.5,0.9) {}; \node[lf] (l4) at (1.5,0.9) {}; \draw[black] (sa) -- (l1); \draw[black] (sa) -- (l2); \draw[black] (sa) -- (l3); \draw[black] (sa) -- (l4); \node[anchor=north, font=\footnotesize] at (0,0.55) {expected update (all s')}; \end{scope} % ---- right: sample ---- \begin{scope}[xshift=6.0cm] \node[ac, label={[font=\footnotesize]above:(s a)}] (sa2) at (0,2.4) {}; \node[lf] (r1) at (0,0.9) {}; \draw[acc, thick] (sa2) -- (r1) node[midway, right, font=\footnotesize] {R}; \node[anchor=north, font=\footnotesize] at (0,0.55) {sample update (one S')}; \end{scope} \end{tikzpicture} $$ If there is time to finish an expected update its estimate is generally better. But with large $b$, the $b$ units of compute for one expected update instead buy $b$ sample updates spread over $b$ pairs, improving each a little. The RMS error of sample updating falls roughly as $$ \text{RMS error} \;\propto\; \sqrt{\frac{b-1}{bt}} $$ after $t$ sample updates, so for moderately large $b$ the error drops to within a few percent of the expected result after only a small fraction of $b$ sample updates. On large stochastic problems with many state–action pairs, spreading cheap sample updates across many pairs beats spending the same effort on a few exact ones; and because the successor estimates become accurate sooner, later backups off those successors compound the advantage.[^sb-expected] ## Trajectory sampling and real-time DP A second axis governs not what an update looks like but which states get updated. Classical DP **sweeps** the whole state space, updating every state once per pass. On large tasks this is hopeless: most states are irrelevant — visited only under poor policies or with tiny probability — yet the sweep spends equal effort on them. The alternative is to draw states from the **on-policy distribution**: follow the current policy $\pi$ in simulation and update the states it visits. This is **trajectory sampling**, and it needs no explicit representation of the distribution — the trajectory generates it for free. Empirically it speeds early planning, at some risk of neglecting the rest of the space in the long run. **Real-time dynamic programming** (RTDP) is the on-policy trajectory-sampling version of value iteration: it applies expected value-iteration updates to the states visited along real or simulated trajectories, in whatever order those visits occur. RTDP is a form of _asynchronous_ DP — no systematic sweeps, updates in trajectory order — and it has a strong guarantee. On stochastic optimal-path problems (undiscounted episodic tasks with an absorbing goal, zero goal value, negative step rewards, and a policy guaranteed to reach the goal), RTDP converges to a policy optimal on the **relevant** states — those reachable from a start state under some optimal policy — without ever visiting the irrelevant ones, and sometimes without visiting some relevant states at all. On a racetrack task with over 9,000 reachable states, RTDP reached near-optimal control with about half the updates that sweep-based value iteration needed, leaving a large fraction of states barely touched or untouched.[^sb-rtdp] ## Beyond the table: model-based deep reinforcement learning Dyna's claim — that a learned model plus replayed simulated experience extracts more policy improvement from scarce interaction — is the founding idea of model-based deep RL, and the modern versions keep the Dyna skeleton while replacing its tabular pieces with neural networks. **Prioritized sweeping, generalized.** Moore and Atkeson's prioritized sweeping, the backward-focusing method above, predates Dyna's tabular limits and was designed precisely for "reinforcement learning with less data and less time."[^moore-plan] Its priority-queue idea survives in deep RL as _prioritized experience replay_: sample stored transitions in proportion to their TD error rather than uniformly, so the network spends gradient steps where the TD error — and thus the learning signal — is largest.[^per] **Learned models you can plan in.** Ha and Schmidhuber's _World Models_ trained a recurrent latent dynamics model from pixels and then trained a controller almost entirely inside that model's "dream," transferring the result back to the real environment — Dyna's indirect route with a deep generative model as the sample model.[^worldmodels] Hafner and colleagues' _Dreamer_ line made this a strong general method: learn a latent world model, then optimize the policy by backpropagating value gradients through imagined (simulated) trajectories rolled out in the model, achieving state-of-the-art data efficiency on continuous control and, in DreamerV3, across a wide span of domains with fixed hyperparameters.[^dreamer] **Planning in a learned model at decision time.** MuZero closes the loop with the decision-time methods below: it learns a latent dynamics model and runs Monte Carlo tree search over it, with no access to the environment's rules, matching AlphaZero on Go, chess, and shogi and setting records on Atari.[^muzero-plan] The Dyna picture — experience improves a model, the model generates simulated experience, and the same backups improve the policy — is intact; every box is now a network. $$ % caption: Dyna's descendants in deep RL. The tabular Dyna loop (experience trains % a model; the model generates simulated experience; shared backups improve the % policy) survives with each piece replaced by a neural network: World Models and % Dreamer learn latent dynamics and plan by imagined rollouts; prioritized replay % is prioritized sweeping's focus; MuZero plans in a learned model at decision time. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=26mm, minimum height=9mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[box, draw=acc, text=acc, thick] (dyna) at (0,0) {tabular Dyna-Q}; \node[box] (wm) at (5.4,2.4) {World Models /\\Dreamer (latent model)}; \node[box] (per) at (5.4,0) {prioritized replay}; \node[box, draw=red, text=red] (mz) at (5.4,-2.4) {MuZero (plan in\\learned model)}; \draw[->, acc, thick] (dyna) to[bend left=12] (wm); \draw[->, thick] (dyna) -- (per); \draw[->, red, thick] (dyna) to[bend right=12] (mz); \node[anchor=south, font=\scriptsize, text=black] at (2.7,1.9) {deep sample model}; \node[anchor=north, font=\scriptsize, text=black] at (2.7,-1.9) {decision-time search}; \end{tikzpicture} $$ ## Planning at decision time Everything so far is **background planning**: use simulated experience to improve a policy or value function over _all_ states ahead of time, so that when $S_t$ arrives its action is already cached and reading it off is trivial. Planning is not focused on the current state; it has already happened. The alternative is **decision-time planning**: on encountering $S_t$, run a planning computation whose only output is the single action $A_t$, then discard the values it produced and start over at $S_{t+1}$. The values and policy are specific to $S_t$ and its likely successors — usually wasteful to save (you rarely revisit the exact state soon), but concentrating all computation on the decision at hand. | | Background planning | Decision-time planning | | --- | --- | --- | | When | ahead of time, over all states | on demand, for the current $S_t$ | | Output | global policy / value function | one action $A_t$ | | Action selection | cheap table lookup | deep focused lookahead | | Reused | yes | discarded after the move | | Suits | low-latency control | seconds–minutes per move (board games) | > **Definition (Background vs. decision-time planning).** **Background** planning > improves a global policy or value function ahead of time, so action selection is > a cheap table lookup. **Decision-time** planning runs on demand for the current > state, producing one action from a focused, deep lookahead and then throwing the > intermediate results away. ### Heuristic search and rollouts **Heuristic search** is the classical decision-time method from artificial intelligence. At each state it builds a large tree of continuations, applies a stored approximate value function at the leaves, and backs those values toward the root exactly as the expected $v_\ast$/$q_\ast$ updates do throughout this course — then picks the best root action and discards the rest. Deeper search improves action selection by pushing the imperfect value function further from the decision; if the search reaches terminal states, the value-function error vanishes entirely. Its main advantage is _focus_: the tree concentrates on the states and actions that might immediately follow $S_t$, so computation and memory are spent where they most affect the imminent decision. **Rollout algorithms** are decision-time planners built on [Monte Carlo](/reinforcement-learning/tabular-methods/monte-carlo-methods) control. At state $s$, for each candidate action $a$ the algorithm simulates many trajectories that start with $a$ and thereafter follow a fixed **rollout policy** $\pi$, estimating $q_\pi(s,a) \approx \frac{1}{m}\sum_{i=1}^{m} G_i$ by the mean of the $m$ sampled returns. It then plays $$ A_t = \arg\max_a \hat q_\pi(s,a). $$ This is one step of policy improvement: by the [policy improvement theorem](/reinforcement-learning/tabular-methods/dynamic-programming), acting greedily w.r.t. $q_\pi$ at $s$ and following $\pi$ after yields a policy at least as good as $\pi$. A rollout algorithm does not learn an optimal value function; it improves over $\pi$ on the fly, then discards the estimates. Better rollout policies and more trajectories give better decisions, at more computation per move. ## Monte Carlo Tree Search **Monte Carlo Tree Search** (MCTS) is a rollout algorithm with memory. Plain rollouts throw away every simulated trajectory; MCTS accumulates the value estimates from successive simulations to steer later ones toward high-reward parts of the space. It is the method that carried computer Go from weak amateur in 2005 to grandmaster by 2015, and, combined with a deep neural network, it is the search inside AlphaGo and AlphaZero.[^sb-mcts] MCTS maintains a **tree** rooted at the current state, holding Monte Carlo value estimates for the state–action pairs most likely to be reached in a few steps. Inside the tree it selects actions by an informed **tree policy** (e.g. $\varepsilon$-greedy or a UCB rule) that balances exploration and exploitation — a UCB tree policy selects $$ \arg\max_a \left[\, Q(s,a) + c\sqrt{\tfrac{\ln N(s)}{N(s,a)}} \,\right], $$ with $N(s,a)$ the edge visit count and $N(s) = \sum_a N(s,a)$; outside the tree it falls back to the simple **rollout policy**. Each iteration runs four steps, repeated until the move-time budget is exhausted. $$ % caption: One iteration of Monte Carlo Tree Search. Selection descends the tree % by the tree policy to a leaf; expansion adds a child; simulation rolls out to a % terminal state under the rollout policy; backup carries the return up the % traversed edges. Iterations repeat until time runs out, then the root's action % is chosen by visit count or value. \begin{tikzpicture}[>=stealth, font=\small, st/.style={circle, draw, fill=white, minimum size=5.5mm, inner sep=0pt}, lf/.style={circle, draw, fill=black!8, minimum size=5mm, inner sep=0pt}, stepbox/.style={draw, minimum width=22mm, minimum height=7mm, align=center, font=\footnotesize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} % --- phase labels across the top --- \node[stepbox] (s1) at (0,4.4) {1. Selection}; \node[stepbox] (s2) at (3.5,4.4) {2. Expansion}; \node[stepbox] (s3) at (7.0,4.4) {3. Simulation}; \node[stepbox] (s4) at (10.5,4.4){4. Backup}; \draw[->, black] (s1) -- (s2); \draw[->, black] (s2) -- (s3); \draw[->, black] (s3) -- (s4); % --- a small shared tree drawn under each phase (schematic) --- % Selection: descend by tree policy to a leaf (blue path) \begin{scope}[shift={(0,0)}] \node[st] (r) at (0,3.3) {}; \node[st] (a) at (-0.7,2.2) {}; \node[st] (b) at (0.7,2.2) {}; \node[st] (c) at (0.35,1.1) {}; \draw[acc, thick] (r) -- (b); \draw[acc, thick] (b) -- (c); \draw[black] (r) -- (a); \node[acc, anchor=west, font=\scriptsize] at (0.9,1.6) {tree policy}; \end{scope} % Expansion: add a new child (blue node) \begin{scope}[shift={(3.5,0)}] \node[st] (r2) at (0,3.3) {}; \node[st] (b2) at (0.7,2.2) {}; \node[st] (c2) at (0.35,1.1) {}; \node[st, draw=acc, thick] (n2) at (0.7,0.1) {}; \draw[black] (r2) -- (b2); \draw[black] (b2) -- (c2); \draw[acc, thick] (c2) -- (n2); \node[acc, anchor=west, font=\scriptsize] at (0.95,0.1) {new node}; \end{scope} % Simulation: roll out to terminal under rollout policy (dashed) \begin{scope}[shift={(7.0,0)}] \node[st] (r3) at (0,3.3) {}; \node[st] (c3) at (0.35,1.1) {}; \node[st] (n3) at (0.7,0.1) {}; \node[draw, fill=black!12, minimum size=4mm, inner sep=0pt] (term) at (0.7,-1.3) {}; \draw[black] (r3) -- (c3); \draw[black] (c3) -- (n3); \draw[acc, dashed, thick, ->] (n3) -- (term); \node[acc, anchor=west, font=\scriptsize] at (0.95,-0.6) {rollout}; \node[anchor=north, font=\scriptsize] at (0.7,-1.5) {terminal}; \end{scope} % Backup: return flows up the traversed edges (red) \begin{scope}[shift={(10.5,0)}] \node[st] (r4) at (0,3.3) {}; \node[st] (c4) at (0.35,1.1) {}; \node[st] (n4) at (0.7,0.1) {}; \node[draw, fill=black!12, minimum size=4mm, inner sep=0pt] (t4) at (0.7,-1.3) {}; \draw[red, thick, ->] (t4) -- (n4); \draw[red, thick, ->] (n4) -- (c4); \draw[red, thick, ->] (c4) -- (r4); \node[red, anchor=west, font=\scriptsize] at (0.95,1.7) {return}; \end{scope} \end{tikzpicture} $$ ```algorithm caption: $\textsc{MCTS}$ — one move from root state $S_t$ initialize tree with root $S_t$ repeat until time budget exhausted // 1. Selection $s \gets$ root while $s$ is in the tree and nonterminal do $a \gets$ tree-policy($s$); $s \gets$ successor of $s, a$ // 2. Expansion if $s$ is nonterminal then add $s$ as a new leaf // 3. Simulation $G \gets$ return of a rollout-policy episode from $s$ to termination // 4. Backup for each edge $(s',a')$ traversed by the tree policy this iteration do $N(s',a') \gets N(s',a') + 1$ $Q(s',a') \gets Q(s',a') + \tfrac{1}{N(s',a')}\big[G - Q(s',a')\big]$ return $\arg\max_a N(S_t, a)$ ``` $$ % caption: A single MCTS state–action estimate keeps running statistics — a visit % count and a mean return — updated on backup. High-count, high-value edges pull % future selections down the same branch, so simulations concentrate on promising % continuations. \begin{tikzpicture}[>=stealth, font=\small, st/.style={circle, draw, fill=white, minimum size=6.5mm, inner sep=0pt}] \definecolor{acc}{HTML}{2348F2} \node[st] (root) at (0,2.0) {}; \node[st] (l) at (-2.2,0) {}; \node[st, draw=acc, thick] (m) at (0,0) {}; \node[st] (r) at (2.2,0) {}; \draw[black] (root) -- (l) node[midway, left, font=\scriptsize] {N=3}; \draw[acc, thick] (root) -- (m) node[midway, left, font=\scriptsize] {N=12}; \draw[black] (root) -- (r) node[midway, right, font=\scriptsize] {N=5}; \node[font=\scriptsize, anchor=north] at (-2.2,-0.45) {Q=0.2}; \node[acc, font=\scriptsize, anchor=north] at (0,-0.45) {Q=0.6}; \node[font=\scriptsize, anchor=north] at (2.2,-0.45) {Q=0.4}; \node[acc, anchor=west, font=\scriptsize] at (0.35,1.65) {most-visited edge}; \end{tikzpicture} $$ The root action is chosen by accumulated statistics — usually the largest visit count $N(S_t,a)$ (robust to outliers) or the largest $Q$. The environment then advances, and MCTS runs again, reusing the surviving subtree and discarding the rest. MCTS is a decision-time rollout algorithm, so it inherits online, incremental, sample-based value estimation and the policy-improvement guarantee of acting greedily on Monte Carlo estimates. Beyond that, by saving edge statistics and expanding along high-return trajectories, it grows a partial action-value table concentrated on the initial segments of promising trajectories — the benefit of a learned $Q$ without approximating a value function globally, while still using past experience to guide exploration. Finally, the leaf-node evaluation and the rollout policy need not stay simple. Replace the hand-written rollout with a deep neural network that evaluates leaf positions and suggests moves, train that network by self-play, and MCTS becomes the search inside AlphaGo and AlphaZero — the subject of the [deep-RL case studies](/reinforcement-learning/deep-rl/case-studies). The tree search supplies the deep, focused lookahead; the network supplies the value and policy priors that make the search tractable. [^sb-expected]: **Sutton & Barto**, §8.4–§8.5 — Prioritized Sweeping (backward focusing, the priority-queue algorithm) and Expected vs. Sample Updates (equations 8.1–8.2, branching factor $b$, and the $\sqrt{(b-1)/(bt)}$ error analysis of Figure 8.7). [^sb-rtdp]: **Sutton & Barto**, §8.6–§8.7 — Trajectory Sampling and Real-time Dynamic Programming: on-policy vs. uniform update distributions, RTDP as asynchronous value iteration, and its convergence on relevant states for stochastic optimal-path problems (racetrack example). [^sb-mcts]: **Sutton & Barto**, §8.8–§8.11 — Planning at Decision Time, Heuristic Search, Rollout Algorithms, and Monte Carlo Tree Search: the four steps selection/expansion/simulation/backup (Figure 8.10), the tree and rollout policies, and the extension to AlphaGo in §16.6. [^moore-plan]: **Moore, A. W., & Atkeson, C. G.** (1993), "Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time," _Machine Learning_ 13(1):103–130 — the priority-queue backward-focusing method, reaching optimality with far fewer updates than uniform Dyna-style planning. [^per]: **Schaul, T., Quan, J., Antonoglou, I., & Silver, D.** (2016), "Prioritized Experience Replay," _ICLR_ — sampling stored transitions in proportion to TD-error magnitude, the deep-RL analogue of prioritized sweeping's priority queue. [^worldmodels]: **Ha, D., & Schmidhuber, J.** (2018), "World Models," _NeurIPS_ — a learned recurrent latent dynamics model in which a compact controller is trained largely inside the model's simulated rollouts before transfer to the real environment. [^dreamer]: **Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M.** (2020), "Dream to Control: Learning Behaviors by Latent Imagination," _ICLR_ (Dreamer); and **Hafner, D., et al.** (2023), "Mastering Diverse Domains through World Models," (DreamerV3) — optimizing policies by backpropagating value gradients through trajectories imagined in a learned latent world model. [^muzero-plan]: **Schrittwieser, J., et al.** (2020), "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model," _Nature_ 588:604–609 (MuZero) — learning a latent dynamics model and running Monte Carlo tree search over it, without access to the environment's true rules.