---
title: "Model-Based Deep RL: Sample Efficiency and PETS"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 5
order: 505
summary: >
  A model turns experience into imagined planning. This lesson makes the
  sample-efficiency case for learning a dynamics model, works through why a learned
  model's errors compound over the planning horizon, and builds the most direct
  model-based method: PETS plans online with a probabilistic ensemble under
  model-predictive control, distrusting the model exactly where its members disagree.
  A companion lesson takes up latent world models (Dreamer) and MuZero.
topics: [Deep RL]
sources:
  - book: Grokking Deep RL
    ref: "Ch. 12 — model-based methods; sample efficiency and learned models"
  - book: Sutton & Barto
    ref: "Ch. 8 — Planning and Learning (Dyna, MCTS), scaled to function approximation"
---

The [tabular planning chapter](/reinforcement-learning/tabular-methods/planning-and-learning)
made one argument worth carrying into deep RL: planning and learning are the same
backup machine, differing only in whether the experience they consume is real or
_simulated_ by a model. Dyna kept a table of observed transitions and replayed
them; MCTS grew a search tree from a known simulator. Both assumed the model was
either exact (the game rules) or tiny (a lookup table). Neither assumption holds
for pixels, continuous control, or dynamics the agent is never handed.

Model-based deep RL asks the harder question: what if you must **learn** the
model, in high dimensions, from the same limited experience you are trying to
act on? Get it right and the payoff is large: a learned model lets you generate
unlimited imagined experience and plan against it, so each real interaction goes
much further. Get it wrong and the model's errors compound over the planning
horizon into confident nonsense. This lesson is about that trade and the most direct
way to win it: **PETS** — learn a probabilistic model, then plan against it while
distrusting it where it is unsure. A
[companion lesson](/reinforcement-learning/modern-deep-rl/model-based-rl-part-2) takes
the two more radical lines: **World Models / Dreamer** (learn a compact latent world
and imagine inside it) and **MuZero** (learn a latent model that predicts only what
planning needs).

## Why model-based: the sample-efficiency argument

[Model-free](/reinforcement-learning/deep-rl/deep-q-networks) methods learn a
value function or policy directly from reward, discarding each transition after a
gradient step (or, with replay, after a few). They are simple and unbiased by any
model's mistakes, but they are **sample-hungry**: DQN needed tens of millions of
Atari frames, and model-free continuous control routinely spends millions of
environment steps. When a "step" is a real robot moving, or a real dollar spent,
that price is the whole problem.

A model changes this arithmetic. A single real transition $(s, a, r, s')$, once
absorbed into a learned dynamics model $\hat p(s' \mid s, a)$, can seed thousands
of _imagined_ transitions that never touch the environment. Planning against those
imagined transitions is planning against a compressed summary of everything the
agent has seen. That is the sample-efficiency case in one line: **a good model
turns experience into imagined planning**, and imagined planning is nearly free.

$$
% caption: Model-free vs. model-based on the sample-efficiency axis. Both approach
% the same asymptotic return, but the model-based learner reaches good performance
% in far fewer real environment steps, because each real step is amortized over
% many imagined ones. The model-free curve wins asymptotically only if the model
% carries residual bias.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, black] (0,0) -- (8.4,0) node[anchor=north, font=\footnotesize, text=black] {real environment steps};
  \draw[->, black] (0,0) -- (0,4.4) node[anchor=east, font=\footnotesize, text=black, rotate=90, yshift=6mm] {return};
  % asymptote line
  \draw[black, dashed] (0,3.7) -- (8.2,3.7);
  \node[anchor=east, font=\scriptsize, text=black] at (8.2,3.95) {near-optimal};
  % model-based: steep early climb
  \draw[acc, very thick] (0.2,0.2) .. controls (1.0,2.9) and (2.0,3.5) .. (5.5,3.6);
  \node[acc, anchor=west, font=\footnotesize] at (5.6,3.55) {model-based};
  % model-free: slow climb
  \draw[red, very thick] (0.2,0.15) .. controls (3.5,0.6) and (6.0,3.0) .. (8.0,3.5);
  \node[red, anchor=west, font=\footnotesize] at (7.2,2.7) {model-free};
\end{tikzpicture}
$$

The catch is in the same figure. A model-free method fits its value directly to
returns, so it is asymptotically unbiased; a model-based method inherits whatever
bias its model carries, and if the model is imperfect the planned policy can plateau
below what a patient model-free learner would reach. The engineering of the whole
field is about getting the early steepness without inheriting too much of the late
gap.

> **Definition (Sample efficiency).** How much _real_ environment interaction a
> method needs to reach a target level of performance. Model-based RL trades extra
> computation — fitting a model and planning against it — for fewer real samples,
> which pays whenever real interaction is expensive relative to compute.

### The compounding-error problem

The reason a learned model is not simply a free win is that planning uses it
**recursively**. To imagine a trajectory of length $H$, you feed the model's own
output back as its next input: $\hat s_1 = \hat p(\cdot \mid s_0, a_0)$, then
$\hat s_2 = \hat p(\cdot \mid \hat s_1, a_1)$, and so on. Each step adds a little
error, and each subsequent step is now conditioned on an already-wrong state, so
the errors do not merely add; they compound.

$$
% caption: Compounding model error. A one-step model with small per-step error
% is queried on its own predicted state, so the predicted trajectory (blue)
% drifts away from the true trajectory (black) and the gap widens with the
% horizon H. This is why long imagined rollouts through a learned model become
% unreliable, and why model-based methods keep the planning horizon short or
% correct the state periodically.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % true trajectory
  \draw[black, very thick] (0,0) .. controls (2.5,0.3) and (5.0,0.5) .. (8.0,0.7);
  \node[black, anchor=west, font=\footnotesize] at (8.05,0.7) {true};
  % predicted trajectory drifting away
  \draw[acc, very thick] (0,0) .. controls (2.5,0.7) and (5.0,1.7) .. (8.0,3.2);
  \node[acc, anchor=west, font=\footnotesize] at (8.05,3.2) {predicted};
  % start dot
  \fill[black] (0,0) circle (2.2pt);
  \node[anchor=east, font=\scriptsize] at (-0.1,0) {s0};
  % error gap markers at increasing horizons
  \foreach \x/\yt/\yp in {2.5/0.34/0.72, 5.0/0.52/1.68, 7.4/0.66/2.86} {
    \draw[red, thick] (\x,\yt) -- (\x,\yp);
  }
  \node[red, anchor=north west, font=\scriptsize] at (5.15,1.1) {growing error};
  % horizon axis
  \draw[->, black] (0,-0.7) -- (8.0,-0.7) node[anchor=north, font=\footnotesize, text=black] {planning horizon H};
\end{tikzpicture}
$$

For example, suppose the model's one-step prediction is
unbiased but noisy, adding an independent error of standard deviation $\sigma = 0.1$
(in state units) at each step, and — the pessimistic-but-realistic case — that the
dynamics are mildly expansive, so an error at one step is amplified by a factor
$L = 1.2$ before the next error is added. After $H$ steps the accumulated error has
standard deviation

$$
\sigma_H = \sigma \sqrt{\sum_{k=0}^{H-1} L^{2k}}
= \sigma \sqrt{\frac{L^{2H} - 1}{L^2 - 1}}.
$$

At $H = 1$ this is just $0.1$. At $H = 5$ it is $0.1\sqrt{(1.2^{10}-1)/(1.44-1)} =
0.1\sqrt{(6.19-1)/0.44} = 0.1\sqrt{11.8} = 0.34$. At $H = 15$ it is
$0.1\sqrt{(1.2^{30}-1)/0.44} = 0.1\sqrt{(237.4-1)/0.44} = 0.1\sqrt{537} = 2.3$ — a
$23\times$ blow-up of the per-step error over a horizon of fifteen. Even with a
_contractive_ system ($L < 1$) the sum converges but the error still grows before it
saturates; with $L \ge 1$ it grows without bound. This is the arithmetic behind the
figure and the reason every method below either keeps $H$ short (so $\sigma_H$ stays
near $\sigma$), quantifies the growing uncertainty so the planner discounts far-horizon
rollouts, or sidesteps state prediction entirely.

Three responses run through the rest of the lesson. **Quantify the uncertainty** so
planning distrusts the model where it is unsure — the ensemble idea behind PETS.
**Shorten the effective horizon** by replanning every step from the true state
instead of committing to a long imagined plan — model-predictive control, also in
PETS. And **change what the model predicts**: instead of reconstructing full future
states (pixels), predict only the quantities planning consumes — reward and value —
so per-step errors in irrelevant detail never enter the loop, which is the move
MuZero makes.

## Learn a model, then plan: PETS

The most direct model-based recipe is the one Dyna sketched: fit a dynamics model
to observed transitions, then use it to choose actions. **PETS**, Probabilistic
Ensembles with Trajectory Sampling, Chua et al. (2018), is the version that made
this competitive with model-free methods on continuous control, and it does so by
taking the compounding-error problem seriously.[^pets]

Two design choices carry it. First, the model is **probabilistic**: each network
outputs not a point prediction but a Gaussian over the next state,
$\hat p_\phi(s' \mid s, a) = \mathcal{N}\big(\mu_\phi(s,a),\, \Sigma_\phi(s,a)\big)$,
capturing the environment's own stochasticity (**aleatoric** uncertainty). Second,
the model is an **ensemble** of $B$ such networks, trained on different bootstraps
of the data; the disagreement _between_ ensemble members captures uncertainty about
the model itself (**epistemic** uncertainty), which is large exactly where data is
scarce and the model should not be trusted.

$$
% caption: A probabilistic ensemble. B networks each output a Gaussian over the
% next state given (s, a); each captures the environment noise (aleatoric), and
% their disagreement captures model uncertainty (epistemic). Where data is
% plentiful the members agree; where data is scarce they spread out, warning the
% planner not to trust its rollouts there.
\begin{tikzpicture}[>=stealth, font=\small,
  net/.style={draw, minimum width=17mm, minimum height=8mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[font=\footnotesize] (in) at (0,0) {(s, a)};
  \node[net] (n1) at (2.6,1.5) {net 1};
  \node[net] (n2) at (2.6,0.0) {net 2};
  \node[net] (nb) at (2.6,-1.5) {net B};
  \draw[->, black] (in) -- (n1);
  \draw[->, black] (in) -- (n2);
  \draw[->, black] (in) -- (nb);
  \node[net, draw=acc, text=acc] (g1) at (5.6,1.5) {Gaussian 1};
  \node[net, draw=acc, text=acc] (g2) at (5.6,0.0) {Gaussian 2};
  \node[net, draw=acc, text=acc] (gb) at (5.6,-1.5) {Gaussian B};
  \draw[->, acc] (n1) -- (g1);
  \draw[->, acc] (n2) -- (g2);
  \draw[->, acc] (nb) -- (gb);
  \node[anchor=north, font=\scriptsize, text=black] at (2.6,-2.1) {each on a data bootstrap};
  \node[red, anchor=west, font=\scriptsize] at (7.1,0.0) {spread = model doubt};
  \draw[red, ->] (6.9,0.0) -- (6.5,0.6);
  \draw[red, ->] (6.9,0.0) -- (6.5,-0.6);
\end{tikzpicture}
$$

With a model in hand, PETS does not train a policy at all. It plans **online** by
**model-predictive control (MPC)**: at the current real state $s_t$, it searches
over short action sequences, evaluates each by rolling it forward through the
ensemble and summing predicted rewards, executes only the _first_ action of the
best sequence, then throws the plan away and replans from the next real state.
Replanning every step is what limits compounding error — the imagined rollout only
has to be trusted for $H$ steps, after which the true state resets the plan.

> **Definition (Model-predictive control).** A control loop that, at each step,
> optimizes an action sequence over a finite horizon $H$ using a model, executes
> only the first action, then re-optimizes from the newly observed state. Closing
> the loop on the real state each step keeps model error from accumulating over a
> long open-loop plan.

The evaluation of a candidate action sequence must fold in the ensemble's
uncertainty rather than trusting one network. PETS uses **trajectory sampling**:
to score a sequence, it propagates several particles forward, each particle at each
step drawing a successor from a (bootstrap-chosen) ensemble member's Gaussian. The
particles fan out where the ensemble disagrees, so a sequence that scores well only
under one over-confident member earns a poor average return. The action
search itself is a sampling optimizer — the cross-entropy method (CEM), which
repeatedly samples action sequences, keeps the top-scoring fraction, and refits the
sampling distribution to them.

$$
% caption: PETS rollouts under model-predictive control. From the current state,
% CEM proposes candidate action sequences; each is evaluated by propagating
% particles through the probabilistic ensemble (the fan of light trajectories),
% and the summed predicted reward ranks the sequences. Only the first action of
% the best sequence (blue) is executed, then the whole search repeats from the
% next real state.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % current state
  \fill[acc] (0,0) circle (2.6pt);
  \node[anchor=east, font=\footnotesize] at (-0.15,0) {state};
  % fanned-out candidate particle rollouts (light)
  \draw[black] (0,0) .. controls (1.6,1.3) and (3.2,1.6) .. (4.6,1.9);
  \draw[black] (0,0) .. controls (1.6,0.9) and (3.2,1.5) .. (4.6,1.3);
  \draw[black] (0,0) .. controls (1.6,-0.9) and (3.2,-1.3) .. (4.6,-1.1);
  \draw[black] (0,0) .. controls (1.6,-1.3) and (3.2,-1.9) .. (4.6,-2.2);
  % the chosen best sequence (blue, bold)
  \draw[acc, very thick] (0,0) .. controls (1.6,0.2) and (3.2,0.3) .. (4.6,0.35);
  \node[acc, anchor=west, font=\footnotesize] at (4.65,0.35) {best sequence};
  % first action executed marker
  \fill[acc] (1.15,0.1) circle (2.2pt);
  \node[acc, anchor=north, font=\scriptsize] at (1.55,-0.55) {execute f\/irst action};
  % horizon bracket
  \draw[black, ->] (0,-2.7) -- (4.6,-2.7) node[anchor=north, font=\footnotesize, text=black] {horizon H};
  % relabel the particle fan
  \node[anchor=west, font=\scriptsize, text=black] at (4.65,1.6) {ensemble particles};
\end{tikzpicture}
$$

One cross-entropy-method iteration on a scalar toy problem shows the loop; the same
loop runs (vectorized over the action sequence) inside PETS. Say the
planner is choosing a single action $a \in \mathbb{R}$ and the true reward peaks near
$a = 2$. CEM starts with a sampling distribution $\mathcal{N}(\mu_0, \sigma_0^2) =
\mathcal{N}(0, 2^2)$ and draws, say, six candidates: $\{-2.1,\, -0.4,\, 0.9,\, 1.8,\,
2.3,\, 3.5\}$. Rolling each through the model and scoring by predicted reward, the two
best turn out to be the ones nearest the peak, $\{1.8,\, 2.3\}$ (the **elite set**).
CEM refits the sampling distribution to the elites: the new mean is their average,
$\mu_1 = (1.8 + 2.3)/2 = 2.05$, and the new standard deviation is their spread,
$\sigma_1 = \sqrt{\tfrac{1}{2}[(1.8-2.05)^2 + (2.3-2.05)^2]} = 0.25$. The next round
samples from $\mathcal{N}(2.05, 0.25^2)$ — already tightly centered on the optimum —
and after a few such rounds the distribution collapses onto the best action. The whole
method is: sample, keep the elite fraction, refit to them, repeat. It needs no
gradients, which is what lets it optimize through a black-box learned model.

$$
% caption: One cross-entropy-method iteration. Candidates are sampled from the current
% Gaussian (top), scored through the model, and the top-scoring elite fraction (filled)
% is kept; the Gaussian is refit to the elites (bottom), tightening around the optimum.
% Repeating collapses the sampler onto the best action sequence.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axis
  \draw[black, ->] (-2.6,0) -- (4.2,0) node[anchor=north east, text=black] {action a};
  % round 0 broad gaussian
  \draw[acc, thick] plot[domain=-2.4:4.0, samples=60] (\x, {1.1*exp(-((\x-0)^2)/8) + 1.5});
  \node[acc, anchor=west, font=\scriptsize] at (-2.4,2.4) {round 0: broad};
  % candidates as dots on the axis
  \foreach \x in {-2.1, -0.4, 0.9, 3.5} \fill[black] (\x,1.5) circle (1.8pt);
  \foreach \x in {1.8, 2.3} \fill[red] (\x,1.5) circle (2.4pt);
  \node[red, anchor=south, font=\scriptsize] at (2.05,1.75) {elites};
  % round 1 narrow gaussian
  \draw[red, thick] plot[domain=1.0:3.1, samples=60] (\x, {1.3*exp(-((\x-2.05)^2)/0.12)});
  \node[red, anchor=west, font=\scriptsize] at (3.0,0.9) {round 1: tight at 2.05};
  \node[anchor=north, font=\scriptsize, text=black] at (2.0,-0.15) {2};
\end{tikzpicture}
$$

Empirically, on continuous-control benchmarks like the
half-cheetah, PETS matched the asymptotic performance of strong model-free methods
(SAC, PPO) while using **roughly an order of magnitude fewer environment steps** to
get there.[^pets] The probabilistic ensemble was the essential part: ablating
it to a single deterministic network collapsed the gains, because without a
calibrated estimate of its own uncertainty the planner exploits the model's
errors.

## Where this leaves us

PETS is the most literal reading of the model-based promise: fit a dynamics model to
observed transitions, then plan against it. Two ideas make it work despite compounding
error. The **probabilistic ensemble** gives the planner a calibrated estimate of model
uncertainty — members agree where data is plentiful and spread out where it is scarce, so
trajectory sampling naturally distrusts rollouts into unexplored regions. And
**model-predictive control** replans from the true state every step, so any imagined
rollout only has to be trusted for a short horizon before reality resets the plan.

PETS still predicts full next states in the environment's native coordinates. The two
methods in the companion lesson push on that. **World Models and Dreamer** learn a
compact _latent_ state and do almost all their learning by imagining inside it;
**MuZero** goes further and predicts neither states nor pixels, only the reward, value,
and policy that planning actually reads. Both continue in
[Model-Based Deep RL: World Models, Dreamer, and MuZero](/reinforcement-learning/modern-deep-rl/model-based-rl-part-2).

[^pets]: **Chua, Calandra, McAllister, Levine (2018)**, "Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models," _NeurIPS_. PETS: a bootstrap ensemble of probabilistic (Gaussian-output) dynamics networks separating aleatoric from epistemic uncertainty, planned online by model-predictive control with cross-entropy-method action search and particle-based trajectory sampling; matched model-free asymptotic performance on continuous control (e.g. half-cheetah) with roughly an order of magnitude fewer samples.