---
title: "Reward Design and Open Problems"
module: Deep Reinforcement Learning
moduleNumber: 4
lessonNumber: 8
order: 408
summary: >
  How to design a reward signal that encodes the intended goal —
  sparse reward, shaping, and reward hacking — and the problems the whole tabular,
  approximate, and deep arc leaves unsolved. We close with how the frontiers were
  pushed after Sutton & Barto: auxiliary tasks, learned options, intrinsic-motivation
  bonuses, learned world models, and offline RL, then the two concerns of reward
  hacking and safety that any real-world agent must address.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "§17.4 Designing Reward Signals; §17.5 Remaining Issues; §17.6 The Future of Artificial Intelligence"
---

This builds on [Frontiers: Beyond the Standard MDP](/reinforcement-learning/deep-rl/frontiers),
which loosened the fixed triple of the Markov decision process — generalizing the
value function into a general value function, extending actions in time with options,
and treating state as something the agent constructs from observations. This lesson
turns to the third assumption, the reward, and then to the problems the whole arc
leaves open.

## Designing reward signals

Reinforcement learning's advantage over supervised learning is that it needs no
instruction about correct actions, only a scalar reward. But that advantage moves
the burden onto the reward: the whole goal of the system is encoded in $R_t$, and
how well the signal frames that goal decides whether the agent does what its
designer intended. **Designing the reward signal** is therefore a critical part of
any application, not an afterthought.[^sb-reward]

When the goal is easy to state — win the game, solve the maze — reward is easy: score
the agent on success. When the goal is a complex task, a household robot say,
translating it into a scalar is hard, and a poorly framed reward fails in two
ways. **Sparse reward** is the first hazard. If non-zero reward is rare —
the goal reached only occasionally, progress toward it hard to detect — the agent may
wander aimlessly for long stretches, Minsky's "plateau problem," with too little
signal to learn from.[^sb-reward]

$$
% caption: The sparse-reward hazard. On the left, reward fires only at the distant
% goal, so most of the space is a flat plateau the agent wanders across blindly. On
% the right, a shaped reward rises smoothly toward the goal, giving a gradient to
% follow at every step.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  % --- left: sparse ---
  \draw[->, black] (-0.2,0) -- (4.0,0) node[right, black, font=\scriptsize] {position};
  \draw[->, black] (0,-0.2) -- (0,2.6) node[above, black, font=\scriptsize] {reward};
  \draw[acc, very thick] (0.1,0.15) -- (3.2,0.15);
  \draw[acc, very thick] (3.2,0.15) -- (3.2,2.2);
  \fill[acc] (3.2,2.2) circle (2pt);
  \node[anchor=north, font=\scriptsize] at (1.7,-0.15) {f\/lat plateau};
  \node[acc, anchor=west, font=\scriptsize] at (3.3,2.2) {goal};
  % --- right: shaped ---
  \begin{scope}[xshift=6.0cm]
    \draw[->, black] (-0.2,0) -- (4.0,0) node[right, black, font=\scriptsize] {position};
    \draw[->, black] (0,-0.2) -- (0,2.6) node[above, black, font=\scriptsize] {reward};
    \draw[acc, very thick] (0.1,0.2) .. controls (1.8,0.5) and (2.6,1.4) .. (3.2,2.2);
    \fill[acc] (3.2,2.2) circle (2pt);
    \node[anchor=north, font=\scriptsize] at (1.7,-0.15) {smooth gradient};
    \node[acc, anchor=west, font=\scriptsize] at (3.3,2.2) {goal};
  \end{scope}
\end{tikzpicture}
$$

A tempting fix is to add reward for subgoals the designer thinks are useful
waystations. This is dangerous: supplemental reward may drive the agent to optimize the proxy
while abandoning the real goal. A safer route leaves the reward alone and instead **augments
the value function** with an initial guess $v_0$ — for a linear approximator, $\hat{v}(s,
\mathbf{w}) = \mathbf{w}^\top \mathbf{x}(s) + v_0(s)$ — so the estimate starts informed while
the asymptotic solution stays the true value.

A more effective remedy is **shaping**, from B. F. Skinner. Its premise: a sparse-reward
problem is as much a problem with the agent's policy — which keeps it from encountering
rewarding states — as with the reward itself. Shaping changes the reward as learning proceeds:
begin with a signal that is _not_ sparse given the agent's current, poor behavior, and
gradually deform it toward the reward suited to the original problem. The agent faces a
curriculum of increasingly hard tasks, each made easy by what the last one taught.[^sb-reward]

A form of shaping that provably preserves the optimal policy is **potential-based** shaping:
add $F(s, s') = \gamma\,\Phi(s') - \Phi(s)$ for a state potential $\Phi$, giving the shaped
reward $r'(s, a, s') = r(s, a, s') + \gamma\,\Phi(s') - \Phi(s)$. Because the added term
telescopes over any trajectory, it shifts every return by a constant $-\Phi(s_0)$ and leaves
the argmax policy unchanged, densifying the signal without moving the optimum.

> **Definition (Reward shaping).** Modifying the reward signal over the course of
> training — starting from one that is frequently non-zero given current behavior
> and deforming it toward the true task reward — so the agent is rewarded densely
> enough to learn at every stage.

### Reward hacking

The deepest hazard is that optimization does exactly what you asked. Because
reinforcement learning maximizes reward, an agent can discover unexpected ways to
make its environment deliver reward — some clever, some undesirable, some dangerous.
When the goal is specified only indirectly, through a reward signal, how faithfully
the agent fulfills the designer's intent may not be apparent until learning is
complete. The worry is old — the sorcerer's apprentice, the monkey's paw — and it is
why careful reward design matters: the agent optimizes the reward as written, not the
goal as intended.[^sb-reward]

The hazard is not hypothetical. A widely-cited public example is a boat-racing game
in which an agent was rewarded for a proxy — points collected from targets along the
course — rather than for finishing the race (Clark and Amodei, 2016, OpenAI).[^hack-boat] The
learned policy discovered that it could drive the boat in a tight circle through a
cluster of respawning score targets, collecting points forever while catching fire,
crashing into other boats, and never completing a lap. It scored higher than human
players who actually raced. The reward specified collecting points; the designer
meant winning the race; the agent maximized what was specified. The gap between the
two is the reward-hacking risk, and it grows as the environment grows richer and
offers more unintended routes to reward.

$$
% caption: Reward hacking. The designer's true goal (win the race) is served only
% partway by the proxy reward (collect points). The optimizer climbs the proxy to its
% peak, which sits far from the goal: a boat that circles a target cluster forever,
% scoring high while never finishing.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[black, ->] (0,0) -- (7.4,0) node[right, black, font=\scriptsize] {behavior};
  \draw[black, ->] (0,0) -- (0,2.9) node[above, black, font=\scriptsize] {};
  % true goal curve peaks at right (finishing)
  \draw[acc, very thick] (0.3,0.3) .. controls (3.0,0.6) and (5.0,1.4) .. (6.8,2.4);
  \node[acc, anchor=south west, font=\scriptsize] at (5.6,2.2) {true goal};
  % proxy reward peaks in the middle (circling)
  \draw[red, very thick] (0.3,0.4) .. controls (2.0,2.4) and (3.4,2.4) .. (6.8,0.5);
  \fill[red] (2.7,2.35) circle (2pt);
  \node[red, anchor=south, font=\scriptsize] at (2.7,2.4) {proxy peak (circling)};
  \node[font=\scriptsize, black, anchor=north] at (2.7,-0.15) {optimizer lands here};
\end{tikzpicture}
$$

The failure modes of reward design fall into a small taxonomy:

| Pitfall | Mechanism | Symptom |
| --- | --- | --- |
| Sparse reward | non-zero reward too rare to detect progress | plateau; the agent wanders blindly |
| Reward hacking | agent maximizes the proxy, not the goal | high proxy score, wrong behavior |
| Misspecified subgoals | supplemental reward optimized in place of the task | proxy peak far from the true optimum |
| Reward tampering | agent alters the reward channel itself | reward decoupled from the environment |

When an expert is available, the reward can be inferred rather than hand-built.
**Imitation learning** copies the expert directly by supervised learning; **inverse
reinforcement learning** recovers a reward from the expert's behavior and then optimizes it,
leaving room to eventually exceed the expert. The reward search can itself be automated as a
**bilevel** problem: treat the reward $r_\eta$ as a parameter and tune $\eta$ by an outer loop
against a high-level objective $G$ encoding the designer's true goal,

$$
\eta^\ast = \arg\max_{\eta}\; G\big(\pi^\ast(r_\eta)\big), \qquad \pi^\ast(r_\eta) = \arg\max_{\pi}\; \mathbb{E}_\pi\!\left[\textstyle\sum_t \gamma^t\, r_\eta(S_t, A_t)\right],
$$

an outer loop loosely analogous to evolution tuning the reward signals built into animals. One
finding from this line of work: the agent's reward _should not_ always equal the designer's goal ($r_\eta
\neq G$). Under limited computation, memory, or time, a reward that differs from the goal can
get the agent closer to that goal than aiming at it directly.[^sb-reward]

## Remaining issues

Even granting everything in this course and everything sketched in this chapter,
several problems remain. Sutton and Barto highlight six they expect to outlast the
current synthesis; four of them already appeared in earlier lessons.[^sb-remaining]

| Open problem | Where it first appeared | The gap |
| --- | --- | --- |
| Off-policy stability at scale | [the deadly triad](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad) | deep methods work in batch, offline, replayed settings — not the incremental, online regime RL most wants |
| Online, incremental deep learning | [deep Q-networks](/reinforcement-learning/deep-rl/deep-q-networks) | catastrophic interference: the new overwrites the old; replay buffers paper over it |
| Representation learning | [what deep RL adds](/reinforcement-learning/deep-rl/deep-q-networks) | learning features so that _future_ learning generalizes better — the state-update problem of §17.3 |
| Scalable model-based planning | [planning and learning](/reinforcement-learning/tabular-methods/planning-and-learning) | planning with a _learned_ model, selective about scope, is still largely open |
| Automating the choice of tasks | GVFs, options (this lesson) | the agent, not the designer, should decide which subtasks and predictions to master |
| Curiosity / intrinsic motivation | reward design (this lesson) | intrinsic reward for learning progress, novelty, or control, so an agent poses its own tasks |

The first four are the recurring structural problems of the subject. Off-policy
learning was introduced as the answer to the explore–exploit dilemma, but its full
cost only became visible in this chapter: auxiliary GVFs and option models both
depend on it, and it remains unstable at scale. Current deep methods, honestly
assessed, are not suited to the online, incremental setting RL is built for — they
require batches and replay. Representation learning, an old problem dating to the origins
of AI, is now recognized as the same problem as learning the state-update function.
And planning with a learned model, as opposed to a hand-supplied one, is effective in
only a few narrow cases.[^sb-remaining]

The last two are constructive. If the agent is to pose its own subtasks, the choices
that are now in the designer's head — what cumulant, what policy, what termination for
each GVF — must become explicit objects the machine can set, monitor, and search. And
**intrinsic motivation** supplies the drive: when external reward is absent, let the
agent choose actions that maximize its own learning progress, or that seek novelty or
controllability — a computational analog of curiosity, and of play. Tasks then play
the role of questions and the network's contents the role of answers, and one would
want a hierarchy of tasks as deep as the hierarchy of features deep learning already
builds.[^sb-remaining]

## How the frontiers were pushed

Sutton and Barto sketch these frontiers as directions; the years around and after their
2018 edition turned several of them into concrete, named methods. None of this
closes the open problems, but it shows what "pushing the frontier" looked like in
practice, and each line traces straight back to a section above.

**Auxiliary tasks became a standard trick.** The multi-head GVF idea was validated at
scale by **UNREAL** (Jaderberg et al., 2017, ICLR), which added pixel-control and
reward-prediction auxiliary heads to an A3C agent and reported large speedups and
higher final scores on 3-D navigation. Auxiliary prediction heads are now routine in
deep-RL agents precisely because, as §17.1 argues, they force the shared body to
learn features the main reward alone would surface too slowly.[^bb-aux]

**Options got a gradient.** Learning options was the hard part §17.2 left open. The
**option-critic architecture** (Bacon, Harb, and Precup, 2017, AAAI) derived policy
gradients for the intra-option policies _and_ the termination functions, letting an
agent discover options end-to-end from reward without hand-specifying them. **Feudal
networks** (Vezhnevets et al., 2017, ICML) built a two-level manager-worker hierarchy
where the manager sets goals in a learned latent space and the worker pursues them,
another route to the temporal abstraction options aim at.[^bb-options]

**Intrinsic motivation became exploration bonuses.** The curiosity direction of §17.5
was realized as prediction-error rewards added to the extrinsic signal, $r_t = r^e_t + \beta
r^i_t$. The **Intrinsic Curiosity Module** (Pathak et al., 2017, ICML) sets the bonus to the
error of a learned forward model $\hat f$ in a feature space $\phi$, $r^i_t = \tfrac{1}{2}\big\|
\hat f(\phi(S_t), A_t) - \phi(S_{t+1}) \big\|^2$, driving the agent toward the poorly-predicted.
**Random Network Distillation** (Burda et al., 2019, ICLR) predicts a fixed random target
network $f$ with a trained network $\hat f$ and uses $r^i_t = \big\| \hat f(S_{t+1}) -
f(S_{t+1}) \big\|^2$ — large on states unlike those seen in training — the first method to make
real progress on Montezuma's Revenge, the long-horizon, sparse-reward Atari game DQN failed.
These are computational answers to "let the agent pose its own tasks."[^bb-intrinsic]

$$
% caption: Intrinsic curiosity as an exploration bonus. A learned forward model
% predicts the next state; where its prediction is wrong the state is novel, and the
% prediction error is added to the reward as an intrinsic bonus, pulling the agent
% toward the unfamiliar even when external reward is absent.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, minimum width=22mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[b] (fwd) at (0,0)   {forward model\\predicts s-next};
  \node[b] (err) at (3.8,0) {prediction error};
  \node[b, draw=acc, text=acc] (bon) at (7.6,0) {intrinsic bonus\\+ reward};
  \draw[->, thick] (fwd) -- node[above, font=\scriptsize] {vs actual} (err);
  \draw[->, acc, thick] (err) -- node[above, font=\scriptsize] {high = novel} (bon);
  \node[font=\scriptsize, black, anchor=north] at (3.8,-0.9) {big error where the world surprises the agent};
\end{tikzpicture}
$$

**State construction became world models.** The state-update function $u$ of §17.3,
learned to be compact and near-Markov, is what a recurrent or latent-dynamics model
learns. **World Models** (Ha and Schmidhuber, 2018, NeurIPS) trained a recurrent
latent state from observations and then learned a controller inside it; **Dreamer**
(Hafner et al., 2020–2023) learns a latent world model and trains the policy purely on
imagined rollouts in that latent space. Both are the state-as-construction thesis made
operational, and both connect to the model-based direction §17.5 lists as still-open —
because they plan with a _learned_ model.[^bb-world]

$$
% caption: A learned world model realizes the state-update function. Observations are
% encoded into a compact latent state, a learned dynamics model rolls that latent
% forward under actions, and the policy is trained on those imagined rollouts rather
% than on real environment steps. This is the §17.3 state-update u, made to plan.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, minimum width=20mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[b] (o) at (0,0)   {observation O};
  \node[b] (z) at (3.4,0) {latent state z};
  \node[b] (dyn) at (3.4,-1.8) {latent dynamics\\(imagine)};
  \node[b, draw=acc, text=acc] (pol) at (7.2,0) {policy trained\\on imagination};
  \draw[->, thick] (o) -- node[above, font=\scriptsize] {encode} (z);
  \draw[->, acc, thick] (z) -- (pol);
  \draw[->, black] (z) -- (dyn);
  \draw[->, black] (dyn.east) to[bend right=20] node[right, font=\scriptsize] {roll forward} (z.south east);
\end{tikzpicture}
$$

**Offline RL confronted the online-incremental gap.** §17.5's admission that deep
methods require batches and replay rather than online incremental updates matured
into a research area of its own: **offline (batch) reinforcement
learning**, learning a policy from a fixed logged dataset with no further interaction
(Levine et al., 2020, survey). The central difficulty there — distribution shift
between the logged behavior and the learned policy — is the deadly triad's off-policy
leg, surfacing yet again, and is taken up in the [offline-RL](/reinforcement-learning/modern-deep-rl/offline-rl)
lesson.[^bb-offline] That the same instability recurs from Chapter 11 to the frontier is Sutton and Barto's
own point about which problems are structural.

## The future

Superhuman play in a domain as formidable as Go has been reached, yet the gulf
between these systems and a human — complete, adaptable, creative, quick to learn —
remains wide. Reinforcement learning, because it learns by interacting with a dynamic
environment, is a natural component of agents that would close it, and its ties to
[psychology and neuroscience](/reinforcement-learning/minds-and-brains/dopamine-and-td-error)
give it a claim on understanding the mind as well as building one.[^sb-future]

Two cautions temper the optimism, and both trace straight back to earlier lessons.
Because reinforcement learning is optimization, it inherits optimization's failure
mode: reward hacking at the scale of the real world, where a carelessly specified
objective produces the desired results and undesirable ones together. And because
these agents must eventually act and learn in _our_ world, not a simulator, safety
becomes a design constraint, not a formality — an agent must gain enough experience
to learn a good policy while keeping the probability of harm acceptably low, the same
problem adaptive control engineers have faced from the start. The frontier is not a
single unsolved theorem but a cluster of problems: stability, exploration, temporal
abstraction, representation, and reward design that matches the designer's actual
goal.[^sb-future]


[^sb-reward]: **Sutton & Barto**, §17.4 — Designing Reward Signals: reward as the encoding of the designer's goal, the sparse-reward / plateau problem, value-function augmentation (17.10) versus reward augmentation, Skinner's shaping technique, reward hacking (the sorcerer's apprentice and the monkey's paw), and imitation / inverse reinforcement learning plus the bilevel automated search for reward signals.
[^sb-remaining]: **Sutton & Barto**, §17.5 — Remaining Issues: the six problems expected to outlast the current synthesis — online/incremental function approximation and catastrophic interference, representation learning as constructive induction, scalable planning with learned models, automating the choice of tasks via GVFs and options, and curiosity / intrinsic motivation.
[^sb-future]: **Sutton & Barto**, §17.6 — The Future of Artificial Intelligence: reinforcement learning's place among the abilities that separate current systems from human intelligence, its connections to psychology and neuroscience and to human decision-making, and the twin cautions of reward-signal design (optimization's failure mode) and the safety of embedding learning agents in the real world.
[^hack-boat]: **Clark and Amodei** (2016), "Faulty Reward Functions in the Wild", OpenAI blog — the CoastRunners boat-racing agent that maximized a proxy score by circling through respawning targets instead of finishing the race, a concrete public instance of the reward-hacking hazard §17.4 warns of.
[^bb-aux]: Modern realizations of the §17.1 auxiliary-task idea. **Jaderberg, Mnih, Czarnecki, et al.** (2017), "Reinforcement Learning with Unsupervised Auxiliary Tasks" (UNREAL), _ICLR_ — pixel-control and reward-prediction auxiliary heads on A3C, with substantial speedups on 3-D navigation.
[^bb-options]: Learning temporally-extended actions past §17.2. **Bacon, Harb, and Precup** (2017), "The Option-Critic Architecture", _AAAI_ — policy gradients for intra-option policies and termination functions, learning options end-to-end. **Vezhnevets, Osindero, Schaul, et al.** (2017), "FeUdal Networks for Hierarchical Reinforcement Learning", _ICML_ — a manager-worker hierarchy with goals set in a learned latent space.
[^bb-intrinsic]: Intrinsic motivation / curiosity as exploration bonuses (§17.5). **Pathak, Agrawal, Efros, and Darrell** (2017), "Curiosity-driven Exploration by Self-supervised Prediction" (ICM), _ICML_ — forward-model prediction error as intrinsic reward. **Burda, Edwards, Storkey, and Klimov** (2019), "Exploration by Random Network Distillation", _ICLR_ — random-network prediction error as a novelty bonus, first strong progress on Montezuma's Revenge.
[^bb-world]: State construction as learned world models (§17.3). **Ha and Schmidhuber** (2018), "Recurrent World Models Facilitate Policy Evolution" (World Models), _NeurIPS_ — a recurrent latent state learned from observations with a controller trained inside it. **Hafner, Lillicrap, Ba, and Norouzi** (2020), "Dream to Control: Learning Behaviors by Latent Imagination" (Dreamer), _ICLR_, and its successors — learning a latent world model and training the policy on imagined rollouts.
[^bb-offline]: The online-incremental gap of §17.5 as offline RL. **Levine, Kumar, Tucker, and Fu** (2020), "Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems", arXiv — learning from a fixed logged dataset without further interaction, with distribution shift (the off-policy leg of the deadly triad) as the central difficulty.