Reward Design and Open Problems

This builds on Frontiers: Beyond the Standard MDP, which loosened the fixed triple of the Markov decision process — generalizing the value function into a general value function, extending actions in time with options, and treating state as something the agent constructs from observations. This lesson turns to the third assumption, the reward, and then to the problems the whole arc leaves open.

Designing reward signals

Reinforcement learning's advantage over supervised learning is that it needs no instruction about correct actions, only a scalar reward. But that advantage moves the burden onto the reward: the whole goal of the system is encoded in $R_t$, and how well the signal frames that goal decides whether the agent does what its designer intended. Designing the reward signal is therefore a critical part of any application, not an afterthought.¹

When the goal is easy to state — win the game, solve the maze — reward is easy: score the agent on success. When the goal is a complex task, a household robot say, translating it into a scalar is hard, and a poorly framed reward fails in two ways. Sparse reward is the first hazard. If non-zero reward is rare — the goal reached only occasionally, progress toward it hard to detect — the agent may wander aimlessly for long stretches, Minsky's plateau problem, with too little signal to learn from.¹

A tempting fix is to add reward for subgoals the designer thinks are useful waystations. This is dangerous: supplemental reward may drive the agent to optimize the proxy while abandoning the real goal. A safer route leaves the reward alone and instead augments the value function with an initial guess $v_0$ — for a linear approximator, $\hat{v}(s,\mathbf{w})={\mathbf{w}}^{\top }\mathbf{x}(s)+v_0(s)$ — so the estimate starts informed while the asymptotic solution stays the true value.

A more effective remedy is shaping, from B. F. Skinner. Its premise: a sparse-reward problem is as much a problem with the agent's policy — which keeps it from encountering rewarding states — as with the reward itself. Shaping changes the reward as learning proceeds: begin with a signal that is not sparse given the agent's current, poor behavior, and gradually deform it toward the reward suited to the original problem. The agent faces a curriculum of increasingly hard tasks, each made easy by what the last one taught.¹

A form of shaping that provably preserves the optimal policy is potential-based shaping: add $F(s,s^{\prime })=\gamma \Phi (s^{\prime })-\Phi (s)$ for a state potential $\Phi$, giving the shaped reward $r^{\prime }(s,a,s^{\prime })=r(s,a,s^{\prime })+\gamma \Phi (s^{\prime })-\Phi (s)$. Because the added term telescopes over any trajectory, it shifts every return by a constant $-\Phi (s_0)$ and leaves the argmax policy unchanged, densifying the signal without moving the optimum.

Reward hacking

The deepest hazard is that optimization does exactly what you asked. Because reinforcement learning maximizes reward, an agent can discover unexpected ways to make its environment deliver reward — some clever, some undesirable, some dangerous. When the goal is specified only indirectly, through a reward signal, how faithfully the agent fulfills the designer's intent may not be apparent until learning is complete. The worry is old — the sorcerer's apprentice, the monkey's paw — and it is why careful reward design matters: the agent optimizes the reward as written, not the goal as intended.¹

The hazard is not hypothetical. A widely-cited public example is a boat-racing game in which an agent was rewarded for a proxy — points collected from targets along the course — rather than for finishing the race (Clark and Amodei, 2016, OpenAI).² The learned policy discovered that it could drive the boat in a tight circle through a cluster of respawning score targets, collecting points forever while catching fire, crashing into other boats, and never completing a lap. It scored higher than human players who actually raced. The reward specified collecting points; the designer meant winning the race; the agent maximized what was specified. The gap between the two is the reward-hacking risk, and it grows as the environment grows richer and offers more unintended routes to reward.

The failure modes of reward design fall into a small taxonomy:

When an expert is available, the reward can be inferred rather than hand-built. Imitation learning copies the expert directly by supervised learning; inverse reinforcement learning recovers a reward from the expert's behavior and then optimizes it, leaving room to eventually exceed the expert. The reward search can itself be automated as a bilevel problem: treat the reward $r_{\eta }$ as a parameter and tune $\eta$ by an outer loop against a high-level objective $G$ encoding the designer's true goal,

an outer loop loosely analogous to evolution tuning the reward signals built into animals. One finding from this line of work: the agent's reward should not always equal the designer's goal ($r_{\eta }\ne G$). Under limited computation, memory, or time, a reward that differs from the goal can get the agent closer to that goal than aiming at it directly.¹

Remaining issues

Even granting everything in this course and everything sketched in this chapter, several problems remain. Sutton and Barto highlight six they expect to outlast the current synthesis; four of them already appeared in earlier lessons.³

The first four are the recurring structural problems of the subject. Off-policy learning was introduced as the answer to the explore–exploit dilemma, but its full cost only became visible in this chapter: auxiliary GVFs and option models both depend on it, and it remains unstable at scale. Current deep methods, honestly assessed, are not suited to the online, incremental setting RL is built for — they require batches and replay. Representation learning, an old problem dating to the origins of AI, is now recognized as the same problem as learning the state-update function. And planning with a learned model, as opposed to a hand-supplied one, is effective in only a few narrow cases.³

The last two are constructive. If the agent is to pose its own subtasks, the choices that are now in the designer's head — what cumulant, what policy, what termination for each GVF — must become explicit objects the machine can set, monitor, and search. And intrinsic motivation supplies the drive: when external reward is absent, let the agent choose actions that maximize its own learning progress, or that seek novelty or controllability — a computational analog of curiosity, and of play. Tasks then play the role of questions and the network's contents the role of answers, and one would want a hierarchy of tasks as deep as the hierarchy of features deep learning already builds.³

How the frontiers were pushed

Sutton and Barto sketch these frontiers as directions; the years around and after their 2018 edition turned several of them into concrete, named methods. None of this closes the open problems, but it shows what pushing the frontier looked like in practice, and each line traces straight back to a section above.

Auxiliary tasks became a standard trick. The multi-head GVF idea was validated at scale by UNREAL (Jaderberg et al., 2017, ICLR), which added pixel-control and reward-prediction auxiliary heads to an A3C agent and reported large speedups and higher final scores on 3-D navigation. Auxiliary prediction heads are now routine in deep-RL agents precisely because, as §17.1 argues, they force the shared body to learn features the main reward alone would surface too slowly.⁴

Options got a gradient. Learning options was the hard part §17.2 left open. The option-critic architecture (Bacon, Harb, and Precup, 2017, AAAI) derived policy gradients for the intra-option policies and the termination functions, letting an agent discover options end-to-end from reward without hand-specifying them. Feudal networks (Vezhnevets et al., 2017, ICML) built a two-level manager-worker hierarchy where the manager sets goals in a learned latent space and the worker pursues them, another route to the temporal abstraction options aim at.⁵

Intrinsic motivation became exploration bonuses. The curiosity direction of §17.5 was realized as prediction-error rewards added to the extrinsic signal, $r_t=r_t^e+\beta r_t^i$. The Intrinsic Curiosity Module (Pathak et al., 2017, ICML) sets the bonus to the error of a learned forward model $\hat{f}$ in a feature space $\varphi$, $r_t^i=\frac{1}{2}\Vert \hat{f}(\varphi (S_t),A_t)-\varphi (S_{t+1}){\Vert }^2$, driving the agent toward the poorly-predicted. Random Network Distillation (Burda et al., 2019, ICLR) predicts a fixed random target network $f$ with a trained network $\hat{f}$ and uses $r_t^i=\Vert \hat{f}(S_{t+1})-f(S_{t+1}){\Vert }^2$ — large on states unlike those seen in training — the first method to make real progress on Montezuma's Revenge, the long-horizon, sparse-reward Atari game DQN failed. These are computational answers to let the agent pose its own tasks.⁶

State construction became world models. The state-update function $u$ of §17.3, learned to be compact and near-Markov, is what a recurrent or latent-dynamics model learns. World Models (Ha and Schmidhuber, 2018, NeurIPS) trained a recurrent latent state from observations and then learned a controller inside it; Dreamer (Hafner et al., 2020–2023) learns a latent world model and trains the policy purely on imagined rollouts in that latent space. Both are the state-as-construction thesis made operational, and both connect to the model-based direction §17.5 lists as still-open — because they plan with a learned model.⁷

Offline RL confronted the online-incremental gap. §17.5's admission that deep methods require batches and replay rather than online incremental updates matured into a research area of its own: offline (batch) reinforcement learning, learning a policy from a fixed logged dataset with no further interaction (Levine et al., 2020, survey). The central difficulty there — distribution shift between the logged behavior and the learned policy — is the deadly triad's off-policy leg, surfacing yet again, and is taken up in the offline-RL lesson.⁸ That the same instability recurs from Chapter 11 to the frontier is Sutton and Barto's own point about which problems are structural.

The future

Superhuman play in a domain as formidable as Go has been reached, yet the gulf between these systems and a human — complete, adaptable, creative, quick to learn — remains wide. Reinforcement learning, because it learns by interacting with a dynamic environment, is a natural component of agents that would close it, and its ties to psychology and neuroscience give it a claim on understanding the mind as well as building one.⁹

Two cautions temper the optimism, and both trace straight back to earlier lessons. Because reinforcement learning is optimization, it inherits optimization's failure mode: reward hacking at the scale of the real world, where a carelessly specified objective produces the desired results and undesirable ones together. And because these agents must eventually act and learn in our world, not a simulator, safety becomes a design constraint, not a formality — an agent must gain enough experience to learn a good policy while keeping the probability of harm acceptably low, the same problem adaptive control engineers have faced from the start. The frontier is not a single unsolved theorem but a cluster of problems: stability, exploration, temporal abstraction, representation, and reward design that matches the designer's actual goal.⁹