Reinforcement Learning
Most machine learning trains on labeled answers. Reinforcement learning gets only a number — a reward — and must work out, from its own trial and error, which of its actions earned it.
The setting is an agent acting in an environment: at each step it observes a state, picks an action, and receives a reward and a next state. The goal is not to match a label but to maximize reward accumulated over the long run — which means trading a sure gain now against a larger one later.
One idea organizes the subject: learn a value, the expected long-run reward of a state, and act greedily toward it. The methods differ in how they estimate that value: sweeping a known model with dynamic programming, averaging sampled returns with Monte Carlo, or temporal-difference learning, which updates each estimate toward the next before the episode is even over.
Bootstrapping from one estimate to the next is what makes it work online. It is also what the brain's dopamine neurons appear to compute: a reward-prediction error, the same signal the algorithms are built on.
Scale the tables into function approximators and the same equations drive deep reinforcement learning: the Q-network that first played Atari from pixels, the policy gradients behind modern robot control, and the self-play search that took AlphaZero from the rules of Go to superhuman in a day.
Contents.
1. Foundations
2. Tabular Solution Methods
- Dynamic Programming
- Dynamic Programming: Asynchronous DP and Generalized Policy Iteration
- Monte Carlo Methods
- Monte Carlo Methods: Off-Policy Learning
- Temporal-Difference Learning
- TD Control: Sarsa, Q-learning, and Double Learning
- n-Step Bootstrapping
- n-Step Bootstrapping: Off-Policy Methods
- Planning and Learning
- Planning: Focusing Updates and Decision-Time Search
- Decision-Time Planning
- Monte Carlo Tree Search
3. Approximate Solution Methods
- On-Policy Prediction with Approximation
- Feature Construction and Nonlinear Approximation
- On-Policy Control with Approximation
- Average-Reward Control for Continuing Tasks
- Off-Policy Methods and the Deadly Triad
- Value-Function Geometry and Gradient-TD Methods
- Eligibility Traces
- True Online TD(λ) and Sarsa(λ)
- Policy Gradient Methods
- Actor-Critic Methods and Continuous Actions
- Least-Squares TD
- Memory-Based and Kernel Methods
- Off-Policy Eligibility Traces
- Stable Off-Policy Methods with Traces
4. Deep Reinforcement Learning
5. Modern Deep Reinforcement Learning
- Sharpening DQN: Improvements and the Distributional Idea
- Distributional RL and Rainbow
- Continuous Control: DDPG and TD3
- Continuous Control: SAC and Beyond
- Model-Based Deep RL: Sample Efficiency and PETS
- Model-Based Deep RL: World Models, Dreamer, and MuZero
- Exploration in Deep RL: Novelty as Reward
- Exploration in Deep RL: Posterior Sampling and Go-Explore
- Offline RL: The Problem and Value-Based Fixes
- Offline RL: Implicit Methods, Sequence Models, and Beyond
- Imitation Learning: Cloning, DAgger, and Inverse RL
- Imitation as Adversarial Matching: GAIL and AIRL
- Multi-Agent RL: Markov Games and Centralized Training
- Multi-Agent RL: Self-Play and Solution Concepts
- Hierarchical RL: Options and the Option-Critic
- Hierarchical RL: Goal-Conditioned Hierarchies and Skills
- RLHF and Language Models
- Partial Observability: POMDPs and the Belief State
- Partial Observability: Planning and Recurrent Policies
- Safe and Constrained RL: The CMDP and Policy Methods
- Safe RL: Risk, Safe Exploration, and Alignment
- Meta-RL and Generalization