# Reinforcement Learning

Most machine learning trains on labeled answers. Reinforcement learning
gets only a number — a reward — and must work out, from its own trial
and error, which of its actions earned it.


_Figure 001 — The loop: the agent acts, the environment answers with a state and reward.
_

The setting is an **agent** acting in an
**environment**: at each step it observes a state, picks an
action, and receives a reward and a next state. The goal is not to match a
label but to maximize reward accumulated over the long run — which means
trading a sure gain now against a larger one later.


_Figure 002 — A Markov decision process: states, transitions, and the actions between.
_

One idea organizes the subject: learn a _value_, the expected
long-run reward of a state, and act greedily toward it. The methods
differ in how they estimate that value: sweeping a known model with
dynamic programming, averaging sampled returns with Monte Carlo, or
**temporal-difference learning**, which updates each
estimate toward the next before the episode is even over.


_Figure 003 — A gridworld: value spreads from the goal and the greedy policy points home.
_

Bootstrapping from one estimate to the next is what makes it work
online. It is also what the brain's dopamine neurons appear to compute:
a reward-prediction error, the same signal the algorithms are built on.


_Figure 004 — Temporal-difference learning: the prediction error backs up along the chain.
_

Scale the tables into function approximators and the same equations drive
deep reinforcement learning: the Q-network that first played Atari from
pixels, the policy gradients behind modern robot control, and the
self-play search that took AlphaZero from the rules of Go to superhuman in
a day.


_Figure 005 — A bandit: pulls sharpen each arm's estimate until the best one wins out.
_

---

## Contents

### 1. Foundations

1. [What Is Reinforcement Learning?](/reinforcement-learning/foundations/what-is-reinforcement-learning)
2. [A Brief History of Reinforcement Learning](/reinforcement-learning/foundations/a-brief-history-of-rl)
3. [Multi-Armed Bandits](/reinforcement-learning/foundations/multi-armed-bandits)
4. [Bandit Exploration Algorithms](/reinforcement-learning/foundations/bandit-exploration-algorithms)
5. [Markov Decision Processes](/reinforcement-learning/foundations/markov-decision-processes)
6. [Value Functions and Optimality](/reinforcement-learning/foundations/value-functions-and-optimality)

### 2. Tabular Solution Methods

1. [Dynamic Programming](/reinforcement-learning/tabular-methods/dynamic-programming)
2. [Dynamic Programming: Asynchronous DP and Generalized Policy Iteration](/reinforcement-learning/tabular-methods/dp-async-and-gpi)
3. [Monte Carlo Methods](/reinforcement-learning/tabular-methods/monte-carlo-methods)
4. [Monte Carlo Methods: Off-Policy Learning](/reinforcement-learning/tabular-methods/monte-carlo-off-policy)
5. [Temporal-Difference Learning](/reinforcement-learning/tabular-methods/temporal-difference-learning)
6. [TD Control: Sarsa, Q-learning, and Double Learning](/reinforcement-learning/tabular-methods/td-control-sarsa-and-q-learning)
7. [n-Step Bootstrapping](/reinforcement-learning/tabular-methods/n-step-bootstrapping)
8. [n-Step Bootstrapping: Off-Policy Methods](/reinforcement-learning/tabular-methods/n-step-off-policy-methods)
9. [Planning and Learning](/reinforcement-learning/tabular-methods/planning-and-learning)
10. [Planning: Focusing Updates and Decision-Time Search](/reinforcement-learning/tabular-methods/planning-focusing-and-decision-time)
11. [Decision-Time Planning](/reinforcement-learning/tabular-methods/decision-time-planning)
12. [Monte Carlo Tree Search](/reinforcement-learning/tabular-methods/monte-carlo-tree-search)

### 3. Approximate Solution Methods

1. [On-Policy Prediction with Approximation](/reinforcement-learning/approximation/on-policy-prediction)
2. [Feature Construction and Nonlinear Approximation](/reinforcement-learning/approximation/feature-construction-and-nonlinear)
3. [On-Policy Control with Approximation](/reinforcement-learning/approximation/on-policy-control)
4. [Average-Reward Control for Continuing Tasks](/reinforcement-learning/approximation/average-reward-control)
5. [Off-Policy Methods and the Deadly Triad](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad)
6. [Value-Function Geometry and Gradient-TD Methods](/reinforcement-learning/approximation/bellman-error-and-gradient-td)
7. [Eligibility Traces](/reinforcement-learning/approximation/eligibility-traces)
8. [True Online TD(λ) and Sarsa(λ)](/reinforcement-learning/approximation/true-online-and-sarsa-lambda)
9. [Policy Gradient Methods](/reinforcement-learning/approximation/policy-gradient-methods)
10. [Actor-Critic Methods and Continuous Actions](/reinforcement-learning/approximation/actor-critic-and-continuous-actions)
11. [Least-Squares TD](/reinforcement-learning/approximation/least-squares-and-memory-based-methods)
12. [Memory-Based and Kernel Methods](/reinforcement-learning/approximation/memory-and-kernel-methods)
13. [Off-Policy Eligibility Traces](/reinforcement-learning/approximation/off-policy-eligibility-traces)
14. [Stable Off-Policy Methods with Traces](/reinforcement-learning/approximation/stable-off-policy-traces)

### 4. Deep Reinforcement Learning

1. [Deep Q-Networks](/reinforcement-learning/deep-rl/deep-q-networks)
2. [DQN Improvements: Double, Dueling, and Prioritized Replay](/reinforcement-learning/deep-rl/dqn-improvements)
3. [Actor–Critic and GAE](/reinforcement-learning/deep-rl/actor-critic-and-ppo)
4. [PPO and Continuous Control](/reinforcement-learning/deep-rl/ppo-and-continuous-control)
5. [Case Studies: Learning to Play](/reinforcement-learning/deep-rl/case-studies)
6. [Reinforcement Learning Beyond Games](/reinforcement-learning/deep-rl/rl-beyond-games)
7. [Frontiers: Beyond the Standard MDP](/reinforcement-learning/deep-rl/frontiers)
8. [Reward Design and Open Problems](/reinforcement-learning/deep-rl/reward-design-and-open-problems)

### 5. Modern Deep Reinforcement Learning

1. [Sharpening DQN: Improvements and the Distributional Idea](/reinforcement-learning/modern-deep-rl/distributional-and-rainbow)
2. [Distributional RL and Rainbow](/reinforcement-learning/modern-deep-rl/distributional-and-rainbow-part-2)
3. [Continuous Control: DDPG and TD3](/reinforcement-learning/modern-deep-rl/continuous-control)
4. [Continuous Control: SAC and Beyond](/reinforcement-learning/modern-deep-rl/continuous-control-part-2)
5. [Model-Based Deep RL: Sample Efficiency and PETS](/reinforcement-learning/modern-deep-rl/model-based-rl)
6. [Model-Based Deep RL: World Models, Dreamer, and MuZero](/reinforcement-learning/modern-deep-rl/model-based-rl-part-2)
7. [Exploration in Deep RL: Novelty as Reward](/reinforcement-learning/modern-deep-rl/exploration)
8. [Exploration in Deep RL: Posterior Sampling and Go-Explore](/reinforcement-learning/modern-deep-rl/exploration-part-2)
9. [Offline RL: The Problem and Value-Based Fixes](/reinforcement-learning/modern-deep-rl/offline-rl)
10. [Offline RL: Implicit Methods, Sequence Models, and Beyond](/reinforcement-learning/modern-deep-rl/offline-rl-part-2)
11. [Imitation Learning: Cloning, DAgger, and Inverse RL](/reinforcement-learning/modern-deep-rl/imitation-and-inverse-rl)
12. [Imitation as Adversarial Matching: GAIL and AIRL](/reinforcement-learning/modern-deep-rl/imitation-and-inverse-rl-part-2)
13. [Multi-Agent RL: Markov Games and Centralized Training](/reinforcement-learning/modern-deep-rl/multi-agent-rl)
14. [Multi-Agent RL: Self-Play and Solution Concepts](/reinforcement-learning/modern-deep-rl/multi-agent-rl-part-2)
15. [Hierarchical RL: Options and the Option-Critic](/reinforcement-learning/modern-deep-rl/hierarchical-rl)
16. [Hierarchical RL: Goal-Conditioned Hierarchies and Skills](/reinforcement-learning/modern-deep-rl/hierarchical-rl-part-2)
17. [RLHF and Language Models](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models)
18. [Partial Observability: POMDPs and the Belief State](/reinforcement-learning/modern-deep-rl/partial-observability-pomdps)
19. [Partial Observability: Planning and Recurrent Policies](/reinforcement-learning/modern-deep-rl/partial-observability-pomdps-part-2)
20. [Safe and Constrained RL: The CMDP and Policy Methods](/reinforcement-learning/modern-deep-rl/safe-and-constrained-rl)
21. [Safe RL: Risk, Safe Exploration, and Alignment](/reinforcement-learning/modern-deep-rl/safe-and-constrained-rl-part-2)
22. [Meta-RL and Generalization](/reinforcement-learning/modern-deep-rl/meta-rl-and-generalization)

### 6. Reinforcement Learning in Minds and Brains

1. [The Psychology of Reinforcement](/reinforcement-learning/minds-and-brains/psychology-of-reinforcement)
2. [The Psychology of Reinforcement: Instrumental Control](/reinforcement-learning/minds-and-brains/instrumental-conditioning-and-control)
3. [Dopamine and the TD Error](/reinforcement-learning/minds-and-brains/dopamine-and-td-error)
4. [Dopamine in the Brain: The Neural Actor–Critic](/reinforcement-learning/minds-and-brains/dopamine-in-the-brain)
5. [Animal Learning and Cognition](/reinforcement-learning/minds-and-brains/animal-learning-and-cognition)
6. [Cognitive Maps and Model-Based Learning](/reinforcement-learning/minds-and-brains/cognitive-maps-and-planning)
7. [The Neuroscience of Reinforcement](/reinforcement-learning/minds-and-brains/neuroscience-of-reinforcement)
8. [The Brain's Several Learning Systems](/reinforcement-learning/minds-and-brains/several-learning-systems)