---
title: "Reinforcement Learning Beyond Games"
module: Deep Reinforcement Learning
moduleNumber: 4
lessonNumber: 6
order: 406
summary: >
  The same value-and-reward machinery, pointed at problems with no opponent. Web
  personalization as a contextual bandit and then a full MDP for life-time value;
  thermal soaring, where a glider learns to climb on turbulent air and reward design
  does most of the work; and the industrial-scale systems that carried the same
  design past Sutton & Barto — AlphaStar, OpenAI Five, GT Sophy, and RLHF, where the
  reward itself is learned from human preference.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "§16.7 Personalized Web Services; §16.8 Thermal Soaring"
---

This builds on [Case Studies: Learning to Play](/reinforcement-learning/deep-rl/case-studies),
which traced one skeleton — a value function learned by self-play, refined by search,
carried by a deep network — from Samuel's checkers player through TD-Gammon, DQN, and
AlphaGo Zero. Those systems played games. The cases here point the same machinery at
problems with no opponent, and then follow the skeleton to industrial scale.

## Personalized web services: bandits at internet scale

The systems above played games. The next two show the same value-and-reward
machinery pointed at problems with no opponent — first at web-scale content
selection, where the "game" is a stream of anonymous visits and the reward is a
click.[^sb-web] Personalizing which news story or advertisement a page shows is a
recommendation problem, and it maps cleanly onto the
[multi-armed bandit](/reinforcement-learning/foundations/multi-armed-bandits)
foundations. The plain bandit is _nonassociative_: one situation, a fixed set of
arms, and the task is to find the single best arm. Marketing's long-standing
**A/B test** — show version A or version B and keep the one users prefer — is
exactly a two-armed bandit run on the whole population at once, and because it is
nonassociative it cannot personalize.

Adding **context** is what personalizes. Attach to each decision a feature vector
describing the user (time since last visit, visit count, location, inferred
interests, demographics) and the candidate content, and the problem becomes the
**contextual bandit** of the bandits lesson: on each round the learner sees a
context, picks an arm, and collects a reward, and the best arm now depends on the
context.[^sb-web] Li, Chu, Langford, and Schapire applied a contextual-bandit
algorithm to choosing the featured story on the Yahoo! Front Page, maximizing the
**click-through rate** — clicks divided by visits — and beat a context-free bandit
by 12.5%. The lift is entirely the value of conditioning on who is looking.

$$
% caption: The contextual-bandit serving loop. On each visit the site observes a
% context $x$ (user and content features), the policy selects one piece of content
% from $K$ candidate arms, and the user's click (or non-click) is the reward, 1 or
% 0, that trains the policy for the next visit.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=25mm, minimum height=11mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box] (ctx)  at (0,0)    {context x\\(user, content)};
  \node[box, draw=acc, text=acc, thick] (pol) at (3.7,0) {policy\\pick 1 of K};
  \node[box] (serve) at (7.4,0) {serve content};
  \node[box] (rew)  at (11.0,0) {reward\\click = 1 / 0};
  \draw[->, acc, thick] (ctx) -- (pol);
  \draw[->, acc, thick] (pol) -- (serve);
  \draw[->, acc, thick] (serve) -- (rew);
  \draw[->, red, thick] (rew.south) .. controls (11.0,-1.7) and (3.7,-1.7) .. (pol.south)
    node[midway, below, font=\scriptsize, text=red] {update policy on the observed reward};
\end{tikzpicture}
$$

### Greedy versus long-term: the exploration the scale hides

The contextual bandit has a quiet defect at scale. It is **greedy across time**: it
maximizes the probability of an immediate click and treats every visit as if a
fresh, anonymous visitor drew it from the population. But many users return, and a
policy that ignores that throws away the whole point of _sequential_ interaction.
Theocharous, Thomas, and Ghavamzadeh reframed personalization as a full MDP whose
reward is the total clicks a user makes over _repeated_ visits, so the state
carries the history of one identified user across sessions rather than resetting
each time.[^sb-web]

Their running example is a car ad. A greedy policy offers the closing discount on
the first visit; the user either bites or leaves, and on any later visit sees the
same offer again. A longer-term policy walks the user down a sales funnel —
mention favorable financing, praise the service department, and only on a later
visit present the discount — trading a lower immediate click probability for more
clicks, and more eventual sales, across the relationship. This is the multi-armed
bandit's exploration–exploitation tradeoff lifted from single actions to whole
_sequences_ of offers: acting greedily on each visit is a local optimum that an
agent planning over the user's lifetime can beat.

$$
% caption: Greedy (contextual-bandit) selection versus long-term (MDP) selection
% for a car ad. The greedy branch offers the discount immediately and often ends
% the interaction; the long-term branch spends early visits building interest and
% presents the discount later, earning more clicks over the user's repeated
% visits.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=24mm, minimum height=10mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % greedy row
  \node[font=\scriptsize, anchor=east, text=red] at (-0.3,1.6) {greedy};
  \node[box, draw=red, text=red] (g1) at (1.4,1.6) {of\/fer discount};
  \node[box] (g2) at (5.6,1.6) {take it or leave};
  \draw[->, red, thick] (g1) -- (g2);
  % long-term row
  \node[font=\scriptsize, anchor=east, text=acc] at (-0.3,-0.6) {long-term};
  \node[box, draw=acc, text=acc] (l1) at (1.4,-0.6)  {show f\/inancing};
  \node[box] (l2) at (5.6,-0.6)  {praise service};
  \node[box] (l3) at (9.8,-0.6)  {of\/fer discount};
  \draw[->, acc, thick] (l1) -- (l2) node[midway, above, font=\scriptsize, text=black] {visit 1};
  \draw[->, acc, thick] (l2) -- (l3) node[midway, above, font=\scriptsize, text=black] {next visit};
\end{tikzpicture}
$$

The Adobe team tested this on real campaign data: one bank data set of about
200,000 interactions over 7 possible offers, another of 4,000,000 over 12 offers,
each interaction a customer trajectory with a click labelled reward 1 and no-click
reward 0. Rewards are sparse (users rarely click) and returns are high-variance
(clicking is noisy), which is why both approaches lean on strong function
approximation. The **greedy optimization** policy learned a click-probability map
by a random forest and acted ε-greedily on it. The **life-time value (LTV)**
policy used batch-mode **fitted Q iteration** — value iteration adapted to
Q-learning with a regression approximator, run over the fixed data set rather than
online — warm-started from the greedy map and, because FQI's convergence here is
not monotonic, kept the best iterate selected by _off-policy_ evaluation on a
validation set. Deploying an untested policy on a live marketing platform is
risky, so **high-confidence off-policy evaluation** — estimating a new policy's
value from data logged under other policies, with a probabilistic guarantee of
improvement — was the load-bearing tool, not a footnote.

The two policies split exactly along the metric that names them. Define

$$
\text{CTR} = \frac{\text{total clicks}}{\text{total visits}},
\qquad
\text{LTV} = \frac{\text{total clicks}}{\text{total visitors}},
$$

which differ only in the denominator: CTR counts every _visit_, LTV counts every
_visitor_. When users revisit, LTV exceeds CTR, and the gap measures how well a
policy sustains an extended relationship.

$$
% caption: CTR versus LTV. Each circle is a visit, filled circles are clicks, each
% row is one visitor across repeated visits. Counting visits gives CTR = 7 clicks
% / 20 visits = 0.35; counting visitors gives LTV = 7 clicks / 4 visitors = 1.75.
% CTR rewards immediate clicks; LTV rewards sustained engagement.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  vis/.style={circle, draw, minimum size=3.6mm, inner sep=0pt},
  clk/.style={circle, draw, fill=black, minimum size=3.6mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % row 1: 5 visits, 2 clicks
  \node[font=\scriptsize, anchor=east] at (-0.3,3.0) {user 1};
  \node[clk] at (0.4,3.0) {}; \node[vis] at (1.0,3.0) {}; \node[clk] at (1.6,3.0) {};
  \node[vis] at (2.2,3.0) {}; \node[vis] at (2.8,3.0) {};
  % row 2: 6 visits, 2 clicks
  \node[font=\scriptsize, anchor=east] at (-0.3,2.2) {user 2};
  \node[vis] at (0.4,2.2) {}; \node[clk] at (1.0,2.2) {}; \node[vis] at (1.6,2.2) {};
  \node[vis] at (2.2,2.2) {}; \node[clk] at (2.8,2.2) {}; \node[vis] at (3.4,2.2) {};
  % row 3: 5 visits, 2 clicks
  \node[font=\scriptsize, anchor=east] at (-0.3,1.4) {user 3};
  \node[vis] at (0.4,1.4) {}; \node[vis] at (1.0,1.4) {}; \node[clk] at (1.6,1.4) {};
  \node[clk] at (2.2,1.4) {}; \node[vis] at (2.8,1.4) {};
  % row 4: 4 visits, 1 click
  \node[font=\scriptsize, anchor=east] at (-0.3,0.6) {user 4};
  \node[vis] at (0.4,0.6) {}; \node[clk] at (1.0,0.6) {}; \node[vis] at (1.6,0.6) {};
  \node[vis] at (2.2,0.6) {};
  % legend
  \node[clk] at (4.7,2.6) {};
  \node[anchor=west, font=\scriptsize] at (5.0,2.6) {visit with a click};
  \node[vis] at (4.7,1.9) {};
  \node[anchor=west, font=\scriptsize] at (5.0,1.9) {visit, no click};
\end{tikzpicture}
$$

As designed, greedy optimization won on CTR while LTV optimization won on LTV, and
the high-confidence evaluation certified that LTV would, with high probability,
improve on the policies then in production. Adobe shipped the LTV algorithm into
its Marketing Cloud in 2016. At internet scale a greedy contextual-bandit policy
is a strong baseline, but treating each user as a fresh draw ignores the temporal
structure; an agent that plans over a user's repeated visits captures return the
bandit misses.

## Thermal soaring: riding a rising column of air

The last case leaves the screen entirely. Birds and gliders climb without power by
riding **thermals**, columns of air pushed upward by ground heating; staying inside
one long enough to gain altitude is a real skill, made hard because the updraft is
wrapped in turbulence that jostles the aircraft on every timescale. Reddy, Celani,
Sejnowski, and Vergassola used reinforcement learning to find soaring policies that
work _inside_ that turbulence, aiming both to understand which cues birds must sense
and to inform autonomous gliders.[^sb-soar]

They modeled it as a **continuing, discounted MDP**. A glider flies in a simulated
cubic kilometer of air whose flow is generated by physics-based partial
differential equations over velocity, temperature, and pressure; small random
perturbations make the simulation throw off thermal updrafts and their turbulence.
The glider is steered by two controls, its **bank angle** (roll) and its **angle of
attack** (the angle between wing and airflow), and the flight itself follows
aerodynamic equations for lift, drag, and glide.

$$
% caption: The soaring geometry. The glider banks by angle $\mu$ (roll about the
% velocity direction) and meets the oncoming air at angle of attack $\alpha$; lift
% L acts perpendicular to the wing and drag D along the airflow. Banking turns the
% glider so it can circle to stay inside a rising column; a stall occurs near
% $\alpha \approx 16^\circ$.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ---- left panel: side view (angle of attack) ----
  % velocity axis (horizontal)
  \draw[->, black, thick] (0,0) -- (4.3,0);
  \node[anchor=north, font=\scriptsize] at (3.6,-0.15) {velocity direction};
  % wing direction, above velocity => angle of attack
  \draw[black, thick] (0,0) -- (4.0,1.0);
  \node[anchor=west, font=\scriptsize] at (4.05,1.05) {wing direction};
  \draw[acc, thick] (1.7,0) arc (0:14:1.7);
  \node[acc, font=\scriptsize, anchor=west] at (2.05,0.28) {a (attack)};
  % lift, perpendicular to wing, upward
  \draw[->, red, thick] (2.3,0.575) -- (2.85,1.95) node[anchor=south, font=\scriptsize, text=red] {lift L};
  % drag along airflow (backwards)
  \draw[->, black, thick] (1.2,0) -- (-0.4,0);
  \node[anchor=south east, font=\scriptsize] at (-0.05,0.02) {drag D};
  % ---- right panel: front view (bank angle) ----
  \begin{scope}[xshift=7.4cm, yshift=0.2cm]
    \draw[->, black] (0,0) -- (0,2.0) node[anchor=south, font=\scriptsize] {vertical};
    % rolled wing line
    \draw[acc, thick] (-1.4,0.7) -- (1.4,-0.05);
    \node[acc, font=\scriptsize, anchor=north] at (0.4,-0.35) {rolled wing};
    % bank angle arc between vertical and the wing normal
    \draw[acc, thick] (0,1.5) arc (90:69:1.5);
    \node[acc, font=\scriptsize, anchor=west] at (0.12,1.75) {u (bank)};
  \end{scope}
\end{tikzpicture}
$$

The action set is deliberately tiny. For each of the two controls the agent may
increment, decrement, or hold — bank by 5 degrees, angle of attack by 2.5 degrees —
giving nine combined actions, with the bank angle clamped to plus or minus 15
degrees. Because a stated goal was to find the _minimal_ set of sensory cues that
suffices, the state was built from aggregated environmental signals. They began
with a four-dimensional state — local vertical wind speed, vertical wind
acceleration, a torque from the difference in vertical wind velocity at the two
wingtips, and local temperature — each dimension coarsely
[state-aggregated](/reinforcement-learning/approximation/on-policy-prediction) into
three bins (high positive, high negative, near zero).

### Reward design: what actually taught the glider to climb

The obvious reward failed. Rewarding only the altitude gained at the _end_ of each
episode (with a large penalty for touching the ground) did not produce learning at
realistic episode lengths, and eligibility traces did not help — the signal is
too sparse and delayed. What worked was a **dense, local reward**: at each step,
a linear combination of the vertical wind velocity and the vertical wind
acceleration observed on the previous step. That per-step signal turns "gain
altitude eventually" into a moment-to-moment gradient the agent can follow, and it
is the single design choice that made the problem learnable.

$$
% caption: Reward design for soaring. The sparse end-of-episode altitude reward
% (left) gave no learnable signal at realistic durations; the dense per-step
% reward (right), a linear mix of vertical wind velocity and vertical wind
% acceleration, made learning succeed. The dense reward converts a delayed goal
% into a step-by-step gradient.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=30mm, minimum height=12mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=red, text=red] (sparse) at (0,0) {sparse:\\altitude at\\episode end};
  \node[anchor=north, font=\scriptsize, text=red] at (0,-0.85) {learning fails};
  \node[box, draw=acc, text=acc] (dense) at (5.4,0) {dense:\\wind velocity +\\wind acceleration};
  \node[anchor=north, font=\scriptsize, text=acc] at (5.4,-0.85) {learning succeeds};
  \draw[->, black, thick] (sparse) -- (dense) node[midway, above, font=\scriptsize, text=black] {redesign};
\end{tikzpicture}
$$

Learning was **one-step Sarsa** with a soft-max policy over the action values,
using a temperature that started at 2.0 and was annealed to 0.2 so exploration
gave way to near-greedy control; the step size was 0.1 and the discount 0.98. Each
episode was 2.5 minutes of independently generated turbulent air at a one-second
step, and learning converged within a few hundred episodes.

$$
% caption: Trajectories before and after learning, from the same start (altitude
% scales shifted). Before learning the agent acts randomly and the glider sinks;
% after learning it holds a tight spiral inside the rising column and climbs. The
% number of ground contacts fell to nearly zero as training progressed.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ---- left: before learning, descending zigzag ----
  \draw[black] (0,0) rectangle (3.2,4.2);
  \node[anchor=south, font=\scriptsize] at (1.6,4.25) {before learning};
  \draw[red, thick, ->] (1.6,3.9) -- (0.9,3.2) -- (2.1,2.6) -- (1.0,1.9)
    -- (2.2,1.3) -- (1.1,0.6) -- (1.6,0.2);
  \node[font=\scriptsize, text=red, anchor=west] at (1.7,0.5) {sinks};
  % ---- right: after learning, rising spiral ----
  \draw[black] (4.4,0) rectangle (7.6,4.2);
  \node[anchor=south, font=\scriptsize] at (6.0,4.25) {after learning};
  \draw[acc, thick, ->]
    (6.0,0.3)
    .. controls (7.2,0.7) and (7.2,1.5) .. (6.0,1.5)
    .. controls (4.9,1.5) and (4.9,2.3) .. (6.0,2.5)
    .. controls (7.1,2.7) and (7.1,3.4) .. (6.0,3.5)
    .. controls (5.1,3.6) and (5.1,4.0) .. (6.0,4.0);
  \node[font=\scriptsize, text=acc, anchor=west] at (6.7,2.0) {climbs};
\end{tikzpicture}
$$

The findings are the substance. Sweeping which cues the agent could sense, the best
pair was **vertical wind acceleration and torque** — not the vertical wind velocity
one might expect. The authors' reading: acceleration and torque together report the
_gradient_ of vertical wind velocity in two directions, which is what tells the
controller whether to bank into a turn or hold course to stay inside the updraft;
raw vertical velocity says how strong the thermal is but not how to remain in it.
Temperature helped little, and the angle of attack turned out useful for travelling
_between_ thermals (cross-country and migration), not for staying in one. Policies
learned in strong turbulence were more conservative, preferring small bank angles,
while weak turbulence rewarded banking hard; a common rule across regimes was to
bank sharply toward the higher-lift wing on negative wind acceleration and to hold
still under large positive acceleration with no torque. Finally, raising the
discount rate improved the altitude gained, peaking near $\gamma = 0.99$ —
concrete evidence that good soaring is a genuinely long-horizon control problem,
not a greedy one. The same Sarsa-and-reward frame that scheduled DRAM commands
here teaches a glider to climb.

## The skeleton at industrial scale

Sutton and Barto's case studies stop around 2017. The same skeleton — a value or
policy learned from experience, often self-play, sometimes sharpened by search —
carried the field into domains far harder than board games in the years after. Four
systems, each cited to its published account, show the pattern extended.

**AlphaStar (StarCraft II).** DeepMind's AlphaStar (Vinyals et al., 2019, _Nature_)
reached grandmaster rank — above 99.8% of ranked human players — at real-time strategy,
a game with imperfect information, a huge action space, and horizons of thousands of
steps.[^alphastar] Where backgammon and Go are perfect-information turn games,
StarCraft hides the map under a fog of war and demands actions many times per second.
Plain self-play fails here: it collapses onto a single strategy that beats its own
recent copies but loses to anything different. AlphaStar's fix was a **league** — a population of agents with distinct
objectives, including "exploiters" trained specifically to punish the main agents'
weaknesses — so the main agent had to stay robust against a whole spectrum of
opponents, not just its mirror image. The value function and policy were still deep
networks trained by reinforcement learning (bootstrapped from human replays, then
improved by self-play), the same two pieces TD-Gammon put together, now facing an
opponent population instead of a single copy.

**OpenAI Five (Dota 2).** OpenAI Five (2019, "Dota 2 with Large Scale Deep
Reinforcement Learning") reached professional level at a five-versus-five game with
partial observability and matches lasting 45 minutes at roughly 30 decisions per
second per hero.[^openaifive] It kept the method simple:
[proximal policy optimization](/reinforcement-learning/approximation/policy-gradient-methods),
a single-layer LSTM of 4096 units per hero, and self-play — no search, no human game
data. What it added was **scale**. Training ran on 256 GPUs and 128,000 CPU cores,
consuming the equivalent of about 180 years of Dota per day and running for months,
which forced a new technique the authors called _surgery_: continuing to train the
same network across changes to the model and the game rather than restarting from
scratch. The reward was mostly the win/loss signal, shaped with intermediate terms
(gold, kills, tower damage) to make credit assignment tractable across the long
horizon — the same reward-shaping problem the DRAM controller and the soaring glider
faced, here at a much larger scale.

**Gran Turismo Sophy (sim racing).** Sony AI's GT Sophy (Wurman et al., 2022, _Nature_)
beat the world's best human Gran Turismo drivers in head-to-head racing.[^gtsophy]
Racing is continuous control — steering and throttle are real-valued, not a small
discrete menu — so the discrete-action methods of DQN do not apply directly. Sophy
used a continuous-control, off-policy actor-critic method (a quantile-regression
variant of [soft actor-critic](/reinforcement-learning/approximation/policy-gradient-methods))
that learns a distribution over returns rather than a single mean, trained from
experience gathered on many PlayStation consoles in parallel. Its hardest problem was
not raw speed but **the reward**: encoding racing etiquette — do not punt a rival off
the track, yield when you must — as a reward the agent could optimize without either
driving too timidly or too aggressively. The lesson is the one Samuel's checkers player
first taught: what you reward is what you get, and getting the reward right is most of
the work.

**RLHF and ChatGPT (language).** The most consequential recent transfer of this
machinery is to language. Christiano et al. (2017, "Deep Reinforcement Learning from
Human Preferences") showed that an agent could learn a task from _human comparisons_ of
its behavior instead of a hand-written reward: a **reward model** trained to predict
which of two trajectory segments a human prefers, then optimized by reinforcement
learning, solving Atari and simulated locomotion from feedback on under 1% of the
agent's steps.[^christiano] Ouyang et al. (2022, "Training Language Models to Follow
Instructions with Human Feedback") turned that into the recipe behind InstructGPT and
ChatGPT.[^instructgpt] It has three stages: supervised fine-tuning on human-written
demonstrations; training a reward model on human _rankings_ of the model's outputs; and
optimizing the language model against that reward model with PPO — the language model is
the policy, a token sequence is a trajectory, and the reward model supplies the return.
The headline result was that a 1.3-billion-parameter model tuned this way was preferred
by humans over the 175-billion-parameter GPT-3 that was 100 times larger, evidence that
aligning behavior to a learned reward can matter more than raw scale.

$$
% caption: The reinforcement-learning-from-human-feedback loop behind InstructGPT and
% ChatGPT. Humans rank the model's candidate outputs; a reward model $r$ learns to
% predict those preferences; then the language model (the policy) is optimized by PPO
% to maximize $r$, with a KL penalty keeping it close to the supervised model. The
% same value-and-reward skeleton, now with the reward itself learned from people.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=27mm, minimum height=11mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box] (out)  at (0,0)     {model outputs\\(candidates)};
  \node[box] (rank) at (3.9,0)   {human ranks\\them};
  \node[box, draw=acc, text=acc, thick] (rm) at (7.8,0) {reward model r\\predicts rank};
  \node[box, draw=red, text=red, thick] (pol) at (7.8,-2.4) {language model\\(policy), PPO};
  \draw[->, acc, thick] (out) -- (rank);
  \draw[->, acc, thick] (rank) -- (rm);
  \draw[->, red, thick] (rm) -- (pol) node[midway, right=1mm, font=\scriptsize, text=black] {reward};
  \draw[->, black, thick] (pol.west) .. controls (2.0,-2.4) and (0,-1.4) .. (out.south)
    node[midway, below, font=\scriptsize, text=black] {generates new outputs};
\end{tikzpicture}
$$

Read against Sutton and Barto's studies, these systems change the scale and the domain but not
the shape. AlphaStar and OpenAI Five are TD-Gammon's self-play grown to opponent
populations and industrial compute; GT Sophy is the same actor-critic value learning
pushed into continuous control with a carefully engineered reward; RLHF keeps the
policy-and-reward core but _learns the reward itself_ from human judgment, the missing
piece Samuel had to hand-fix by freezing his piece-advantage weight. The skeleton did
not change; the reward, the approximator, and the compute did.

Samuel tied a value function to itself but forgot to anchor it
to winning; TD-Gammon anchored it, learned it by self-play, and carried it in a
network; DQN made the network deep and the input raw; AlphaGo added learned search;
AlphaGo Zero threw away the human data and let self-play plus search build
everything from the rules alone. The same skeleton, filled in over sixty years,
leads to AlphaZero — which drops even the knowledge of which game it is playing —
and to the systems that scale it to real-time strategy, continuous control, and
rewards learned from human preference.


[^sb-web]: **Sutton & Barto**, §16.7 — Personalized Web Services: recommendation as a contextual bandit (associative search, §2.9) maximizing click-through rate, with A/B testing as its nonassociative two-armed special case; Li et al.'s Yahoo! Front Page study beat a context-free bandit by 12.5%. Theocharous et al. recast it as an MDP maximizing life-time value (clicks over repeated visits), contrasting a greedy random-forest policy against LTV optimization by batch-mode fitted Q iteration, resolved by high-confidence off-policy evaluation; greedy won on CTR, LTV won on the visitor-normalized LTV metric, and Adobe shipped LTV in 2016.
[^sb-soar]: **Sutton & Barto**, §16.8 — Thermal Soaring: Reddy et al. modeled a glider in simulated turbulent air as a continuing discounted MDP, controls incrementing/decrementing bank angle and angle of attack, learned by one-step Sarsa with a soft-max policy (annealed temperature 2.0 to 0.2, $\alpha = 0.1$, $\gamma = 0.98$). A dense per-step reward mixing vertical wind velocity and acceleration succeeded where a sparse end-of-episode altitude reward failed; vertical wind acceleration and wingtip torque were the decisive cues, and altitude gain peaked near $\gamma = 0.99$.
[^alphastar]: **O. Vinyals, I. Babuschkin, W. M. Czarnecki, et al.**, "Grandmaster level in StarCraft II using multi-agent reinforcement learning," _Nature_ 575:350–354 (2019). AlphaStar reached grandmaster rank (above 99.8% of ranked human players) at real-time strategy using deep networks bootstrapped from human replays and improved by reinforcement learning, with a _league_ of diverse agents (main agents plus exploiters) to prevent the strategy collapse that plain self-play suffers in a cyclic game.
[^openaifive]: **OpenAI (C. Berner, G. Brockman, B. Chan, et al.)**, "Dota 2 with Large Scale Deep Reinforcement Learning," arXiv:1912.06680 (2019). OpenAI Five reached professional level at five-versus-five Dota 2 using proximal policy optimization, a per-hero single-layer 4096-unit LSTM, and self-play with no search and no human game data, trained at scale (256 GPUs, ~128,000 CPU cores, roughly 180 years of self-play per day) with reward shaping over the long horizon and a "surgery" technique for continuing training across model and rule changes.
[^gtsophy]: **P. R. Wurman, S. Barrett, K. Kawamoto, et al.**, "Outracing champion Gran Turismo drivers with deep reinforcement learning," _Nature_ 602:223–228 (2022). Gran Turismo Sophy beat top human drivers in head-to-head racing using a model-free, off-policy continuous-control actor-critic (a quantile-regression soft actor-critic) trained across many consoles in parallel, with a reward engineered to encode racing etiquette alongside speed.
[^christiano]: **P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei**, "Deep Reinforcement Learning from Human Preferences," _Advances in Neural Information Processing Systems_ 30 (NeurIPS 2017). Learns a reward model from human pairwise comparisons of trajectory segments, then optimizes it by reinforcement learning, solving Atari and simulated robot locomotion from feedback on under 1% of the agent's interactions — the origin of the reward-model idea behind RLHF.
[^instructgpt]: **L. Ouyang, J. Wu, X. Jiang, et al.**, "Training Language Models to Follow Instructions with Human Feedback," _Advances in Neural Information Processing Systems_ 35 (NeurIPS 2022). The InstructGPT recipe behind ChatGPT: supervised fine-tuning, then a reward model trained on human rankings of model outputs, then optimization of the language model (the policy) against that reward model with PPO and a KL penalty; the 1.3-billion-parameter RLHF model was preferred by human evaluators over the 175-billion-parameter GPT-3.