Reinforcement Learning Beyond Games

This builds on Case Studies: Learning to Play, which traced one skeleton — a value function learned by self-play, refined by search, carried by a deep network — from Samuel's checkers player through TD-Gammon, DQN, and AlphaGo Zero. Those systems played games. The cases here point the same machinery at problems with no opponent, and then follow the skeleton to industrial scale.

Personalized web services: bandits at internet scale

The systems above played games. The next two show the same value-and-reward machinery pointed at problems with no opponent — first at web-scale content selection, where the game is a stream of anonymous visits and the reward is a click.¹ Personalizing which news story or advertisement a page shows is a recommendation problem, and it maps cleanly onto the multi-armed bandit foundations. The plain bandit is nonassociative: one situation, a fixed set of arms, and the task is to find the single best arm. Marketing's long-standing A/B test — show version A or version B and keep the one users prefer — is exactly a two-armed bandit run on the whole population at once, and because it is nonassociative it cannot personalize.

Adding context is what personalizes. Attach to each decision a feature vector describing the user (time since last visit, visit count, location, inferred interests, demographics) and the candidate content, and the problem becomes the contextual bandit of the bandits lesson: on each round the learner sees a context, picks an arm, and collects a reward, and the best arm now depends on the context.¹ Li, Chu, Langford, and Schapire applied a contextual-bandit algorithm to choosing the featured story on the Yahoo! Front Page, maximizing the click-through rate — clicks divided by visits — and beat a context-free bandit by 12.5%. The lift is entirely the value of conditioning on who is looking.

Greedy versus long-term: the exploration the scale hides

The contextual bandit has a quiet defect at scale. It is greedy across time: it maximizes the probability of an immediate click and treats every visit as if a fresh, anonymous visitor drew it from the population. But many users return, and a policy that ignores that throws away the whole point of sequential interaction. Theocharous, Thomas, and Ghavamzadeh reframed personalization as a full MDP whose reward is the total clicks a user makes over repeated visits, so the state carries the history of one identified user across sessions rather than resetting each time.¹

Their running example is a car ad. A greedy policy offers the closing discount on the first visit; the user either bites or leaves, and on any later visit sees the same offer again. A longer-term policy walks the user down a sales funnel — mention favorable financing, praise the service department, and only on a later visit present the discount — trading a lower immediate click probability for more clicks, and more eventual sales, across the relationship. This is the multi-armed bandit's exploration–exploitation tradeoff lifted from single actions to whole sequences of offers: acting greedily on each visit is a local optimum that an agent planning over the user's lifetime can beat.

The Adobe team tested this on real campaign data: one bank data set of about 200,000 interactions over 7 possible offers, another of 4,000,000 over 12 offers, each interaction a customer trajectory with a click labelled reward 1 and no-click reward 0. Rewards are sparse (users rarely click) and returns are high-variance (clicking is noisy), which is why both approaches lean on strong function approximation. The greedy optimization policy learned a click-probability map by a random forest and acted ε-greedily on it. The life-time value (LTV) policy used batch-mode fitted Q iteration — value iteration adapted to Q-learning with a regression approximator, run over the fixed data set rather than online — warm-started from the greedy map and, because FQI's convergence here is not monotonic, kept the best iterate selected by off-policy evaluation on a validation set. Deploying an untested policy on a live marketing platform is risky, so high-confidence off-policy evaluation — estimating a new policy's value from data logged under other policies, with a probabilistic guarantee of improvement — was the load-bearing tool, not a footnote.

The two policies split exactly along the metric that names them. Define

which differ only in the denominator: CTR counts every visit, LTV counts every visitor. When users revisit, LTV exceeds CTR, and the gap measures how well a policy sustains an extended relationship.

As designed, greedy optimization won on CTR while LTV optimization won on LTV, and the high-confidence evaluation certified that LTV would, with high probability, improve on the policies then in production. Adobe shipped the LTV algorithm into its Marketing Cloud in 2016. At internet scale a greedy contextual-bandit policy is a strong baseline, but treating each user as a fresh draw ignores the temporal structure; an agent that plans over a user's repeated visits captures return the bandit misses.

Thermal soaring: riding a rising column of air

The last case leaves the screen entirely. Birds and gliders climb without power by riding thermals, columns of air pushed upward by ground heating; staying inside one long enough to gain altitude is a real skill, made hard because the updraft is wrapped in turbulence that jostles the aircraft on every timescale. Reddy, Celani, Sejnowski, and Vergassola used reinforcement learning to find soaring policies that work inside that turbulence, aiming both to understand which cues birds must sense and to inform autonomous gliders.²

They modeled it as a continuing, discounted MDP. A glider flies in a simulated cubic kilometer of air whose flow is generated by physics-based partial differential equations over velocity, temperature, and pressure; small random perturbations make the simulation throw off thermal updrafts and their turbulence. The glider is steered by two controls, its bank angle (roll) and its angle of attack (the angle between wing and airflow), and the flight itself follows aerodynamic equations for lift, drag, and glide.

The action set is deliberately tiny. For each of the two controls the agent may increment, decrement, or hold — bank by 5 degrees, angle of attack by 2.5 degrees — giving nine combined actions, with the bank angle clamped to plus or minus 15 degrees. Because a stated goal was to find the minimal set of sensory cues that suffices, the state was built from aggregated environmental signals. They began with a four-dimensional state — local vertical wind speed, vertical wind acceleration, a torque from the difference in vertical wind velocity at the two wingtips, and local temperature — each dimension coarsely state-aggregated into three bins (high positive, high negative, near zero).

Reward design: what actually taught the glider to climb

The obvious reward failed. Rewarding only the altitude gained at the end of each episode (with a large penalty for touching the ground) did not produce learning at realistic episode lengths, and eligibility traces did not help — the signal is too sparse and delayed. What worked was a dense, local reward: at each step, a linear combination of the vertical wind velocity and the vertical wind acceleration observed on the previous step. That per-step signal turns gain altitude eventually into a moment-to-moment gradient the agent can follow, and it is the single design choice that made the problem learnable.

Learning was one-step Sarsa with a soft-max policy over the action values, using a temperature that started at 2.0 and was annealed to 0.2 so exploration gave way to near-greedy control; the step size was 0.1 and the discount 0.98. Each episode was 2.5 minutes of independently generated turbulent air at a one-second step, and learning converged within a few hundred episodes.

The findings are the substance. Sweeping which cues the agent could sense, the best pair was vertical wind acceleration and torque — not the vertical wind velocity one might expect. The authors' reading: acceleration and torque together report the gradient of vertical wind velocity in two directions, which is what tells the controller whether to bank into a turn or hold course to stay inside the updraft; raw vertical velocity says how strong the thermal is but not how to remain in it. Temperature helped little, and the angle of attack turned out useful for travelling between thermals (cross-country and migration), not for staying in one. Policies learned in strong turbulence were more conservative, preferring small bank angles, while weak turbulence rewarded banking hard; a common rule across regimes was to bank sharply toward the higher-lift wing on negative wind acceleration and to hold still under large positive acceleration with no torque. Finally, raising the discount rate improved the altitude gained, peaking near $\gamma =0.99$ — concrete evidence that good soaring is a genuinely long-horizon control problem, not a greedy one. The same Sarsa-and-reward frame that scheduled DRAM commands here teaches a glider to climb.

The skeleton at industrial scale

Sutton and Barto's case studies stop around 2017. The same skeleton — a value or policy learned from experience, often self-play, sometimes sharpened by search — carried the field into domains far harder than board games in the years after. Four systems, each cited to its published account, show the pattern extended.

AlphaStar (StarCraft II). DeepMind's AlphaStar (Vinyals et al., 2019, Nature) reached grandmaster rank — above 99.8% of ranked human players — at real-time strategy, a game with imperfect information, a huge action space, and horizons of thousands of steps.³ Where backgammon and Go are perfect-information turn games, StarCraft hides the map under a fog of war and demands actions many times per second. Plain self-play fails here: it collapses onto a single strategy that beats its own recent copies but loses to anything different. AlphaStar's fix was a league — a population of agents with distinct objectives, including exploiters trained specifically to punish the main agents' weaknesses — so the main agent had to stay robust against a whole spectrum of opponents, not just its mirror image. The value function and policy were still deep networks trained by reinforcement learning (bootstrapped from human replays, then improved by self-play), the same two pieces TD-Gammon put together, now facing an opponent population instead of a single copy.

OpenAI Five (Dota 2). OpenAI Five (2019, Dota 2 with Large Scale Deep Reinforcement Learning) reached professional level at a five-versus-five game with partial observability and matches lasting 45 minutes at roughly 30 decisions per second per hero.⁴ It kept the method simple: proximal policy optimization, a single-layer LSTM of 4096 units per hero, and self-play — no search, no human game data. What it added was scale. Training ran on 256 GPUs and 128,000 CPU cores, consuming the equivalent of about 180 years of Dota per day and running for months, which forced a new technique the authors called surgery: continuing to train the same network across changes to the model and the game rather than restarting from scratch. The reward was mostly the win/loss signal, shaped with intermediate terms (gold, kills, tower damage) to make credit assignment tractable across the long horizon — the same reward-shaping problem the DRAM controller and the soaring glider faced, here at a much larger scale.

Gran Turismo Sophy (sim racing). Sony AI's GT Sophy (Wurman et al., 2022, Nature) beat the world's best human Gran Turismo drivers in head-to-head racing.⁵ Racing is continuous control — steering and throttle are real-valued, not a small discrete menu — so the discrete-action methods of DQN do not apply directly. Sophy used a continuous-control, off-policy actor-critic method (a quantile-regression variant of soft actor-critic) that learns a distribution over returns rather than a single mean, trained from experience gathered on many PlayStation consoles in parallel. Its hardest problem was not raw speed but the reward: encoding racing etiquette — do not punt a rival off the track, yield when you must — as a reward the agent could optimize without either driving too timidly or too aggressively. The lesson is the one Samuel's checkers player first taught: what you reward is what you get, and getting the reward right is most of the work.

RLHF and ChatGPT (language). The most consequential recent transfer of this machinery is to language. Christiano et al. (2017, Deep Reinforcement Learning from Human Preferences) showed that an agent could learn a task from human comparisons of its behavior instead of a hand-written reward: a reward model trained to predict which of two trajectory segments a human prefers, then optimized by reinforcement learning, solving Atari and simulated locomotion from feedback on under 1% of the agent's steps.⁶ Ouyang et al. (2022, Training Language Models to Follow Instructions with Human Feedback) turned that into the recipe behind InstructGPT and ChatGPT.⁷ It has three stages: supervised fine-tuning on human-written demonstrations; training a reward model on human rankings of the model's outputs; and optimizing the language model against that reward model with PPO — the language model is the policy, a token sequence is a trajectory, and the reward model supplies the return. The headline result was that a 1.3-billion-parameter model tuned this way was preferred by humans over the 175-billion-parameter GPT-3 that was 100 times larger, evidence that aligning behavior to a learned reward can matter more than raw scale.

Read against Sutton and Barto's studies, these systems change the scale and the domain but not the shape. AlphaStar and OpenAI Five are TD-Gammon's self-play grown to opponent populations and industrial compute; GT Sophy is the same actor-critic value learning pushed into continuous control with a carefully engineered reward; RLHF keeps the policy-and-reward core but learns the reward itself from human judgment, the missing piece Samuel had to hand-fix by freezing his piece-advantage weight. The skeleton did not change; the reward, the approximator, and the compute did.

Samuel tied a value function to itself but forgot to anchor it to winning; TD-Gammon anchored it, learned it by self-play, and carried it in a network; DQN made the network deep and the input raw; AlphaGo added learned search; AlphaGo Zero threw away the human data and let self-play plus search build everything from the rules alone. The same skeleton, filled in over sixty years, leads to AlphaZero — which drops even the knowledge of which game it is playing — and to the systems that scale it to real-time strategy, continuous control, and rewards learned from human preference.