Hierarchical RL: Options and the Option-Critic

A deep RL agent chooses one primitive action per step: press a button, apply a torque, move one cell. Over a short horizon that is enough. Over a long one it fails. A task that needs five hundred correct actions in sequence before any reward arrives presents two compounded difficulties at once, and flat RL fails on both.

The first is exploration. To stumble onto the first reward by chance, an undirected policy must produce a specific long run of correct actions; the probability of that decays exponentially in the run's length, so the agent may never see a single nonzero reward to learn from. The second is credit assignment. Even granted a reward at the end, a one-step temporal-difference update propagates that signal backward one transition per visit, so information about a decision five hundred steps early takes hundreds of updates to reach it. Both problems are problems of horizon: they are severe precisely because the number of primitive steps between decision and consequence is large.

Humans do not plan at the level of primitive muscle twitches. Asked to make coffee, you decompose: walk to the kitchen, grind the beans, boil water, pour. Each subgoal is itself a small task that bottoms out in motor actions, but you reason about the sequence of subgoals, not the sequence of twitches. The subgoals are temporally extended — each spans many primitive steps — and choosing among four of them is a far shorter, far easier problem than choosing among thousands of raw action sequences. That is the whole idea of hierarchical reinforcement learning: turn one long-horizon problem into a shallow tree of short ones.

The arithmetic of the horizon

Both difficulties are quantifiable, and the numbers show that hierarchy changes the complexity class of the problem. Take a task where $k=4$ actions are available at each step and reward arrives only after a specific run of $H=500$ correct actions. An undirected policy exploring uniformly at random produces that exact run with probability

so it would never, in any practical number of episodes, see a single reward — there is nothing to learn from. Now introduce a manager that picks among $m=4$ subgoals every $c=50$ steps, where each subgoal is a short reaching task the worker can solve. The manager's horizon collapses to $H/c=10$ decisions, and if it chooses subgoals uniformly the probability of the right sequence of ten is

a $10^{295}$-fold improvement; a few million episodes will hit an event of that probability. Credit assignment improves by the same factor: a one-step backup must propagate $500$ times to reach the first decision, but a manager backup propagates only $10$ times, so the reward signal reaches the earliest choice fifty times faster. The hierarchy changes the exponent, turning $k^H$ into $m^{H/c}$. Every method in this lesson is a way to supply that middle level.

The deep RL frontiers lesson named the central abstraction — the option — and sketched its semi-Markov view. This lesson treats it fully: the three parts of an option, the semi-Markov decision process they generate, and how to learn options end to end rather than hand-design them. The goal-conditioned manager/worker architectures that dominate deep hierarchical RL today, hindsight relabeling for sparse goals, and unsupervised skill discovery continue in a companion lesson. The tabular groundwork for planning with temporally extended actions is in planning and learning; here we carry it into the function-approximation setting.

The options framework

Sutton, Precup, and Singh (1999) formalized temporal abstraction with the option.¹ The frontiers lesson introduced an option as a policy paired with a termination rule; the full definition adds a third part that says where the option is allowed to start.

Executing an option is mechanical. When the agent is in a state $s\in {\mathcal{I}}_{\omega }$ and selects $\omega$, control passes to ${\pi }_{\omega }$: at each step it draws an action $a\sim {\pi }_{\omega }(\cdot \mid s)$, the environment transitions, and then with probability ${\beta }_{\omega }(s^{\prime })$ the option terminates and returns control to the level above; otherwise it continues. The three parts specify when the option may start, what it does while running, and when it stops — nothing more.

The semi-Markov view

An option runs for a random number of primitive steps. That single fact takes us out of the Markov decision process, where every transition consumes exactly one step, and into a semi-Markov decision process (SMDP), where a transition may consume a variable amount of time. If we treat each option as a single temporally-extended action, the top-level process over options is an SMDP: from a state $s$, choosing option $\omega$ lands the agent in a state $s^{\prime }$ after $\tau$ primitive steps, where $\tau$ is itself random and governed by ${\beta }_{\omega }$.

The SMDP view is what lets every result we already have for actions carry over to options with almost no change: the top-level agent sees each option as one action whose reward is the accumulated discounted reward earned while it ran, and whose transition skips to wherever the option terminated. The discount comes out of the run as ${\gamma }^{\tau }$, so an option that takes many steps is discounted more heavily — the same total-return arithmetic, applied to variable-length jumps.

Option models and intra-option learning

To plan over options, the agent needs a model of each one, exactly as planning over primitive actions needs a model of the environment. A primitive model gives, for an action, the expected immediate reward and the next-state distribution. An option model gives, for an option launched in state $s$, two things: the expected accumulated discounted reward earned until termination,

and a discounted distribution over the state where it lands and the time it takes,

The discount folded into $p_{\omega }$ is the subtle part: because the option can end after any number of steps, the next state is really a bundle of possible landing states each weighted by how long the option ran to get there. With these two quantities, the Bellman equations and value iteration go through unchanged over an augmented action set of options plus primitives, and planning is faster because each option covers many steps in a single backup — the payoff of temporal abstraction.¹

There are two ways to learn about options. The naive one is SMDP learning: run an option to termination, treat the whole extended run as one transition, and apply a single update that jumps from where the option started to where it ended. The SMDP Q-learning update is the ordinary one with the single-step discount replaced by ${\gamma }^{\tau }$ and the single reward replaced by the accumulated discounted return over the option's $\tau$ steps:

A worked SMDP backup. Suppose an option runs for $\tau =3$ steps, collecting rewards $r_{t+1}=0$, $r_{t+2}=0$, $r_{t+3}=1$, with $\gamma =0.9$, and lands in a state whose best option value is ${\max }_{{\omega }^{\prime }}Q(s_{t+3},{\omega }^{\prime })=5$. The accumulated return is $R=0+0.9\cdot 0+0.9^2\cdot 1=0.81$, the discount on the bootstrap is ${\gamma }^{\tau }=0.9^3=0.729$, and the target is $R+{\gamma }^{\tau }\cdot 5=0.81+0.729\cdot 5=4.455$. One backup carried the reward across three primitive steps in a single update — where flat Q-learning would need three separate one-step backups, and three separate visits, to move the same information. That leap is the planning payoff of temporal abstraction, in one line of arithmetic.

SMDP learning works but wastes data — it ignores every intermediate state the option passed through. Intra-option learning does better: it updates the option's value (and those of other options consistent with the actions taken) at every primitive step, using the one-step data as it arrives. Intra-option methods learn about an option without ever running it to completion, and they can update many options from a single stream of experience, which is why they are the standard tabular approach to option learning.¹

Learning options end to end

The classical options framework assumes the options are given — a designer supplies the initiation sets, intra-option policies, and termination conditions, and the agent only learns to choose among them and to plan. That is a heavy assumption. Hand-designing good options requires knowing the useful subgoals in advance, which for a hard task is most of the problem. The modern question is whether the options themselves can be discovered by gradient descent, from the task reward alone, with no subgoal ever specified by hand.

The option-critic architecture of Bacon, Harb, and Precup (2017) answers yes.² It parameterizes the intra-option policies and the termination conditions and differentiates the expected return with respect to those parameters, giving policy-gradient-style updates for both. Write the intra-option policy of option $\omega$ as ${\pi }_{\omega ,\theta }(a\mid s)$ and its termination as ${\beta }_{\omega ,\vartheta }(s)$, with a fixed policy-over-options ${\pi }_{\Omega }$ choosing which option to run. Two theorems — the intra-option policy gradient and the termination gradient — give the derivative of the return with respect to $\theta$ and $\vartheta$, learned end to end alongside a critic that estimates option values $Q_{\Omega }(s,\omega )$.

The intra-option policy gradient is the familiar actor-critic update, but with the option-value $Q_{\Omega }$ as the critic:

where $Q_U(s,\omega ,a)$ is the value of taking action $a$ inside option $\omega$. The termination gradient is the informative one. It moves each ${\beta }_{\omega }$ in the direction that shortens or extends the option based on an advantage: an option should be more likely to terminate in a state where continuing it is worse than switching,

When the advantage $A_{\Omega }(s^{\prime },\omega )$ is negative — the current option is worse than the best available — the gradient raises ${\beta }_{\omega }$, making the option end so a better one can start. When the advantage is positive, it lowers${\beta }_{\omega }$, letting a good option keep running. Terminations are learned as a cost-benefit decision rather than scripted.

Option-critic learns a whole hierarchy from the single task reward, which is its appeal and its weakness. Nothing pressures the discovered options to be diverse or meaningful: a common failure is option collapse, where one option monopolizes behavior and the terminations fire every step, degenerating back to flat RL, or where the options carve the state space into pieces that reduce return but carry no interpretable meaning. Later work adds regularizers (deliberation cost, diversity terms) to keep the options distinct, but the bare architecture already shows that temporal abstraction can be learned rather than designed.

Where this leaves us

The options framework formalizes temporal abstraction. An option is a triple — an initiation set, an internal policy, and a termination condition — that packages a temporally extended behavior into something the top level can select as if it were a single action. Options turn an MDP into a semi-Markov decision process over variable-duration decisions, and this shortens the effective horizon: the top level makes far fewer, far more consequential choices, so both exploration and credit assignment improve.

The option-critic then removes the need to hand-design options at all, learning the intra-option policies and the termination conditions end to end by differentiating the option-value objective — at the cost of the collapse pathologies (options that terminate every step, or one option that swallows the task) that need regularization to prevent.

Options let the agent commit to a sub-behavior, but they do not specify what a good sub-behavior is. A complementary family gives the hierarchy an explicit language of goals: a manager proposes a target state (or a direction in a latent space) and a worker is rewarded for reaching it. That goal-conditioned view, hindsight relabeling for learning from sparse reward, and the unsupervised discovery of skills with no reward at all continue in Hierarchical RL: Goal-Conditioned Hierarchies and Skills.