---
title: "Hierarchical RL: Options and the Option-Critic"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 15
order: 515
summary: >
  Flat RL cannot explore a long horizon: reaching reward through hundreds of
  primitive actions is exponentially unlikely, and every credit-assignment update
  crawls one step at a time. Hierarchy breaks one hard long-horizon problem into
  many short ones. This lesson develops temporal abstraction — the options framework
  and its semi-Markov view, and learning options end to end with the option-critic. A
  companion lesson takes up goal-conditioned manager/worker hierarchies (FeUdal
  Networks and HIRO), hindsight relabeling, and unsupervised skill discovery.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "Ch. 17 — Frontiers; §17.2 Temporal Abstraction via Options"
  - book: Grokking Deep RL
    ref: "Ch. 12 — Advanced actor-critic methods; hierarchy and abstraction"
---

A [deep RL agent](/reinforcement-learning/deep-rl/deep-q-networks) chooses one
primitive action per step: press a button, apply a torque, move one cell. Over a
short horizon that is enough. Over a long one it fails. A task that needs
five hundred correct actions in sequence before any reward arrives presents two
compounded difficulties at once, and flat RL fails on both.

The first is **exploration**. To stumble onto the first reward by chance, an
undirected policy must produce a specific long run of correct actions; the
probability of that decays exponentially in the run's length, so the agent may
never see a single nonzero reward to learn from. The second is **credit
assignment**. Even granted a reward at the end, a one-step temporal-difference
update propagates that signal backward one transition per visit, so information
about a decision five hundred steps early takes hundreds of updates to reach it. Both problems are problems of _horizon_: they are severe precisely because
the number of primitive steps between decision and consequence is large.

Humans do not plan at the level of primitive muscle twitches. Asked to make
coffee, you decompose: _walk to the kitchen_, _grind the beans_, _boil water_,
_pour_. Each subgoal is itself a small task that bottoms out in motor actions, but
you reason about the sequence of subgoals, not the sequence of twitches. The
subgoals are **temporally extended** — each spans many primitive steps — and
choosing among four of them is a far shorter, far easier problem than choosing
among thousands of raw action sequences. That is the whole idea of hierarchical
reinforcement learning: **turn one long-horizon problem into a shallow tree of
short ones.**

$$
% caption: A long-horizon task decomposed into subtasks. At the top level the
% agent chooses among a handful of temporally-extended subtasks (walk, grind,
% boil, pour); each subtask expands into many primitive steps below. Choosing
% among four subgoals is a short-horizon problem even though the flat task spans
% hundreds of steps.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  top/.style={draw, minimum width=20mm, minimum height=10mm, align=center, font=\scriptsize},
  prim/.style={circle, draw, minimum size=4.5mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \node[top, draw=acc, text=acc, thick] (task) at (5.2,2.6) {make co{f}{f}ee\\(long task)};
  \node[top] (t1) at (0,0.7)  {walk};
  \node[top] (t2) at (3.5,0.7) {grind};
  \node[top] (t3) at (7.0,0.7) {boil};
  \node[top] (t4) at (10.4,0.7) {pour};
  \draw[->, acc, thick] (task) -- (t1);
  \draw[->, acc, thick] (task) -- (t2);
  \draw[->, acc, thick] (task) -- (t3);
  \draw[->, acc, thick] (task) -- (t4);
  % primitive steps under one subtask
  \foreach \i in {0,1,2,3,4} \node[prim] (p\i) at (2.0+\i*0.62,-1.3) {};
  \draw[->, black] (t2) -- (p0);
  \foreach \i/\j in {0/1,1/2,2/3,3/4} \draw[->, black] (p\i) -- (p\j);
  \node[font=\scriptsize, text=black, anchor=north] at (3.2,-1.75) {primitive actions};
\end{tikzpicture}
$$

### The arithmetic of the horizon

Both difficulties are quantifiable, and the numbers show that hierarchy changes
the complexity class of the problem. Take a task where
$k = 4$ actions are available at each step and reward arrives only after a specific
run of $H = 500$ correct actions. An undirected policy exploring uniformly at random
produces that exact run with probability

$$
\Bigl(\tfrac{1}{k}\Bigr)^{H} = \Bigl(\tfrac{1}{4}\Bigr)^{500} \approx 10^{-301},
$$

so it would never, in any practical number of episodes, see a single reward — there
is nothing to learn from. Now introduce a manager that picks among $m = 4$ subgoals
every $c = 50$ steps, where each subgoal is a short reaching task the worker can
solve. The manager's horizon collapses to $H/c = 10$ decisions, and if it chooses
subgoals uniformly the probability of the right sequence of ten is

$$
\Bigl(\tfrac{1}{m}\Bigr)^{H/c} = \Bigl(\tfrac{1}{4}\Bigr)^{10} \approx 10^{-6},
$$

a $10^{295}$-fold improvement; a few million episodes will hit an event of that
probability. Credit assignment improves by the same factor: a
one-step backup must propagate $500$ times to reach the first decision, but a
manager backup propagates only $10$ times, so the reward signal reaches the earliest
choice fifty times faster. The hierarchy changes the exponent, turning $k^{H}$
into $m^{H/c}$. Every method in this lesson is a way to supply that middle level.

$$
% caption: The arithmetic of hierarchy. The probability of stumbling onto a reward
% by chance decays exponentially in the number of decisions before it. A flat agent
% faces H = 500 primitive decisions (curve plunges off the chart); a manager facing
% H/c = 10 subgoal decisions stays in a reachable range. Hierarchy shrinks the
% exponent, not the base.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (8.6,0) node[anchor=north, black, font=\footnotesize] {number of decisions};
  \draw[->, black] (0,-0.2) -- (0,4.0) node[anchor=east, black, font=\footnotesize] {log chance of reward};
  % flat: steep plunge
  \draw[red, very thick] plot coordinates {(0.3,3.6)(1.0,2.4)(1.8,1.2)(2.6,0.4)(3.4,0.12)};
  \node[red, anchor=west, font=\scriptsize] at (2.3,1.6) {f\/lat: 500 decisions};
  % hierarchical: gentle
  \draw[acc, very thick] plot coordinates {(0.3,3.7)(2.0,3.4)(4.0,3.0)(6.0,2.6)(8.0,2.2)};
  \node[acc, anchor=south, font=\scriptsize] at (4.6,3.15) {manager: 10 decisions};
\end{tikzpicture}
$$

The [deep RL frontiers](/reinforcement-learning/deep-rl/frontiers) lesson named
the central abstraction — the _option_ — and sketched its semi-Markov view. This
lesson treats it fully: the three parts of an option, the semi-Markov decision
process they generate, and how to learn options end to end rather than hand-design
them. The goal-conditioned manager/worker architectures that dominate deep
hierarchical RL today, hindsight relabeling for sparse goals, and unsupervised skill
discovery continue in a
[companion lesson](/reinforcement-learning/modern-deep-rl/hierarchical-rl-part-2). The
tabular groundwork for planning with temporally extended actions is in
[planning and learning](/reinforcement-learning/tabular-methods/planning-and-learning);
here we carry it into the function-approximation setting.

## The options framework

Sutton, Precup, and Singh (1999) formalized temporal abstraction with the
**option**.[^spc] The frontiers lesson introduced an option as a policy paired
with a termination rule; the full definition adds a third part that says _where_
the option is allowed to start.

> **Definition (Option).** An option is a triple
> $\omega = (\mathcal{I}_\omega, \pi_\omega, \beta_\omega)$: an **initiation set**
> $\mathcal{I}_\omega \subseteq \mathcal{S}$ of states from which the option may be
> invoked; an **intra-option policy** $\pi_\omega(a \mid s)$ that selects primitive
> actions while the option runs; and a **termination condition**
> $\beta_\omega(s) \in [0,1]$ giving the probability the option ends on reaching
> state $s$. A primitive action $a$ is the special case
> $\mathcal{I}_a = \{s : a \text{ available}\}$, $\pi_a$ deterministic on $a$,
> $\beta_a \equiv 1$ — an option that runs for exactly one step.

Executing an option is mechanical. When the agent is in a state
$s \in \mathcal{I}_\omega$ and selects $\omega$, control passes to $\pi_\omega$:
at each step it draws an action $a \sim \pi_\omega(\cdot \mid s)$, the environment
transitions, and then with probability $\beta_\omega(s')$ the option terminates
and returns control to the level above; otherwise it continues. The three parts
specify when the option may start, what it does while running, and when it stops —
nothing more.

$$
% caption: An option $\omega = (\mathcal{I}_\omega, \pi_\omega, \beta_\omega)$
% executing. It may start only from a state in the initiation set
% $\mathcal{I}_\omega$; while running, $\pi_\omega$ picks primitive actions; after
% each step it terminates with probability $\beta_\omega$. Here it initiates at
% $s_t$ and terminates at $s_{t+4}$ where $\beta_\omega = 1$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  st/.style={circle, draw, minimum size=8mm, inner sep=0pt, font=\scriptsize},
  term/.style={circle, draw=acc, thick, minimum size=8mm, inner sep=0pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[st]   (s0) at (0,0)   {s(t)};
  \node[st]   (s1) at (2.3,0) {s+1};
  \node[st]   (s2) at (4.6,0) {s+2};
  \node[st]   (s3) at (6.9,0) {s+3};
  \node[term] (s4) at (9.2,0) {s+4};
  \draw[->, thick] (s0) -- node[midway, above, font=\scriptsize] {a(t)} (s1);
  \draw[->, thick] (s1) -- node[midway, above, font=\scriptsize] {a+1} (s2);
  \draw[->, thick] (s2) -- node[midway, above, font=\scriptsize] {a+2} (s3);
  \draw[->, thick] (s3) -- node[midway, above, font=\scriptsize] {a+3} (s4);
  \node[font=\scriptsize, text=black, anchor=north] at (0,-0.7) {in init set: start};
  \node[font=\scriptsize, text=black, anchor=south] at (3.45,0.55) {policy picks actions};
  \node[acc, font=\scriptsize, anchor=north] at (9.2,-0.7) {beta = 1: stop};
\end{tikzpicture}
$$

### The semi-Markov view

An option runs for a _random_ number of primitive steps. That single fact takes
us out of the Markov decision process, where every transition consumes exactly one
step, and into a **semi-Markov decision process (SMDP)**, where a transition may
consume a variable amount of time. If we treat each option as a single
temporally-extended action, the top-level process over options is an SMDP: from a
state $s$, choosing option $\omega$ lands the agent in a state $s'$ after $\tau$
primitive steps, where $\tau$ is itself random and governed by $\beta_\omega$.

> **Definition (Semi-Markov decision process).** A decision process in which the
> time between decisions is a random variable rather than a fixed unit step. Over
> options, the transition from $s$ under $\omega$ is jointly a distribution over
> the next decision state $s'$ and the elapsed duration $\tau$, and the discount
> applied to the option's outcome is $\gamma^\tau$ rather than a single $\gamma$.
> The MDP is the special case $\tau \equiv 1$.

The SMDP view is what lets every result we already have for actions carry over to
options with almost no change: the top-level agent sees each option as one
"action" whose reward is the accumulated discounted reward earned while it ran,
and whose transition skips to wherever the option terminated. The discount comes
out of the run as $\gamma^\tau$, so an option that takes many steps is discounted
more heavily — the same total-return arithmetic, applied to variable-length jumps.

$$
% caption: The semi-Markov view. Below, the primitive MDP takes single steps.
% Above, each option is one SMDP transition that skips from a decision state to
% the state where the option terminated, spanning $\tau$ primitive steps and
% carrying the accumulated reward discounted by $\gamma^\tau$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  st/.style={circle, draw, minimum size=6mm, inner sep=0pt, font=\scriptsize},
  dec/.style={circle, draw=acc, thick, minimum size=7mm, inner sep=0pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  % primitive chain (bottom)
  \foreach \i in {0,1,2,3,4,5,6} \node[st] (p\i) at (\i*1.55,0) {};
  \foreach \i/\j in {0/1,1/2,2/3,3/4,4/5,5/6} \draw[->, black] (p\i) -- (p\j);
  \node[font=\scriptsize, text=black, anchor=north] at (4.65,-0.55) {primitive MDP steps};
  % decision states (bottom, highlighted)
  \node[dec] at (0,0) {s};
  \node[dec] at (4.65,0) {s'};
  \node[dec] at (9.3,0) {s''};
  % SMDP jumps (top)
  \draw[->, acc, thick] (0,1.7) to[bend left=22] node[midway, above, font=\scriptsize] {option w1 (tau = 3)} (4.65,1.7);
  \draw[->, acc, thick] (4.65,1.7) to[bend left=22] node[midway, above, font=\scriptsize] {option w2 (tau = 3)} (9.3,1.7);
  \draw[acc, densely dotted] (0,0.35) -- (0,1.6);
  \draw[acc, densely dotted] (4.65,0.35) -- (4.65,1.6);
  \draw[acc, densely dotted] (9.3,0.35) -- (9.3,1.6);
\end{tikzpicture}
$$

### Option models and intra-option learning

To _plan_ over options, the agent needs a **model** of each one, exactly as
planning over primitive actions needs a model of the environment. A primitive
model gives, for an action, the expected immediate reward and the next-state
distribution. An **option model** gives, for an option launched in state $s$, two
things: the expected accumulated discounted reward earned until termination,
$$
r_\omega(s) \;=\; \mathbb{E}\!\left[\, r_{t+1} + \gamma\, r_{t+2} + \cdots + \gamma^{\tau-1}\, r_{t+\tau} \;\middle|\; s_t = s,\ \omega \,\right],
$$
and a discounted distribution over the state where it lands and the time it takes,
$$
p_\omega(s' \mid s) \;=\; \mathbb{E}\!\left[\, \gamma^{\tau}\, \mathbb{1}\{s_{t+\tau} = s'\} \;\middle|\; s_t = s,\ \omega \,\right].
$$
The discount folded into $p_\omega$ is the subtle part: because the option can end
after any number of steps, the "next state" is really a bundle of possible landing
states each weighted by how long the option ran to get there. With these two
quantities, the Bellman equations and value iteration go through unchanged over an
augmented action set of options plus primitives, and planning is faster because
each option covers many steps in a single backup — the payoff of temporal
abstraction.[^spc]

There are two ways to learn about options. The naive one is **SMDP learning**: run
an option to termination, treat the whole extended run as one transition, and
apply a single update that jumps from where the option started to where it ended.
The SMDP Q-learning update is the ordinary one with the single-step discount
replaced by $\gamma^\tau$ and the single reward replaced by the accumulated
discounted return over the option's $\tau$ steps:

$$
Q(s, \omega) \;\gets\; Q(s, \omega) + \alpha\Bigl[\underbrace{r_{t+1} + \gamma r_{t+2} + \cdots + \gamma^{\tau-1} r_{t+\tau}}_{\text{option return }R}
\;+\; \gamma^{\tau}\max_{\omega'} Q(s_{t+\tau}, \omega') \;-\; Q(s, \omega)\Bigr].
$$

**A worked SMDP backup.** Suppose an option runs for $\tau = 3$ steps, collecting
rewards $r_{t+1} = 0$, $r_{t+2} = 0$, $r_{t+3} = 1$, with $\gamma = 0.9$, and lands
in a state whose best option value is $\max_{\omega'} Q(s_{t+3}, \omega') = 5$. The
accumulated return is $R = 0 + 0.9\cdot 0 + 0.9^2\cdot 1 = 0.81$, the discount on
the bootstrap is $\gamma^\tau = 0.9^3 = 0.729$, and the target is $R + \gamma^\tau
\cdot 5 = 0.81 + 0.729\cdot 5 = 4.455$. One backup carried the reward across three
primitive steps in a single update — where flat Q-learning would need three separate
one-step backups, and three separate visits, to move the same information. That leap
is the planning payoff of temporal abstraction, in one line of arithmetic.

SMDP learning works but wastes data — it ignores every intermediate state the option
passed through. **Intra-option learning** does better: it updates the option's value
(and those of _other_ options consistent with the actions taken) at every primitive
step, using the one-step data as it arrives. Intra-option methods learn about an
option without ever running it to completion, and they can update many options
from a single stream of experience, which is why they are the standard tabular
approach to option learning.[^spc]

## Learning options end to end

The classical options framework assumes the options are _given_ — a designer
supplies the initiation sets, intra-option policies, and termination conditions,
and the agent only learns to choose among them and to plan. That is a heavy
assumption. Hand-designing good options requires knowing the useful subgoals in
advance, which for a hard task is most of the problem. The modern question is
whether the options themselves can be _discovered_ by gradient descent, from the
task reward alone, with no subgoal ever specified by hand.

The **option-critic** architecture of Bacon, Harb, and Precup (2017) answers
yes.[^bacon] It parameterizes the intra-option policies and the termination
conditions and differentiates the expected return with respect to those
parameters, giving policy-gradient-style updates for both. Write the intra-option
policy of option $\omega$ as $\pi_{\omega, \theta}(a \mid s)$ and its termination
as $\beta_{\omega, \vartheta}(s)$, with a fixed policy-over-options $\pi_\Omega$
choosing which option to run. Two theorems — the **intra-option policy gradient**
and the **termination gradient** — give the derivative of the return with respect
to $\theta$ and $\vartheta$, learned end to end alongside a critic that estimates
option values $Q_\Omega(s, \omega)$.

The intra-option policy gradient is the familiar
[actor-critic](/reinforcement-learning/deep-rl/actor-critic-and-ppo) update, but
with the option-value $Q_\Omega$ as the critic:
$$
\frac{\partial}{\partial \theta}\, \mathbb{E}[\,\text{return}\,]
\;\propto\;
\mathbb{E}\!\left[\, \frac{\partial \log \pi_{\omega,\theta}(a \mid s)}{\partial \theta}\; Q_U(s, \omega, a) \,\right],
$$
where $Q_U(s,\omega,a)$ is the value of taking action $a$ inside option $\omega$.
The termination gradient is the informative one. It moves each $\beta_\omega$ in
the direction that shortens or extends the option based on an **advantage**: an
option should be more likely to terminate in a state where continuing it is worse
than switching,
$$
\frac{\partial}{\partial \vartheta}\, \mathbb{E}[\,\text{return}\,]
\;\propto\;
-\,\mathbb{E}\!\left[\, \frac{\partial \beta_{\omega,\vartheta}(s')}{\partial \vartheta}\; A_\Omega(s', \omega) \,\right],
\qquad
A_\Omega(s', \omega) = Q_\Omega(s', \omega) - V_\Omega(s').
$$
When the advantage $A_\Omega(s', \omega)$ is negative — the current option is worse
than the best available — the gradient _raises_ $\beta_\omega$, making the option
end so a better one can start. When the advantage is positive, it _lowers_
$\beta_\omega$, letting a good option keep running. Terminations are learned as a
cost-benefit decision rather than scripted.

$$
% caption: The option-critic. The policy-over-options selects an option; that
% option's intra-option policy emits primitive actions until its termination
% function fires, then control returns to the top. A critic estimates option
% values and drives both the intra-option policy gradient and the termination
% gradient, so options are learned end to end from task reward.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=24mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % top row: the control cycle, left to right
  \node[box, draw=acc, text=acc, thick] (poo) at (0,0) {policy over options};
  \node[box] (io) at (3.9,0) {intra-option\\pi-omega (actor)};
  \node[box] (env) at (7.8,0) {environment};
  \node[box, draw=red, text=red] (beta) at (11.7,0) {termination\\beta-omega};
  % critic below
  \node[box, draw=acc, text=acc] (crit) at (7.8,-3.0) {critic Q-Omega};
  \draw[->, acc, thick] (poo) -- node[midway, above, font=\scriptsize] {pick} (io);
  \draw[->, black] (io) -- node[midway, above, font=\scriptsize] {action a} (env);
  \draw[->, black] (env) -- node[midway, above, font=\scriptsize] {state s'} (beta);
  % return arc over the top
  \draw[->, red, thick] (beta.north) to[out=90, in=90, looseness=0.6] node[pos=0.5, above, font=\scriptsize] {stop: re-pick} (poo.north);
  % critic feeds
  \draw[->, acc, thick] (env) -- node[midway, right, font=\scriptsize] {experience} (crit);
  \draw[->, acc, densely dashed] (crit.west) to[out=180, in=270] node[pos=0.55, below, font=\scriptsize] {policy grad} (io.south);
  \draw[->, red, densely dashed] (crit.east) to[out=0, in=270] node[pos=0.55, below, font=\scriptsize] {term. grad} (beta.south);
\end{tikzpicture}
$$

> **Definition (Option-critic).** An architecture that learns options end to end
> by treating the intra-option policies and termination functions as differentiable
> parameters. A critic estimates option values $Q_\Omega(s,\omega)$; the
> intra-option policy-gradient theorem updates each $\pi_\omega$ toward higher
> value, and the termination-gradient theorem raises $\beta_\omega$ exactly where
> the current option's advantage over switching is negative. No subgoals,
> pseudo-rewards, or initiation sets are specified by hand.

Option-critic learns a whole hierarchy from the single task reward, which is its
appeal and its weakness. Nothing pressures the discovered options to be _diverse_
or _meaningful_: a common failure is **option collapse**, where one option
monopolizes behavior and the terminations fire every step, degenerating back to
flat RL, or where the options carve the state space into pieces that reduce return
but carry no interpretable meaning. Later work adds regularizers (deliberation
cost, diversity terms) to keep the options distinct, but the bare architecture
already shows that temporal abstraction can be learned rather than designed.

## Where this leaves us

The options framework formalizes temporal abstraction. An **option** is a triple — an
initiation set, an internal policy, and a termination condition — that packages a
temporally extended behavior into something the top level can select as if it were a
single action. Options turn an MDP into a **semi-Markov decision process** over
variable-duration decisions, and this shortens the effective horizon: the
top level makes far fewer, far more consequential choices, so both exploration and
credit assignment improve.

The **option-critic** then removes the need to hand-design options at all, learning the
intra-option policies and the termination conditions end to end by differentiating the
option-value objective — at the cost of the collapse pathologies (options that
terminate every step, or one option that swallows the task) that need regularization to
prevent.

Options let the agent commit to a sub-behavior, but they do not specify what a good
sub-behavior _is_. A complementary family gives the hierarchy an explicit language of
**goals**: a manager proposes a target state (or a direction in a latent space) and a
worker is rewarded for reaching it. That goal-conditioned view, hindsight relabeling
for learning from sparse reward, and the unsupervised discovery of skills with
no reward at all continue in
[Hierarchical RL: Goal-Conditioned Hierarchies and Skills](/reinforcement-learning/modern-deep-rl/hierarchical-rl-part-2).

[^spc]: **Sutton, Precup, Singh** (1999), "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning", _Artificial Intelligence_ — introduces the option $\omega = (\mathcal{I}_\omega, \pi_\omega, \beta_\omega)$, the semi-Markov decision process over options, option models (reward and discounted next-state parts), and SMDP versus intra-option learning; see also **Sutton & Barto**, §17.2, Temporal Abstraction via Options.
[^bacon]: **Bacon, Harb, Precup** (2017), "The Option-Critic Architecture", _AAAI_ — derives the intra-option policy-gradient and termination-gradient theorems and learns option policies and termination functions end to end from task reward with no hand-specified subgoals.