---
title: "Hierarchical RL: Goal-Conditioned Hierarchies and Skills"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 16
order: 516
summary: >
  A companion to the options lesson. Options package a behavior; goal-conditioned
  hierarchies instead give the top level an explicit language of goals — a manager
  proposes a target state or a latent direction, and a worker is rewarded for reaching
  it (FeUdal Networks, HIRO). We develop that architecture, the hindsight relabeling
  that lets it learn from sparse reward, and unsupervised skill discovery (DIAYN) that
  learns a repertoire of behaviors with no reward at all. The shared idea throughout:
  shorten the horizon by inserting a level that decides less often.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "Ch. 17 — Frontiers; §17.2 Temporal Abstraction via Options"
  - book: Grokking Deep RL
    ref: "Ch. 12 — Advanced actor-critic methods; hierarchy and abstraction"
---

This builds on
[Hierarchical RL: Options and the Option-Critic](/reinforcement-learning/modern-deep-rl/hierarchical-rl),
which formalized temporal abstraction with the **option** — an initiation set, an
internal policy, and a termination condition — showed how options turn an MDP into a
semi-Markov decision process that shortens the effective horizon, and learned options
end to end with the option-critic.

An option packages a sub-behavior, but nothing in the framework specifies what a _useful_
sub-behavior is; the option-critic discovers them only implicitly, and often collapses.
This lesson takes the complementary route: give the hierarchy an explicit language of
**goals**, so a manager can name what the worker should achieve.

## Goal-conditioned hierarchies

The option-critic learns a fixed, small set of options. A different and now
dominant design makes the top level emit a _continuous subgoal_ rather than pick
from a discrete set. The structure is a
**manager** and a **worker** (equivalently a high-level and low-level policy).
The manager, running at a slow timescale, looks at the state and outputs a
**subgoal** $g$ — a target for the worker to achieve. The worker, running at the
fast primitive timescale, is a **goal-conditioned policy** $\pi(a \mid s, g)$ that
takes the current state and the manager's subgoal and emits primitive actions
trying to reach $g$. Every $c$ steps the manager looks again and issues a fresh
subgoal.

> **Definition (Manager/worker hierarchy).** A two-level policy in which a manager
> (high-level policy) emits a subgoal $g$ every $c$ primitive steps, and a
> goal-conditioned worker (low-level policy) $\pi(a \mid s, g)$ acts to achieve
> that subgoal. The manager is rewarded by the environment's task reward; the
> worker is rewarded by an _intrinsic_ reward measuring progress toward $g$. The
> manager's decision horizon is shortened by a factor of $c$, and the worker faces
> a short, well-specified reaching problem.

The division of labor is what shortens the horizon. The manager acts once
every $c$ steps, so its effective horizon is $1/c$ of the flat horizon — a
five-hundred-step task becomes fifty manager decisions if $c = 10$. The worker
never sees the sparse task reward at all; it is handed a dense intrinsic reward
for moving toward whatever subgoal $g$ it was given, so its learning problem is
short and never reward-starved.

$$
% caption: A manager/worker hierarchy. Every $c$ steps the manager reads the
% state and emits a subgoal $g$; the goal-conditioned worker acts for $c$
% primitive steps to reach $g$, driven by an intrinsic reward for progress toward
% it. The manager is trained on the environment's task reward, so its horizon is
% shortened by the factor $c$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=28mm, minimum height=12mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=acc, text=acc, thick] (man) at (0,1.7) {manager\\(slow: every c steps)};
  \node[box] (wrk) at (0,-1.7) {worker\\pi(a given s, g)};
  \node[box] (env) at (7.2,0) {environment};
  \draw[->, acc, thick] (man) -- node[midway, left, font=\scriptsize] {subgoal g} (wrk);
  \draw[->, black] (wrk.north east) to[out=30, in=180] node[pos=0.6, above, font=\scriptsize] {action a} (env.west);
  \draw[->, black] (env.north) to[out=120, in=0] node[pos=0.4, above, font=\scriptsize] {task reward} (man.east);
  \draw[->, red, thick] (env.south) to[out=240, in=-30] node[pos=0.55, below, font=\scriptsize] {intrinsic: progress to g} (wrk.south east);
  \node[font=\scriptsize, text=black, anchor=east] at (-1.7,-1.7) {fast:\\every step};
\end{tikzpicture}
$$

### FeUdal Networks

FeUdal Networks (FuN) of Vezhnevets et al. (2017) is the neural realization of
this manager/worker idea, borrowing the "feudal" framing — a hierarchy of managers
setting goals for sub-managers — from much older work.[^vezhnevets] Its central
design choice is that the manager sets its subgoal as a **direction in a learned
latent space**, not a raw state. The manager, running at low temporal resolution,
outputs a goal vector $g_t$ in a $d$-dimensional embedding; the worker is rewarded
for _moving_ the state embedding in that direction, measured by cosine similarity.

Two decisions make FuN work. First, the subgoal is directional: the worker's
intrinsic reward is the cosine similarity between the direction the state
embedding actually moved and the direction the manager requested,
$$
r^i_t \;=\; \frac{1}{c}\sum_{i=1}^{c} \cos\!\bigl(\, s_{t} - s_{t-i},\; g_{t-i} \,\bigr),
$$
so the worker is pushed to make progress _along_ the manager's goal rather than to
arrive at a specific point. Second, the manager is trained with a **transition
policy gradient** that treats the direction the state embedding traveled over $c$
steps as the manager's "action," so the manager learns which directions in latent
space lead to reward without ever needing gradients to flow through the worker.
Decoupling the two levels this way — the manager never differentiates through the
worker — is what keeps the architecture stable, and it let FuN outperform a flat
[actor-critic](/reinforcement-learning/deep-rl/actor-critic-and-ppo) on
long-horizon Atari games such as Montezuma's Revenge, where directed subgoals
supply the exploration flat agents lack.

$$
% caption: FeUdal's directional subgoal. The manager emits a goal direction g in a
% learned latent space (arrow). The worker is rewarded by the cosine similarity
% between the direction the state embedding actually moved (s minus its past) and
% the requested direction g, so it is pushed to make progress along the manager's
% goal rather than to hit any specific point.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{grn}{HTML}{1F9D4D}
  % origin (current embedding)
  \fill[black] (0,0) circle (2pt);
  \node[anchor=north, font=\scriptsize] at (0,-0.15) {state now};
  % manager goal direction (steeper)
  \draw[->, acc, very thick] (0,0) -- (4.0,2.9);
  \node[acc, anchor=south west, font=\scriptsize] at (4.0,2.9) {manager goal g};
  % worker actual move (shallower -> a visible angle apart)
  \draw[->, grn, thick] (0,0) -- (4.6,1.4);
  \node[grn, anchor=north west, font=\scriptsize] at (4.6,1.4) {worker move};
  % angle wedge between the two directions near origin
  \draw[red] (1.3,0.94) arc (36:17:1.6);
  \node[red, anchor=west, font=\scriptsize] at (1.7,0.55) {angle};
  \node[red, anchor=north east, font=\scriptsize] at (4.5,0.55) {reward = cos(angle)};
\end{tikzpicture}
$$

### HIRO and off-policy goal relabeling

FeUdal Networks are on-policy and sample-hungry. HIRO — Hierarchical Reinforcement
learning with Off-policy correction — of Nachum et al. (2018) makes the
manager/worker design **off-policy** and therefore far more sample-efficient, which
is what makes it practical for real continuous-control robotics.[^nachum] HIRO
makes two choices opposite to FuN's. First, the manager's subgoal is a target in
the _raw state space_ (or a subset of it), not a latent direction: the manager
outputs a goal state $g_t$, and the worker's intrinsic reward is simply the
negative distance to it,
$$
r^i_t \;=\; -\,\bigl\lVert\, s_t + g_t - s_{t+1} \,\bigr\rVert,
$$
using a directional goal that is shifted each step so it always points at the
target from the current state. Second, both levels are trained off-policy with a
[replay buffer](/reinforcement-learning/deep-rl/deep-q-networks), reusing every
transition many times.

Training the manager off-policy introduces one genuine difficulty — the
"off-policy correction" in HIRO's name. A stored high-level transition records
that in state $s$ the manager issued subgoal $g$, and $c$ steps of worker behavior
later received task reward $R$ and landed in $s'$. But the worker's policy has
_changed_ since that experience was
collected. Replaying the old $(s, g, R, s')$ transition to update the manager is
invalid, because the same subgoal $g$ would no longer produce the same behavior
from the current worker. HIRO corrects this with **off-policy goal relabeling**: it
re-labels the stored subgoal $g$ with the subgoal $\tilde g$ that would have made
the _current_ worker most likely to have produced the actions actually observed,
$$
\tilde g \;=\; \arg\max_{g}\; \prod_{i=t}^{t+c-1} \pi\bigl(a_i \mid s_i, \phi(g, s_i, s_t)\bigr),
$$
found by a small search over candidate goals. The manager is then updated as
though it had issued the corrected subgoal $\tilde g$, so its target matches the
behavior the current worker would actually produce. This one correction is what
lets the manager learn off-policy at all, and it moved hierarchical RL from Atari
scores to learning multi-stage locomotion (a simulated ant navigating and pushing
objects) directly from primitive torques.

$$
% caption: HIRO's off-policy goal relabeling. A stored high-level transition
% recorded subgoal g and the c worker actions it induced. Since the worker has
% changed, g no longer explains those actions; HIRO searches candidate subgoals for
% the g-tilde under which the CURRENT worker would most likely have taken the same
% actions, and updates the manager as if it had issued g-tilde.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=30mm, minimum height=12mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{grn}{HTML}{1F9D4D}
  \node[box] (stored) at (0,0) {stored: subgoal g\\+ actions taken};
  \node[box] (search) at (5.0,0) {search candidate\\subgoals};
  \node[box, draw=grn, text=grn, thick] (relabel) at (10.0,0) {relabel to g-tilde\\best explains actions};
  \draw[->, black, thick] (stored) -- (search) node[midway, above, font=\scriptsize] {worker changed};
  \draw[->, grn, thick] (search) -- (relabel);
  \node[grn, anchor=south, font=\scriptsize] at (7.5,0.75) {argmax likelihood};
  \node[red, anchor=north, font=\scriptsize] at (2.5,-1.0) {g is now stale};
  \draw[->, acc, thick] (relabel.south) to[out=270, in=270] node[pos=0.5, below, font=\scriptsize, text=acc] {update manager with g-tilde} (stored.south);
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{HIRO}$ — off-policy manager/worker with goal relabeling
initialize manager $\mu_{\text{hi}}$, worker $\pi_{\text{lo}}$, replay buffers
for each episode do
  initialize state $s_0$
  for $t = 0, c, 2c, \ldots$ do
    $g_t \gets \mu_{\text{hi}}(s_t)$ // manager sets a subgoal every $c$ steps
    for $i = t$ to $t + c - 1$ do
      $a_i \gets \pi_{\text{lo}}(s_i, g_i)$ // worker acts toward the subgoal
      take $a_i$, observe $r_i$, $s_{i+1}$
      $r^i_i \gets -\lVert s_i + g_i - s_{i+1} \rVert$ // intrinsic worker reward
      $g_{i+1} \gets s_i + g_i - s_{i+1}$ // shift goal to stay directional
    store worker transitions in the low-level buffer
    $\tilde g_t \gets$ relabel $g_t$ to best explain the actions taken
    store $(s_t, \tilde g_t, \textstyle\sum r_i, s_{t+c})$ in the high-level buffer
  update $\pi_{\text{lo}}$ and $\mu_{\text{hi}}$ off-policy from their buffers
```

## A note on hindsight for sparse goals

Goal-conditioned learning has a companion technique for the same sparse-reward
problem. When a worker is asked to reach goal $g$ and _fails_, the standard reward
is $-1$ or $0$ throughout — an uninformative signal, and if the goal is hard to
reach, every early episode returns the same signal, so the worker learns nothing. Hindsight Experience Replay (HER) of Andrychowicz et al.
(2017) extracts a learning signal from failure by a simple relabeling: an episode
that failed to reach $g$ _did_ reach whatever state it ended in.[^andrychowicz] So
store the trajectory twice — once with the original goal $g$ (a failure), and once
with the **achieved** final state as the goal $g'$ (a success). Under $g'$ the same
actions earn the success reward, and the goal-conditioned policy learns _how to
reach $g'$_ from a trajectory that reached it, even though $g'$ was never what we
asked for.

> **Definition (Hindsight Experience Replay).** A relabeling scheme for
> goal-conditioned RL: after collecting a trajectory aimed at goal $g$, additionally
> store its transitions relabeled with a goal $g'$ actually achieved along the
> trajectory (e.g. the final state), so those transitions carry a success reward.
> Every episode, successful or not, yields learnable experience about reaching
> _some_ goal, which turns an otherwise-silent sparse reward into a dense stream of
> achieved-goal signals.

**A worked relabeling.** Say a robot arm is asked to move a puck to goal cell
$g = (9, 9)$ on a grid, with reward $0$ on reaching $g$ and $-1$ every other step.
An early, clumsy episode pushes the puck along the states $(0,0) \to (1,0) \to
(2,1) \to (3,1)$ and stops there, nowhere near $(9,9)$. Under the original goal the
whole trajectory scores $-1$ at every step, with no signal about what worked.
HER stores the same four transitions a second time with the
achieved final state $g' = (3,1)$ substituted as the goal. Now the last transition,
$(2,1) \to (3,1)$ with goal $(3,1)$, earns reward $0$ (goal reached), and the
goal-conditioned value function gains a positive example: from $(2,1)$, this
action reaches $(3,1)$. The policy learns to reach $(3,1)$ from a trajectory that
_did_ reach $(3,1)$, even though $(3,1)$ was never what we asked for. Repeated across
episodes, the arm becomes able to reach an ever-wider set of nearby goals; the
set of reachable goals expands outward until it includes the real target
$(9,9)$.

$$
% caption: Hindsight relabeling. The trajectory (blue) aimed at goal g fails,
% earning -1 throughout under g. Relabeled with the achieved final state g' as the
% goal (green), the same transitions now include a success, so the policy learns
% how to reach g' from a run that actually reached it. Every failed episode becomes
% a success for some goal.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  st/.style={circle, draw, minimum size=6mm, inner sep=0pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{grn}{HTML}{1F9D4D}
  % trajectory
  \node[st, draw=acc] (a) at (0,0) {s0};
  \node[st, draw=acc] (b) at (1.9,0.4) {s1};
  \node[st, draw=acc] (c) at (3.8,0.1) {s2};
  \node[st, draw=acc, fill=grn!15, thick] (d) at (5.7,0.5) {s3};
  \draw[->, acc, thick] (a) -- (b); \draw[->, acc, thick] (b) -- (c); \draw[->, acc, thick] (c) -- (d);
  % original goal far away
  \node[st, draw=red, dashed] (g) at (8.6,-1.2) {g};
  \node[red, anchor=north, font=\scriptsize] at (8.6,-1.6) {asked goal: -1 always};
  \draw[red!50, dotted] (d) -- (g);
  % relabeled goal = achieved
  \node[grn, anchor=south, font=\scriptsize] at (5.7,1.05) {relabel g' = s3: success};
  \node[acc, anchor=north, font=\scriptsize] at (0,-0.55) {start};
\end{tikzpicture}
$$

HER's advantage is that _every_ trajectory teaches the policy how to reach the goals
it happened to hit, and as the policy improves those achieved goals drift toward
the real ones. It composes directly with the hierarchies above: a HIRO or FeUdal
worker is a goal-conditioned policy, so HER-style relabeling of its subgoal
transitions gives it dense signal even before it can reach the manager's harder
requests, which is why hindsight relabeling and goal-conditioned hierarchies are so
often deployed together on sparse-reward robotics.

## Unsupervised skills and goal-space learning

The methods above still need a task reward to discover their abstractions.
Post-Sutton-&-Barto work asks a sharper question: can useful skills be learned
_before_ any task reward, from the structure of the environment alone, and can the
goal space itself be learned rather than hand-picked?

**Unsupervised skill discovery.** **DIAYN** — "Diversity is All You Need"
(Eysenbach et al. 2018) — learns a set of distinguishable skills with _no_ reward at
all.[^diayn] It samples a latent skill $z$, runs a skill-conditioned policy
$\pi(a \mid s, z)$, and rewards the policy by how well a discriminator can infer $z$
from the states visited — an intrinsic reward for making each skill visit a
distinctive region of state space. Formally it maximizes the mutual information
$I(S; Z)$ between visited states and the skill. The result is a repertoire of
diverse behaviors (a cheetah that learns to run, flip, and bound, each a different
$z$) that can then serve as the option set for a hierarchy, learned entirely before
the downstream task is known.

$$
% caption: Unsupervised skill discovery (DIAYN). A skill latent z conditions the
% policy; a discriminator tries to predict which skill produced the visited states.
% Rewarding the policy for being easy to identify pushes different skills into
% different regions of state space, yielding a diverse behavior repertoire with no
% task reward.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=26mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[box, draw=acc, text=acc, thick] (pol) at (0,0) {skill policy\\pi(a given s, z)};
  \node[box] (env) at (4.6,0) {environment:\\visited states};
  \node[box, draw=red, text=red] (disc) at (4.6,-2.5) {discriminator\\predicts z};
  \node[anchor=east, text=black, font=\scriptsize] at (-1.6,0) {skill z};
  \draw[->, black] (-1.55,0) -- (pol.west);
  \draw[->, acc, thick] (pol) -- (env) node[midway, above, font=\scriptsize] {act};
  \draw[->, black, thick] (env) -- (disc) node[midway, right, font=\scriptsize] {states};
  \draw[->, red, thick] (disc.west) to[out=180, in=270] (pol.south);
  \node[red, anchor=north, font=\scriptsize] at (1.5,-2.55) {intrinsic reward: skill is identi\/f\/iable};
\end{tikzpicture}
$$
Related information-theoretic objectives —
empowerment, variational intrinsic control (Gregor et al. 2016) — share the idea
that a good skill is one whose outcome the agent can reliably control and
distinguish.

**Learning the goal space.** HIRO and FeUdal fix the space subgoals live in — raw
states or a latent direction. **HAC** (Levy et al. 2019) stacks more than two levels
and uses hindsight at every level to train them in parallel, and a line of work on
_learned_ goal representations (e.g. representation objectives that make the goal
space metric match reachability) tackles the problem that raw-state subgoals are a
poor target in high-dimensional observation spaces like images. The manager should
propose goals in a space where "distance" means "steps to reach," not "pixels
different."

**Automatic curricula over goals.** Once a worker is goal-conditioned, _which_ goals
should it practice? Hindsight practices the goals the policy happened to hit; a
complementary line generates goals of intermediate difficulty on purpose — **GoalGAN**
(Florensa et al. 2018) trains a generator to propose goals the current policy solves
neither always nor never, keeping the worker at the edge of its competence. This is
the single-agent analogue of the autocurriculum that self-play produced in the
multi-agent lesson: difficulty that tracks competence, generated automatically.

These directions share the hierarchical premise — a level of decision above
primitives — but push the _source_ of the abstraction from the task reward toward
the environment's own structure, which is what lets hierarchy scale to tasks where
no one can name the right subgoals in advance.

## The shared idea

Every method here is one answer to the same question: how do you get an agent to
act coherently over a horizon far longer than one-step credit assignment can
manage? The answer is always to introduce a level of decision above the primitive
one, so that a single high-level choice commits the agent to a directed, extended
course of behavior.

| Method | Abstraction unit | Learned by | Key mechanism |
| --- | --- | --- | --- |
| Options (SPS 1999) | option $(\mathcal{I}_\omega, \pi_\omega, \beta_\omega)$ | designed; planned via SMDP | temporally-extended action, $\gamma^\tau$ discount |
| Option-critic | discrete options | end-to-end policy gradient | intra-option + termination gradients |
| FeUdal Networks | latent-direction subgoal | on-policy, decoupled levels | directional intrinsic reward, transition gradient |
| HIRO | raw-state subgoal | off-policy, replay | goal relabeling for off-policy manager |
| HER | achieved-goal relabel | any off-policy learner | turn failures into successes for some goal |

The rows form a progression. The options framework supplied the abstraction and the
[SMDP planning](/reinforcement-learning/tabular-methods/planning-and-learning) to
use it, but assumed the options were given. Option-critic made the options
themselves differentiable. FeUdal Networks and HIRO replaced the discrete option
menu with a continuous subgoal a manager emits on the fly, and HIRO's off-policy
correction plus HER's hindsight relabeling made that design sample-efficient enough
for real control. Underneath all of it sits the one claim that motivated the
[frontiers](/reinforcement-learning/deep-rl/frontiers) discussion of temporal
abstraction: a hierarchy makes a hard problem easier by arithmetic. It turns one
problem of horizon $H$ into a shallow tree of problems of horizon $H/c$, and
short-horizon RL is the part we already know how to solve.


[^vezhnevets]: **Vezhnevets, Osindero, Schaul, Heess, Jaderberg, Silver, Kavukcuoglu** (2017), "FeUdal Networks for Hierarchical Reinforcement Learning", _ICML_ — a manager sets directional subgoals in a learned latent space at low temporal resolution and a worker is rewarded by cosine similarity for moving in that direction; the manager is trained by a transition policy gradient that decouples it from the worker.
[^nachum]: **Nachum, Gu, Lee, Levine** (2018), "Data-Efficient Hierarchical Reinforcement Learning", _NeurIPS_ — HIRO: a two-level manager/worker with raw-state subgoals trained off-policy, using off-policy goal relabeling to correct high-level transitions for the changing worker, enabling sample-efficient continuous control.
[^andrychowicz]: **Andrychowicz, Wolski, Ray, Schneider, Fong, Welinder, McGrew, Tobin, Abbeel, Zaremba** (2017), "Hindsight Experience Replay", _NeurIPS_ — relabels failed goal-conditioned trajectories with goals actually achieved along them, so every episode yields a success signal for some goal and sparse-reward goal-conditioned tasks become learnable.
[^diayn]: **Eysenbach, Gupta, Ibarz, Levine** (2018), "Diversity is All You Need: Learning Skills without a Reward Function", _ICLR 2019_ — DIAYN learns a set of distinguishable skills by maximizing the mutual information between visited states and a skill latent, rewarding the policy for making each skill identifiable, with no task reward; see also **Gregor, Rezende, Wierstra** (2016), "Variational Intrinsic Control", for the empowerment-style precursor.
[^hac]: **Levy, Konidaris, Platt, Saenko** (2019), "Learning Multi-Level Hierarchies with Hindsight", _ICLR_ — HAC trains a stack of goal-conditioned policies in parallel using hindsight at every level to make multi-level hierarchies learnable.
[^goalgan]: **Florensa, Held, Geng, Abbeel** (2018), "Automatic Goal Generation for Reinforcement Learning Agents", _ICML_ — GoalGAN generates goals of intermediate difficulty (solved neither always nor never) to form an automatic curriculum over goals for a goal-conditioned agent.