Hierarchical RL: Goal-Conditioned Hierarchies and Skills

This builds on Hierarchical RL: Options and the Option-Critic, which formalized temporal abstraction with the option — an initiation set, an internal policy, and a termination condition — showed how options turn an MDP into a semi-Markov decision process that shortens the effective horizon, and learned options end to end with the option-critic.

An option packages a sub-behavior, but nothing in the framework specifies what a useful sub-behavior is; the option-critic discovers them only implicitly, and often collapses. This lesson takes the complementary route: give the hierarchy an explicit language of goals, so a manager can name what the worker should achieve.

Goal-conditioned hierarchies

The option-critic learns a fixed, small set of options. A different and now dominant design makes the top level emit a continuous subgoal rather than pick from a discrete set. The structure is a manager and a worker (equivalently a high-level and low-level policy). The manager, running at a slow timescale, looks at the state and outputs a subgoal $g$ — a target for the worker to achieve. The worker, running at the fast primitive timescale, is a goal-conditioned policy $\pi (a\mid s,g)$ that takes the current state and the manager's subgoal and emits primitive actions trying to reach $g$. Every $c$ steps the manager looks again and issues a fresh subgoal.

The division of labor is what shortens the horizon. The manager acts once every $c$ steps, so its effective horizon is $1/c$ of the flat horizon — a five-hundred-step task becomes fifty manager decisions if $c=10$. The worker never sees the sparse task reward at all; it is handed a dense intrinsic reward for moving toward whatever subgoal $g$ it was given, so its learning problem is short and never reward-starved.

FeUdal Networks

FeUdal Networks (FuN) of Vezhnevets et al. (2017) is the neural realization of this manager/worker idea, borrowing the feudal framing — a hierarchy of managers setting goals for sub-managers — from much older work.¹ Its central design choice is that the manager sets its subgoal as a direction in a learned latent space, not a raw state. The manager, running at low temporal resolution, outputs a goal vector $g_t$ in a $d$-dimensional embedding; the worker is rewarded for moving the state embedding in that direction, measured by cosine similarity.

Two decisions make FuN work. First, the subgoal is directional: the worker's intrinsic reward is the cosine similarity between the direction the state embedding actually moved and the direction the manager requested,

so the worker is pushed to make progress along the manager's goal rather than to arrive at a specific point. Second, the manager is trained with a transition policy gradient that treats the direction the state embedding traveled over $c$ steps as the manager's action, so the manager learns which directions in latent space lead to reward without ever needing gradients to flow through the worker. Decoupling the two levels this way — the manager never differentiates through the worker — is what keeps the architecture stable, and it let FuN outperform a flat actor-critic on long-horizon Atari games such as Montezuma's Revenge, where directed subgoals supply the exploration flat agents lack.

HIRO and off-policy goal relabeling

FeUdal Networks are on-policy and sample-hungry. HIRO — Hierarchical Reinforcement learning with Off-policy correction — of Nachum et al. (2018) makes the manager/worker design off-policy and therefore far more sample-efficient, which is what makes it practical for real continuous-control robotics.² HIRO makes two choices opposite to FuN's. First, the manager's subgoal is a target in the raw state space (or a subset of it), not a latent direction: the manager outputs a goal state $g_t$, and the worker's intrinsic reward is simply the negative distance to it,

using a directional goal that is shifted each step so it always points at the target from the current state. Second, both levels are trained off-policy with a replay buffer, reusing every transition many times.

Training the manager off-policy introduces one genuine difficulty — the off-policy correction in HIRO's name. A stored high-level transition records that in state $s$ the manager issued subgoal $g$, and $c$ steps of worker behavior later received task reward $R$ and landed in $s^{\prime }$. But the worker's policy has changed since that experience was collected. Replaying the old $(s,g,R,s^{\prime })$ transition to update the manager is invalid, because the same subgoal $g$ would no longer produce the same behavior from the current worker. HIRO corrects this with off-policy goal relabeling: it re-labels the stored subgoal $g$ with the subgoal $\tilde{g}$ that would have made the current worker most likely to have produced the actions actually observed,

found by a small search over candidate goals. The manager is then updated as though it had issued the corrected subgoal $\tilde{g}$, so its target matches the behavior the current worker would actually produce. This one correction is what lets the manager learn off-policy at all, and it moved hierarchical RL from Atari scores to learning multi-stage locomotion (a simulated ant navigating and pushing objects) directly from primitive torques.

A note on hindsight for sparse goals

Goal-conditioned learning has a companion technique for the same sparse-reward problem. When a worker is asked to reach goal $g$ and fails, the standard reward is $-1$ or $0$ throughout — an uninformative signal, and if the goal is hard to reach, every early episode returns the same signal, so the worker learns nothing. Hindsight Experience Replay (HER) of Andrychowicz et al. (2017) extracts a learning signal from failure by a simple relabeling: an episode that failed to reach $g$ did reach whatever state it ended in.³ So store the trajectory twice — once with the original goal $g$ (a failure), and once with the achieved final state as the goal $g^{\prime }$ (a success). Under $g^{\prime }$ the same actions earn the success reward, and the goal-conditioned policy learns how to reach $g^{\prime }$ from a trajectory that reached it, even though $g^{\prime }$ was never what we asked for.

A worked relabeling. Say a robot arm is asked to move a puck to goal cell $g=(9,9)$ on a grid, with reward $0$ on reaching $g$ and $-1$ every other step. An early, clumsy episode pushes the puck along the states $(0,0)\to (1,0)\to (2,1)\to (3,1)$ and stops there, nowhere near $(9,9)$. Under the original goal the whole trajectory scores $-1$ at every step, with no signal about what worked. HER stores the same four transitions a second time with the achieved final state $g^{\prime }=(3,1)$ substituted as the goal. Now the last transition, $(2,1)\to (3,1)$ with goal $(3,1)$, earns reward $0$ (goal reached), and the goal-conditioned value function gains a positive example: from $(2,1)$, this action reaches $(3,1)$. The policy learns to reach $(3,1)$ from a trajectory that did reach $(3,1)$, even though $(3,1)$ was never what we asked for. Repeated across episodes, the arm becomes able to reach an ever-wider set of nearby goals; the set of reachable goals expands outward until it includes the real target $(9,9)$.

HER's advantage is that every trajectory teaches the policy how to reach the goals it happened to hit, and as the policy improves those achieved goals drift toward the real ones. It composes directly with the hierarchies above: a HIRO or FeUdal worker is a goal-conditioned policy, so HER-style relabeling of its subgoal transitions gives it dense signal even before it can reach the manager's harder requests, which is why hindsight relabeling and goal-conditioned hierarchies are so often deployed together on sparse-reward robotics.

Unsupervised skills and goal-space learning

The methods above still need a task reward to discover their abstractions. Post-Sutton-&-Barto work asks a sharper question: can useful skills be learned before any task reward, from the structure of the environment alone, and can the goal space itself be learned rather than hand-picked?

Unsupervised skill discovery. DIAYN — Diversity is All You Need (Eysenbach et al. 2018) — learns a set of distinguishable skills with no reward at all.⁴ It samples a latent skill $z$, runs a skill-conditioned policy $\pi (a\mid s,z)$, and rewards the policy by how well a discriminator can infer $z$ from the states visited — an intrinsic reward for making each skill visit a distinctive region of state space. Formally it maximizes the mutual information $I(S;Z)$ between visited states and the skill. The result is a repertoire of diverse behaviors (a cheetah that learns to run, flip, and bound, each a different $z$) that can then serve as the option set for a hierarchy, learned entirely before the downstream task is known.

Related information-theoretic objectives — empowerment, variational intrinsic control (Gregor et al. 2016) — share the idea that a good skill is one whose outcome the agent can reliably control and distinguish.

Learning the goal space. HIRO and FeUdal fix the space subgoals live in — raw states or a latent direction. HAC (Levy et al. 2019) stacks more than two levels and uses hindsight at every level to train them in parallel, and a line of work on learned goal representations (e.g. representation objectives that make the goal space metric match reachability) tackles the problem that raw-state subgoals are a poor target in high-dimensional observation spaces like images. The manager should propose goals in a space where distance means steps to reach, not pixels different.

Automatic curricula over goals. Once a worker is goal-conditioned, which goals should it practice? Hindsight practices the goals the policy happened to hit; a complementary line generates goals of intermediate difficulty on purpose — GoalGAN (Florensa et al. 2018) trains a generator to propose goals the current policy solves neither always nor never, keeping the worker at the edge of its competence. This is the single-agent analogue of the autocurriculum that self-play produced in the multi-agent lesson: difficulty that tracks competence, generated automatically.

These directions share the hierarchical premise — a level of decision above primitives — but push the source of the abstraction from the task reward toward the environment's own structure, which is what lets hierarchy scale to tasks where no one can name the right subgoals in advance.

The shared idea

Every method here is one answer to the same question: how do you get an agent to act coherently over a horizon far longer than one-step credit assignment can manage? The answer is always to introduce a level of decision above the primitive one, so that a single high-level choice commits the agent to a directed, extended course of behavior.

The rows form a progression. The options framework supplied the abstraction and the SMDP planning to use it, but assumed the options were given. Option-critic made the options themselves differentiable. FeUdal Networks and HIRO replaced the discrete option menu with a continuous subgoal a manager emits on the fly, and HIRO's off-policy correction plus HER's hindsight relabeling made that design sample-efficient enough for real control. Underneath all of it sits the one claim that motivated the frontiers discussion of temporal abstraction: a hierarchy makes a hard problem easier by arithmetic. It turns one problem of horizon $H$ into a shallow tree of problems of horizon $H/c$, and short-horizon RL is the part we already know how to solve.