---
title: "Frontiers: Beyond the Standard MDP"
module: Deep Reinforcement Learning
moduleNumber: 4
lessonNumber: 7
order: 407
summary: >
  The standard MDP fixes three things — state, reward, and single-step actions —
  and this lesson loosens two of them. We generalize the value function into a
  general value function that predicts any signal, and use those predictions as
  auxiliary tasks that shape representations; we extend actions in time with the
  options framework; and we treat state as a construction the agent builds from a
  stream of observations. Reward design and the open problems follow in the next
  lesson.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "Ch. 17 — Frontiers; §17.1 General Value Functions and Auxiliary Tasks; §17.2 Temporal Abstraction via Options"
  - book: Sutton & Barto
    ref: "§17.3 Observations and State"
---

Every lesson so far has fixed the same three things and varied only the method:
the state was given, the reward was given, and the action was a single primitive
step. The [Markov decision process](/reinforcement-learning/foundations/markov-decision-processes)
handed the agent a Markov state; the environment emitted a scalar reward; and a
policy chose one action per time step. That triple is a modeling convenience, and
this chapter loosens all three. What we predict need not be reward;
what we do need not last one step; and the state the agent reasons over is
something it constructs, not something it is handed. These are the frontiers —
partly known, partly open — that the deep-learning machinery of the last two
lessons meets outside the benchmarks.[^sb-intro]

## General value functions

A [value function](/reinforcement-learning/foundations/value-functions-and-optimality)
answers one question: starting here and following $\pi$, how much reward will I
accumulate? Over the course of the subject that question has quietly widened.
[Off-policy learning](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad)
let the value be conditioned on a target policy $\pi$ different from the one
generating the data. Generalized discounting then replaced the constant $\gamma$
with a state-dependent **termination function** $\gamma : \mathcal{S} \to [0, 1]$,
so a different horizon could apply at each step. The last generalization is to drop
reward itself. Instead of summing future rewards, sum the future values of _any_
signal — a sound, a pixel intensity, an internal feature, even another
prediction.[^sb-gvf]

A value function predicts a discounted sum into the future, and nothing forces
that sum to be reward. Substituting any other signal gives a forecast of that
signal, learned by the same machinery.

The accumulated signal is the **cumulant** $C_t \in \mathbb{R}$, and the resulting
prediction is a **general value function**, or GVF:

$$
v_{\pi, \gamma, C}(s) \;\doteq\; \mathbb{E}\!\left[\, \sum_{k=t}^{\infty}
\left( \prod_{i=t+1}^{k} \gamma(S_i) \right) C_{k+1}
\;\middle|\; S_t = s,\; A_{t:\infty} \sim \pi \right].
$$

A GVF is specified by three questions: a policy $\pi$ (act how?), a termination
$\gamma$ (over what horizon?), and a cumulant $C$ (accumulate what?). Set
$C_t = R_t$ and $\gamma$ constant and it collapses to the ordinary value function.
Because a GVF has no necessary connection to reward, calling it a _value_ is a
slight misnomer; it is really a **prediction**, or forecast. But it has exactly the
form of a value function, so every method here — semi-gradient TD,
Generalized Policy Iteration, actor–critic — learns and controls it without
change.[^sb-gvf]

> **Definition (General value function).** A prediction of the discounted sum of a
> cumulant signal $C_t$ under a target policy $\pi$ and a state-dependent
> termination $\gamma(s)$, written $v_{\pi, \gamma, C}(s)$. The ordinary value
> function is the special case $C_t = R_t$ with constant $\gamma$.

$$
% caption: A GVF is a value function whose three knobs are set independently:
% the policy $\pi$, the termination $\gamma$, and the cumulant $C$. Fixing
% $C = R$ and constant $\gamma$ recovers the ordinary value function $v_\pi$.
\begin{tikzpicture}[>=stealth, font=\small,
  knob/.style={draw, minimum width=22mm, minimum height=11mm, align=center, font=\footnotesize},
  outbox/.style={draw, thick, minimum width=30mm, minimum height=11mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[knob] (pi)  at (0,1.7)  {policy pi\\(act how?)};
  \node[knob] (ga)  at (0,0)    {termination\\(what horizon?)};
  \node[knob] (cu)  at (0,-1.7) {cumulant C\\(accumulate what?)};
  \node[outbox, draw=acc, text=acc] (gvf) at (5.6,0) {prediction\\v(s)};
  \draw[->, acc, thick] (pi) -- (gvf);
  \draw[->, acc, thick] (ga) -- (gvf);
  \draw[->, acc, thick] (cu) -- (gvf);
  \node[font=\scriptsize, text=black, align=center, anchor=north] at (5.6,-1.0)
    {C = R, constant horizon\\gives ordinary value};
\end{tikzpicture}
$$

### Auxiliary tasks

One prediction is a curiosity; the point is to learn many at once. An artificial
neural network can be built with its final layer split into several **heads**, each
computing a different GVF over one shared body. One head predicts the true value
(reward as its cumulant); the others predict change in pixels, the next step's
reward, the distribution of the return, and so on. All heads backpropagate their
errors into the same preceding layers, so the shared representation must serve
every prediction at once.[^sb-aux]

These extra predictions are **auxiliary tasks**: in addition to, and in service of,
the main task of maximizing reward. The mechanism is representation learning. Some
auxiliary tasks are easier than the main one — less delay, a clearer link between
action and outcome — so good features surface early on them, and those features
then speed the harder main task. Predicting and controlling its sensors over a
few seconds might, for instance, force the network to discover the idea of an
object — the structure long-horizon reward prediction depends on. In
practice this multi-head arrangement has been shown to sharply accelerate learning
on the main task.[^sb-aux]

$$
% caption: Auxiliary tasks: one shared body feeds several heads, each a GVF with
% its own cumulant. The main head predicts reward; the others predict pixel
% change, next-step reward, and the return distribution. All errors flow back
% into the shared body, so its representation must serve every head.
\begin{tikzpicture}[>=stealth, font=\small,
  body/.style={draw, thick, minimum width=16mm, minimum height=30mm, align=center},
  head/.style={draw, minimum width=30mm, minimum height=8mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[body] (b) at (0,0) {shared\\body};
  \node[head, draw=acc, text=acc] (h1) at (5.0,2.1)  {value (cumulant = reward)};
  \node[head] (h2) at (5.0,0.7)  {predict pixel change};
  \node[head] (h3) at (5.0,-0.7) {predict next-step reward};
  \node[head] (h4) at (5.0,-2.1) {predict return distribution};
  \draw[->, thick] (b.east) -- (h1.west);
  \draw[->, thick] (b.east) -- (h2.west);
  \draw[->, thick] (b.east) -- (h3.west);
  \draw[->, thick] (b.east) -- (h4.west);
  \draw[->, acc, dashed] (h1.west) to[bend right=12] node[midway, above, font=\scriptsize] {} (b.north east);
  \node[font=\scriptsize, text=black, anchor=north, align=center] at (0,-1.9) {errors\\f\/low back};
  \draw[->, acc, dashed] (2.6,-2.4) to[bend left=10] (0.4,-1.55);
\end{tikzpicture}
$$

A second reason to care about many predictions is biological. One reading of
classical conditioning is that evolution wired a fixed reflex to the _prediction_
of a signal rather than to the signal itself: an animal blinks when its prediction
of being poked in the eye crosses a threshold, and is spared many pokes. A
self-driving car that predicts a collision can be given a built-in reflex to brake
before one occurs. The prediction is learned; the response to it is designed. This
is the thread that reconnects the algorithms here to the
[psychology of reinforcement](/reinforcement-learning/minds-and-brains/psychology-of-reinforcement).[^sb-aux]

## Temporal abstraction via options

The MDP framework is scale-agnostic: it can formalize which muscle to twitch, which
flight to book, and which career to pursue. The trouble is doing all three inside
_one_ MDP. Planning a flight at the level of muscle twitches is hopeless, yet people
move between such levels without seeming to switch at all. The
[options framework](/reinforcement-learning/tabular-methods/planning-and-learning)
stretches the MDP to cover many time scales at once: formalize the world at a fine
time step, but let the agent plan with extended courses of action that terminate on
their own.[^sb-options]

An **option** couples two of the objects we already have. It is a pair
$\omega = (\pi_\omega, \gamma_\omega)$: a policy $\pi_\omega$ that says which action
to take, and a termination function $\gamma_\omega : \mathcal{S} \to [0, 1]$ that
says, at each state, the probability the option ends. To execute $\omega$ from state
$S_t$, draw the action $A_t$ from $\pi_\omega(\cdot \mid S_t)$; then with probability
$1 - \gamma_\omega(S_{t+1})$ continue, drawing $A_{t+1}$ from the same policy, or
else terminate. A single primitive action is the special case of an option whose
policy always picks that action and which terminates after one step. Options
therefore _extend_ the action space rather than replacing it: the agent may pick a
one-step action or an extended option that runs for many steps before returning
control.[^sb-options]

> **Definition (Option).** A temporally extended action $\omega = (\pi_\omega,
> \gamma_\omega)$, comprising an intra-option policy $\pi_\omega$ and a termination
> function $\gamma_\omega(s)$ giving the probability the option ends in state $s$. A
> primitive action is an option with a one-state policy and immediate termination.

$$
% caption: An option $\omega = (\pi_\omega, \beta)$ spans several primitive time
% steps. Executing it draws actions from $\pi_\omega$ at each step and terminates
% with probability $\beta = \gamma_\omega(s)$; here it runs from $s_t$ through
% four steps before ending at $s_{t+4}$, which is one decision for the policy
% above it.
\begin{tikzpicture}[>=stealth, font=\small,
  st/.style={circle, draw, minimum size=8mm, inner sep=0pt, font=\footnotesize},
  term/.style={circle, draw=acc, thick, minimum size=8mm, inner sep=0pt, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[st]   (s0) at (0,0)   {s(t)};
  \node[st]   (s1) at (2.2,0) {s+1};
  \node[st]   (s2) at (4.4,0) {s+2};
  \node[st]   (s3) at (6.6,0) {s+3};
  \node[term] (s4) at (8.8,0) {s+4};
  \draw[->, thick] (s0) -- node[midway, above, font=\scriptsize] {a(t)} (s1);
  \draw[->, thick] (s1) -- node[midway, above, font=\scriptsize] {a+1} (s2);
  \draw[->, thick] (s2) -- node[midway, above, font=\scriptsize] {a+2} (s3);
  \draw[->, thick] (s3) -- node[midway, above, font=\scriptsize] {a+3} (s4);
  % option span bracket above
  \draw[acc, thick] (0,1.1) -- (0,1.3) -- (8.8,1.3) -- (8.8,1.1);
  \node[acc, anchor=south, font=\footnotesize] at (4.4,1.35) {one option omega = (policy, beta), one decision above};
  \node[font=\scriptsize, text=black, anchor=north] at (0,-0.75) {initiate};
  \node[acc, font=\scriptsize, anchor=north] at (8.8,-0.75) {terminate (beta = 1)};
\end{tikzpicture}
$$

Because an option is interchangeable with an action, everything built on actions
generalizes. The action-value function $q_\pi$ becomes an **option-value** function
that returns the expected return of running an option to termination and then
following $\pi$. A policy becomes a **hierarchical policy** that selects among
options; the chosen option runs to termination before the next choice. In the
simplest learning scheme the update _jumps_ from option initiation to option
termination, treating the whole extended run as one transition; intra-option methods
update on every intermediate step instead, at the cost of requiring off-policy
learning.[^sb-options]

The deeper benefit is a model that spans time. A conventional model gives, for each
action, the next state and the expected immediate reward. An **option model** gives,
for each option, the expected cumulative discounted reward earned along the way and
a discounted distribution over where the option lands and when. The reward part is

$$
r(s, \omega) \;\doteq\; \mathbb{E}\!\left[\, R_1 + \gamma R_2 + \cdots +
\gamma^{\,\tau - 1} R_\tau \;\middle|\; S_0 = s,\; A_{0:\tau-1} \sim \pi_\omega,\;
\tau \sim \gamma_\omega \right],
$$

where $\tau$ is the random step at which the option terminates. The overall
discount $\gamma$ discounts reward along the way, while $\gamma_\omega$ governs
termination — two distinct roles for two distinct discounts. The state part is
subtler: because an option can terminate after any number of steps, each landing
state is discounted by its own $\gamma^k$, so the resulting quantity $p(s' \mid s,
\omega)$ is no longer a transition probability and no longer sums to one. With these
option models one can write Bellman equations and value-iteration algorithms that
plan over options and ordinary actions alike, and planning is faster because each
option can leap over many steps.[^sb-options]

One natural way to _learn_ an option model is to phrase it as a collection of GVFs
and learn them with the methods above: choose the cumulant to be reward for the
reward part, and a discounted transition indicator for the state part. So the two
frontiers meet — GVFs are the mechanism by which temporal abstraction becomes
learnable — though assembling all the pieces, with function approximation, remains
beyond the current state of the art.[^sb-options]

**An option model, worked.** Take a "walk to the door" option that runs a
deterministic four-step path collecting rewards $R_1 = 0$, $R_2 = 0$, $R_3 = -1$
(bumped a wall), $R_4 = +10$ (reached the door), then terminates at step $\tau = 4$.
With $\gamma = 0.9$ the option's reward-model entry is the discounted sum along the
way:

$$
r(s, \omega) = R_1 + \gamma R_2 + \gamma^2 R_3 + \gamma^3 R_4
= 0 + 0 + 0.81(-1) + 0.729(10) = -0.81 + 7.29 = 6.48.
$$

The state-model entry for the landing state $s'$ (the doorway) is
$\gamma^{\tau} = 0.9^4 = 0.656$ — not a probability but a discounted weight, which is
why it lands below $1$ even though the option reaches $s'$ with certainty. Now suppose
the hierarchical policy above compares this option against a "go to the window" option
with model value $r(s, \omega') = 3.0$ and landing weight $0.9^2 = 0.81$ onto a state
$s''$ whose value is $v(s'') = 4.0$, versus the door's landing value $v(s') = 5.0$.
The option-value of each is $r(s, \omega) + \gamma^{\tau} v(\text{landing})$:

$$
q(s, \omega) = 6.48 + 0.656(5.0) = 9.76,
\qquad
q(s, \omega') = 3.0 + 0.81(4.0) = 6.24.
$$

The door option wins, $9.76$ against $6.24$, and the hierarchical policy makes _one_
decision to commit to four primitive steps. Planning over
options collapses a four-step lookahead into a single Bellman backup, which is
why option models make planning faster — the option leaps over its own internal steps.

$$
% caption: One option-value backup. Each option's model gives a discounted reward
% along the way plus a discounted weight onto its landing state's value; the
% hierarchical policy compares the totals and commits to the whole option in a single
% decision. Here the door option (9.76) beats the window option (6.24).
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  o/.style={draw, minimum width=34mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[o, draw=acc, text=acc] (door) at (0,1.0) {door: 6.48 + 0.656 x 5.0 = 9.76};
  \node[o] (win) at (0,-1.0) {window: 3.0 + 0.81 x 4.0 = 6.24};
  \node[draw, minimum width=20mm, minimum height=9mm, align=center, font=\scriptsize] (pick) at (5.6,0) {pick max\\= door};
  \draw[->, acc, thick] (door.east) -- (pick.west);
  \draw[->, black] (win.east) -- (pick.west);
\end{tikzpicture}
$$

## Observations and state

Every learned value function so far has been a function of the environment's
_state_. In Part I that state lived in a table, which is to assume the environment
is fully observed — any state distinguishable in principle gets its own cell. Real
sensory input is not like that. A robot's camera sees part of a room; objects are
occluded, behind the agent, or miles away. Important aspects of the world are simply
not in the observation, and treating a value function as a table over true states is
a strong and unrealistic assumption.[^sb-state]

Parametric [function approximation](/reinforcement-learning/approximation/on-policy-prediction)
already softens this. If a state variable is not observable, the parameterization can
be chosen not to depend on it, and the value behaves as if that variable were absent
— so the approximate-value results carry over to partial observability unchanged. But
some issues need partial observability made explicit. Sutton and Barto lay out four steps.

**First, change the problem.** The environment emits not states but **observations**
$O_t \in \mathcal{O}$ — signals that depend on the underlying state but reveal it only
in part. The interaction becomes an alternating stream of actions and observations,
$A_0, O_1, A_1, O_2, A_2, O_3, \ldots$, with no state anywhere in the data.

**Second, recover state as a construction.** Define the **history** $H_t \doteq A_0,
O_1, \ldots, A_{t-1}, O_t$: everything knowable from the data stream so far. The
history is the most any agent could know, but it grows without bound. **State** is a
compact summary of history that is as good as the full history for predicting the
future. That "as good as" is precise. A summarizing function $S_t = f(H_t)$ has the
**Markov property** when any two histories it maps to the same state yield the same
distribution over the next observation:

$$
f(h) = f(h') \;\;\Rightarrow\;\; \Pr\{O_{t+1} = o \mid H_t = h,\, A_t = a\}
= \Pr\{O_{t+1} = o \mid H_t = h',\, A_t = a\}.
$$

When $f$ has this property, $S_t = f(H_t)$ is a **Markov state** — and the crucial
point is that the Markov state is _built_ by choosing $f$, not read off the world. A
Markov state summarizes not only the next observation but every future _test_ (any
sequence of actions and resulting observations), and hence every GVF; it is enough
to predict and control anything.[^sb-state]

$$
% caption: State is a construction. The agent never sees the environment state; it
% sees a stream of observations $O_t$ and actions $A_t$, and a state-update
% function $u$ folds each new observation into a compact state $S_t$ that is fed
% to the value function and back into $u$. When $u$ yields the Markov property,
% $S_t$ is as good as the whole history.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=16mm, minimum height=10mm, align=center},
  world/.style={draw, thick, minimum width=26mm, minimum height=12mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[world] (w) at (0,0) {world};
  \node[box] (u) at (0,2.4) {update u};
  \node[box, draw=acc, text=acc] (v) at (5.0,2.4) {value, policy};
  \draw[->, thick] (w.north west) -- node[midway, left, font=\scriptsize] {O} ([xshift=-6mm]u.south);
  \draw[->, thick] ([xshift=6mm]u.south) -- node[midway, right, font=\scriptsize] {A} (w.north east);
  \draw[->, acc, thick] (u.east) -- node[midway, above, font=\scriptsize] {state S} (v.west);
  \draw[->, thick] ([yshift=1mm]u.north east) .. controls (2.4,4.2) and (-2.4,4.2) .. ([yshift=1mm]u.north west)
    node[midway, above, font=\scriptsize] {S folds forward};
\end{tikzpicture}
$$

**Third, make it compact and incremental.** The identity function $S_t = H_t$ is
Markov but useless: it grows forever and never recurs, so no state is ever seen
twice. Instead the agent maintains a **state-update function** that folds each new
observation into the previous state,

$$
S_{t+1} \;\doteq\; u(S_t, A_t, O_{t+1}), \qquad t \ge 0,
$$

with $S_0$ given. Learning a good $u$ — one that is compact, recurrent, and
near-Markov — is the representation-learning problem, restated for partial
observability. Two classical constructions illustrate this. A **POMDP** posits a
hidden latent state and takes $S_t$ to be the **belief state**, the posterior
distribution over latent states given the history, updated by Bayes' rule; it is
principled but scales poorly. A **predictive state representation** instead defines
the state as the probabilities of a set of future tests, grounding its meaning in
observable data rather than an unobservable latent, which can make it easier to
learn.[^sb-state]

**A belief update, worked.** The belief state is a probability vector over hidden
states, and Bayes' rule folds each observation into it. Take a two-state tiger
problem: the reward is behind the left door or the right, and the agent starts
believing $b = (0.5,\ 0.5)$. A "listen" action returns a noisy observation: it hears
the correct side with probability $0.85$ and the wrong side with $0.15$. Suppose it
hears "left." The Bayes update multiplies each prior by that state's likelihood of
producing the observation and renormalizes:

$$
b'(\text{left}) = \frac{0.85 \times 0.5}{0.85 \times 0.5 + 0.15 \times 0.5}
= \frac{0.425}{0.500} = 0.85,
\qquad
b'(\text{right}) = 0.15.
$$

One noisy observation moved the belief from $(0.5, 0.5)$ to $(0.85, 0.15)$. Listen
again and hear "left" a second time: the new prior is $(0.85, 0.15)$, so

$$
b''(\text{left}) = \frac{0.85 \times 0.85}{0.85 \times 0.85 + 0.15 \times 0.15}
= \frac{0.7225}{0.7450} = 0.970.
$$

The belief sharpens toward certainty with each confirming observation, and this
belief vector — not the raw observation — is the Markov state a POMDP planner acts
on. The state-update function $u$ here is precisely the Bayes filter; a predictive
state representation would instead track the probabilities of future "listen"
outcomes and reach an equivalent summary without ever naming the hidden door.

$$
% caption: A belief-state update. Starting from uniform belief over which door hides
% the reward, each noisy "left" observation multiplies the prior by the observation
% likelihood and renormalizes (Bayes' rule), sharpening the belief toward the left
% door. The belief vector, not the raw observation, is the Markov state.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bs/.style={draw, minimum width=20mm, minimum height=13mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[bs] (b0) at (0,0)   {b = (0.50, 0.50)\\uniform};
  \node[bs] (b1) at (3.8,0) {b = (0.85, 0.15)};
  \node[bs, draw=acc, text=acc] (b2) at (7.6,0) {b = (0.97, 0.03)};
  \draw[->, thick] (b0) -- node[above, font=\scriptsize] {hear left} (b1);
  \draw[->, thick] (b1) -- node[above, font=\scriptsize] {hear left} (b2);
  \node[font=\scriptsize, black, anchor=north] at (3.8,-1.1) {Bayes f\/ilter = state update u};
\end{tikzpicture}
$$

**Fourth, re-introduce approximation.** In practice $u$ produces an approximate
state that may not be Markov — the simplest being the last observation $S_t = O_t$,
or the last $k$ observations and actions. When the Markov property holds only
approximately, long-horizon predictions can degrade sharply, and there are no strong
theoretical guarantees at present. What remains is a heuristic that echoes the
auxiliary-task argument: a representation that is good for some predictions tends to
be good for others. So partial observability and representation learning fold into
one problem — learn a state-update function by pursuing many predictions and keeping
the features that serve them.[^sb-state]
This lesson loosened the state and the action. The third assumption — the reward, its
design, and the reward-hacking hazard — together with the open problems the whole
subject still faces and how the frontiers were pushed after Sutton & Barto, continues in
[Reward Design and Open Problems](/reinforcement-learning/deep-rl/reward-design-and-open-problems).

[^sb-intro]: **Sutton & Barto**, Ch. 17 — Frontiers: the closing chapter's framing as topics beyond what is reliably known, several of which move past the fixed-state, scalar-reward, single-step MDP assumptions used throughout the text.
[^sb-gvf]: **Sutton & Barto**, §17.1 — General Value Functions and Auxiliary Tasks: the generalization from reward to an arbitrary cumulant $C_t$ under a target policy $\pi$ and state-dependent termination $\gamma(s)$, the GVF definition (17.1), and the point that a GVF is a prediction or forecast that ordinary value-function methods can learn without change.
[^sb-aux]: **Sutton & Barto**, §17.1 — auxiliary tasks: the multi-head ANN sharing one body across several GVFs, easier auxiliary tasks surfacing features that speed the main task, the empirical acceleration reported by Jaderberg et al. (2017), and the classical-conditioning analogy of a built-in reflex triggered by a learned prediction.
[^sb-options]: **Sutton & Barto**, §17.2 — Temporal Abstraction via Options: the option $\omega = (\pi_\omega, \gamma_\omega)$ as a temporally-extended action, primitive actions as one-step options, option-value functions and hierarchical policies, the option model's reward part (17.2) and discounted state part (17.3), the option Bellman/value-iteration equations (17.4), and learning option models as collections of GVFs.
[^sb-state]: **Sutton & Barto**, §17.3 — Observations and State: the four-step extension to partial observability — observations $O_t$ replacing states, the history $H_t$ and the Markov property (17.5) that makes $S_t = f(H_t)$ a Markov state, the state-update function (17.8), and the POMDP belief state (17.9) and predictive state representations as instances, plus the approximate case and its link to representation learning.