---
title: "Safe RL: Risk, Safe Exploration, and Alignment"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 21
order: 521
summary: >
  A companion to the constrained-MDP lesson. Constraining the mean cost is not enough:
  a policy safe on average can be catastrophic in the tail, and a policy safe at
  convergence can violate its limits wildly while learning. This lesson optimizes the
  tail with risk-sensitive objectives (CVaR), then makes exploration itself safe with
  shields, Lyapunov methods, and safety layers that project unsafe actions onto the
  feasible set — closing with benchmarks, safe RLHF, robustness, and the alignment
  framing that ties safety back to the problem of incompletely specified reward.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "Ch. 13 — Policy Gradient Methods (background for the constrained policy-optimization stage)"
  - book: Grokking Deep RL
    ref: "Ch. 11–12 — policy-based and actor-critic methods (trust-region and penalty background)"
---

This builds on
[Safe and Constrained RL: The CMDP and Policy Methods](/reinforcement-learning/modern-deep-rl/safe-and-constrained-rl),
which argued that reward maximization alone invites specification gaming, formalized the
honest alternative as a **constrained MDP** (maximize return subject to a cost budget),
and solved it two ways: Lagrangian primal-dual with a learned multiplier (RCPO), and
constrained policy optimization with a trust-region cost bound (CPO).

Those methods bound the _expected_ cost at convergence. Two gaps remain. A policy safe
on average can still be ruinous in its worst episodes, and a policy that ends up safe
can behave dangerously _while it is still learning_. This lesson closes both gaps and
then connects safety to the alignment framing.

## Risk-sensitive RL: optimizing the tail

Constraints bound the _expected_ cost, but expectation is blind to variance. A
policy that is safe on average can still visit a catastrophe on a small fraction of
episodes, and for high-stakes deployments the tail is the whole point: a self-driving
policy with a great mean and a rare fatal outcome is unacceptable, and no bound on
$\mathbb{E}[\text{cost}]$ rules it out. **Risk-sensitive RL** optimizes a functional
of the _return distribution_ that weights bad outcomes more heavily than the mean
does.[^risk]

This connects directly to
[distributional RL](/reinforcement-learning/modern-deep-rl/distributional-and-rainbow),
which already models the full distribution $Z^\pi(s,a)$ of returns rather than only
its mean $Q^\pi(s,a) = \mathbb{E}[Z^\pi]$. Once you have the distribution, you can
optimize any risk measure of it. The standard choice is **Conditional Value at
Risk**. Let $Z$ be the return (a random variable) and fix a tail level
$\alpha \in (0,1)$. The value-at-risk $\mathrm{VaR}_\alpha$ is the $\alpha$-quantile
of $Z$ — the return you do at least as well as with probability $1-\alpha$ — and
$\mathrm{CVaR}_\alpha$ is the _mean of the returns below that quantile_:

$$
\mathrm{CVaR}_\alpha(Z)
\;=\;
\mathbb{E}\!\bigl[\, Z \;\big|\; Z \le \mathrm{VaR}_\alpha(Z) \,\bigr].
$$

It is the average outcome over the worst $\alpha$ fraction of episodes. Maximizing
$\mathrm{CVaR}_\alpha$ makes the policy improve its _bad_ cases, not its average
one. As $\alpha \to 1$ the whole distribution is in the tail and CVaR reduces to
the ordinary expectation (risk-neutral); as $\alpha \to 0$ it concentrates on the
single worst outcome (worst-case, robust). The knob $\alpha$ dials continuously
between average-case and worst-case optimization.

$$
% caption: The return distribution and its CVaR tail. The alpha-quantile VaR marks
% the boundary of the worst alpha fraction of outcomes (shaded left tail); CVaR is
% the mean return within that tail, well below the distribution's mean. Optimizing
% CVaR improves the shaded region rather than the average.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (9.4,0) node[anchor=north, black, font=\footnotesize] {return Z};
  \draw[->, black] (0,-0.2) -- (0,4.0) node[anchor=east, black, font=\footnotesize] {density};
  % a skewed bell curve
  \draw[black, thick, smooth]
    plot coordinates {(0.5,0.15)(1.3,0.5)(2.1,1.2)(2.9,2.3)(3.7,3.2)(4.5,3.55)(5.3,3.2)(6.1,2.4)(6.9,1.5)(7.7,0.8)(8.5,0.35)};
  % tail shading up to VaR
  \fill[red!12]
    plot coordinates {(0.5,0.15)(1.3,0.5)(2.1,1.2)(2.6,1.75)} -- (2.6,0) -- (0.5,0) -- cycle;
  % VaR line
  \draw[red, thick, dashed] (2.6,0) -- (2.6,3.7);
  \node[red, anchor=south, font=\scriptsize] at (2.6,3.7) {VaR (alpha-quantile)};
  % CVaR marker inside the tail
  \draw[acc, thick] (1.75,0) -- (1.75,2.1);
  \fill[acc] (1.75,0) circle (2.2pt);
  \node[acc, anchor=south east, font=\scriptsize] at (1.75,2.1) {CVaR (tail mean)};
  % mean marker
  \draw[black, dotted] (4.7,0) -- (4.7,3.55);
  \node[black!70, anchor=south, font=\scriptsize] at (4.75,3.55) {mean};
  % tail label
  \node[red, anchor=north, font=\scriptsize] at (1.55,-0.05) {worst alpha fraction};
\end{tikzpicture}
$$

> **Definition (Conditional Value at Risk).** For a return $Z$ and level
> $\alpha \in (0,1)$, $\mathrm{CVaR}_\alpha(Z) = \mathbb{E}[Z \mid Z \le
> \mathrm{VaR}_\alpha(Z)]$, the mean of the worst $\alpha$-fraction of outcomes,
> where $\mathrm{VaR}_\alpha$ is the $\alpha$-quantile of $Z$. Maximizing
> $\mathrm{CVaR}_\alpha$ optimizes the lower tail of the return; $\alpha \to 1$
> recovers the risk-neutral expectation and $\alpha \to 0$ the worst case.

CVaR has two properties that make it the default risk measure. It is a **coherent**
risk measure (it respects diversification and monotonicity in a precise sense that
variance does not), and it has a variational form,

$$
\mathrm{CVaR}_\alpha(Z)
\;=\;
\max_{\nu}\; \Bigl\{\, \nu \;-\; \tfrac{1}{\alpha}\,\mathbb{E}\bigl[(\nu - Z)_+\bigr]\,\Bigr\},
$$

that turns CVaR optimization into a tractable stochastic program — the auxiliary
variable $\nu$ converges to $\mathrm{VaR}_\alpha$, and the objective is
differentiable in the policy parameters, so it plugs into a policy-gradient method.
A distributional critic that already estimates the quantiles of $Z^\pi$ supplies
everything the CVaR objective needs; risk-sensitivity is then almost free on top of
a distributional agent.

#### Worked example: CVaR on a discrete return distribution

Compute CVaR by hand to see how differently it ranks policies from the mean. Take
two policies whose returns are ten equally-likely outcomes each:

- Policy A (steady): returns $\{6, 7, 7, 8, 8, 8, 9, 9, 10, 8\}$, mean $= 8.0$.
- Policy B (gamble): returns $\{-40, 12, 12, 13, 13, 14, 14, 14, 15, 13\}$, mean
  $= 8.0$.

Both have the same mean $8.0$, so a risk-neutral objective is indifferent. Now take
$\alpha = 0.1$, the worst $10\%$ — one outcome out of ten. For A the single worst
outcome is $6$, so $\mathrm{CVaR}_{0.1}(A) = 6$. For B it is $-40$, so
$\mathrm{CVaR}_{0.1}(B) = -40$. The tail measure ranks A vastly above B —
$6$ versus $-40$ — even though their means are identical, because B pays for its
higher typical return with a rare catastrophe. Widen the tail to $\alpha = 0.3$
(the worst three of ten): A averages $(6 + 7 + 7)/3 \approx 6.67$; B averages
$(-40 + 12 + 12)/3 \approx -5.33$. As $\alpha \to 1$ both CVaRs climb back to the
shared mean $8.0$ and the distinction vanishes. The knob $\alpha$ sets how
much of the bad tail you insist on optimizing: a self-driving policy graded at
$\alpha = 0.01$ is forced to fix its one-in-a-hundred worst episode, which is the
one that matters.

## Safe exploration

Constraints and risk measures shape the policy the agent _converges_ to. They say
nothing about the actions it takes _while learning_, and that is where physical
systems get damaged. A robot that must not exceed a joint torque cannot afford to
discover the limit by exceeding it once. **Safe exploration** enforces constraints
_during_ learning, on every action, not merely on the final policy. Three
mechanisms dominate.

$$
% caption: Three safe-exploration mechanisms wrapping a base RL agent. A shield
% vetoes actions that violate a logical safety spec; a Lyapunov constraint keeps
% the agent inside a certified region of state space; a safety layer projects an
% unsafe proposed action onto the nearest safe one before it reaches the world.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, minimum width=26mm, minimum height=12mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[b, draw=acc, text=acc, thick] (agent) at (0,0) {RL agent\\proposes action};
  \node[b] (filter) at (4.3,0) {safety filter\\(shield / layer)};
  \node[b] (env) at (8.6,0) {environment};
  \draw[->, acc, thick] (agent) -- (filter) node[midway, above, font=\scriptsize] {a-raw};
  \draw[->, acc, thick] (filter) -- (env) node[midway, above, font=\scriptsize] {a-safe};
  \draw[->, black] (env.south) .. controls (7.0,-1.8) and (1.6,-1.8) .. (agent.south)
    node[midway, below, black!70, font=\scriptsize] {state, reward, cost};
  % the three mechanisms feeding the filter
  \node[b, draw=red, text=red] (spec) at (2.15,2.3) {shield: logical\\safety spec};
  \node[b] (lyap) at (5.0,2.3) {Lyapunov region\\/ safety layer};
  \draw[->, red!70] (spec.south) -- (filter.north);
  \draw[->, black] (lyap.south) -- ([xshift=6mm]filter.north);
\end{tikzpicture}
$$

### Shielding

A **shield** is a reactive safety layer synthesized from a formal specification of
what must never happen — a temporal-logic formula like "never enter a collision
state," "never let the tank run dry."[^shield] At each step the shield inspects the
action the agent proposes and, if that action could lead to a specification
violation within its horizon, it overrides it with a safe alternative. The agent
learns normally, receiving the state and reward as usual, but its actions are
filtered through a correctness monitor that is provably safe by construction. The
shield ignores reward entirely; it encodes only the safety automaton, and it
guarantees the spec is never violated regardless of what the still-learning policy
proposes. The limitation is that you must be able to _write_ the specification and model
the dynamics well enough to check it — feasible for discrete or well-understood
systems, hard for high-dimensional continuous ones.

### Lyapunov methods

A second approach borrows the **Lyapunov function** from control theory. A Lyapunov
function $L(s)$ is a scalar "energy" that certifies stability: if you can guarantee
$L$ never increases along the agent's trajectories, the agent is trapped inside a
sublevel set $\{s : L(s) \le \ell\}$ — a certified safe region it can never leave.
Lyapunov-based safe RL[^lyapunov] constructs such a function for the cost constraint
and restricts the policy at each state to the actions that keep $L$ from growing,

$$
\pi(s) \in \bigl\{\, a : L\bigl(s'\bigr) \le L(s) \text{ in expectation under } a \,\bigr\},
$$

which is a per-state constraint on the action set rather than a bound on the
episodic cost. The guarantee is that the closed-loop system stays within the
certified region for _all_ time, giving the same during-learning safety a shield
does but derived from the cost dynamics instead of a logical spec. The difficulty
is constructing a valid Lyapunov function; much of the work is in learning or
approximating one that is both safe and not so conservative that it blocks the
task.

### Safety layers

The most direct mechanism assumes a differentiable model of the cost and _projects_
each proposed action onto the safe set. A **safety layer**[^safelayer] sits between
the policy and the environment: the policy proposes an action $a_{\text{raw}}$, and
the layer solves a small optimization to find the closest action that satisfies the
per-step cost constraint,

$$
a_{\text{safe}}
\;=\;
\argmin_{a}\; \tfrac12\,\lVert a - a_{\text{raw}} \rVert^2
\quad\text{s.t.}\quad
\hat c(s) + g(s)^\top a \;\le\; d,
$$

where $\hat c(s) + g(s)^\top a$ is a linear model of the next-step cost learned from
data. With a single linear constraint this projection has a closed-form solution —
the layer either passes $a_{\text{raw}}$ through unchanged (if it is already safe)
or slides it along the constraint's gradient just far enough to satisfy the budget.
Because the correction is analytic and differentiable, gradients still flow back to
the policy, which learns to propose actions that need little correction. The safety
layer is the continuous-control counterpart to the shield: where the shield vetoes
discrete actions against a logical spec, the layer projects continuous actions
against a learned cost model.

$$
% caption: A safety layer projecting an unsafe action. The proposed action a-raw
% lies outside the safe set (the half-plane where next-step cost stays under d);
% the layer returns a-safe, the nearest point on the safe boundary, by sliding
% a-raw along the constraint gradient. Safe proposals pass through untouched.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes for a 2-D action
  \draw[->, black] (-0.2,0) -- (6.4,0) node[anchor=north, black, font=\scriptsize] {action dim 1};
  \draw[->, black] (0,-0.2) -- (0,4.6) node[anchor=east, black, font=\scriptsize] {action dim 2};
  % safe set: half-plane below the constraint line
  \fill[acc!10] (0,0) -- (5.6,0) -- (0,3.7) -- cycle;
  \draw[red, thick] (5.6,0) -- (0,3.7);
  \node[red, anchor=west, font=\scriptsize] at (3.2,1.9) {cost = d};
  \node[acc, font=\scriptsize] at (1.5,1.0) {safe set};
  % unsafe proposed action (outside)
  \fill[red] (4.6,3.1) circle (2.4pt);
  \node[red, anchor=west, font=\scriptsize] at (4.7,3.1) {a-raw (unsafe)};
  % projected safe action on the boundary (nearest point)
  \coordinate (asafe) at (2.85,2.18);
  \fill[acc] (asafe) circle (2.4pt);
  \node[acc, anchor=south east, font=\scriptsize] at (2.75,2.25) {a-safe};
  \draw[->, acc, very thick] (4.6,3.1) -- (asafe);
  \node[black, font=\scriptsize, anchor=west] at (3.55,2.75) {projection};
\end{tikzpicture}
$$

The three mechanisms trade off generality against assumptions. A shield needs a
logical spec and a checkable model; Lyapunov methods need a certified energy
function; a safety layer needs a differentiable cost model. Each provides what
the constrained and risk-sensitive methods do not: safety _during_ learning,
enforced action by action.

| Mechanism | What it needs | What it guarantees |
| --- | --- | --- |
| Shield | temporal-logic spec + discrete model | never violates the logical spec |
| Lyapunov | a certified $L(s)$ for the cost | stays inside a certified region for all time |
| Safety layer | differentiable per-step cost model | each action satisfies the per-step cost budget |
| Lagrangian / CPO | only a cost signal and threshold $d$ | expected cost $\le d$ (asymptotic / near-monotonic) |

## Benchmarks and safe RLHF

The methods above are a decade of theory; two later developments ground them in
practice and connect the thread back to the RLHF lesson.

**Standardized safe-RL benchmarks.** For years constrained RL lacked a common
testbed, and every paper reported on its own tasks. **Safety Gym** fixed this: a
suite of continuous-control navigation tasks where an agent must reach goals while a
_separate_ cost signal counts contacts with hazards, and every method is measured on
the same reward-versus-cost frontier.[^safetygym] Its empirical finding is worth
stating plainly, because it tempers the theory: a well-tuned **PPO-Lagrangian**
baseline — plain PPO with the primal-dual multiplier of this lesson bolted on — is
hard to beat, and CPO's per-step guarantee often does not yield a better final
frontier despite its stronger theory. The practical default in constrained deep RL
is the Lagrangian method, not the trust-region one; CPO is preferred only when
violations _during_ training must be bounded, not merely at convergence.

$$
% caption: The reward-cost frontier that safe-RL benchmarks report. Each method
% traces a curve of achievable reward against incurred cost; the constraint is a
% vertical line at the cost budget d, and the score is the highest reward on or left
% of it. A method dominates if its frontier sits up and to the left. PPO-Lagrangian
% and CPO reach similar frontiers; the difference is how safe they stay while getting
% there.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (7.4,0) node[anchor=north, black] {incurred cost};
  \draw[->, black] (0,0) -- (0,4.2) node[anchor=east, black] {reward};
  % budget line
  \draw[red, thick, dashed] (4.2,0) -- (4.2,4.0);
  \node[red, anchor=south, font=\scriptsize] at (4.2,4.0) {budget d};
  % infeasible region shading (draw first so curve sits on top)
  \fill[red!7] (4.2,0) rectangle (7.2,4.0);
  \node[red!70, anchor=north, font=\scriptsize] at (5.7,0.9) {over budget};
  % frontier curve (reward rises with allowed cost, saturating)
  \draw[acc, very thick, smooth] plot coordinates {
    (0.3,0.5)(1.2,1.6)(2.2,2.5)(3.2,3.1)(4.2,3.45)(5.4,3.7)(6.6,3.85)};
  \node[acc, anchor=north, font=\scriptsize] at (2.4,1.8) {reward-cost frontier};
  % operating point at the budget
  \fill[acc] (4.2,3.45) circle (2.4pt);
  \node[black, anchor=south east, font=\scriptsize] at (4.1,3.5) {best feasible point};
\end{tikzpicture}
$$

**Safe RLHF.** The constrained formalism came back to language models directly. The
[RLHF lesson](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models)
optimized a single reward that had to bundle helpfulness and harmlessness together,
which pits the two against each other inside one scalar. **Safe RLHF** splits them:
it trains a _separate_ helpfulness reward model and a _cost_ model for harm, then
optimizes helpfulness subject to a harmlessness constraint using exactly the
Lagrangian primal-dual machinery of this lesson — a learned multiplier trading
helpfulness for safety on a slow timescale.[^saferlhf] The over-optimization the
RLHF lesson controlled with a KL leash is, in this framing, one constraint among
several, and the CMDP is the right container for all of them. The boat spinning in
the lagoon and a language model that is helpful but unsafe are the same failure, and
the same constrained-optimization discipline answers both.

## Robustness and the alignment framing

Two threads sit just outside the CMDP formalism but belong to the same concern:
making a policy behave when the world is not the training distribution.

The first is **sim-to-real robustness**. Policies trained in simulation exploit the
simulator's quirks and fail on the real system whose friction, latency, and mass
differ from the model — a distribution shift that shows up as a safety failure.
**Domain randomization**[^domainrand] addresses it directly: randomize the simulator's
physics parameters (masses, friction, delays, sensor noise) across a wide range
during training, so the policy must succeed on a whole _family_ of dynamics rather
than one. A policy robust across the randomized family tends to treat the real
system as just another sample from it. This is a form of worst-case optimization —
optimizing for the hardest dynamics you might face is a robust objective, related to
the CVaR tail, and it is why domain randomization and risk-sensitive RL are two
answers to one question: how to be safe under uncertainty you cannot fully model.

The second is the broader **AI-safety / alignment** framing. Every technique here
is a response to the same fact: **a scalar reward is
an incomplete statement of intent, and a competent optimizer will exploit the
incompleteness.** Specification gaming, reward hacking, and over-optimization are
one phenomenon seen at different scales — a boat spinning in a lagoon, a language
model padding an answer, a cleaning robot hiding the mess — and the constrained,
risk-sensitive, and shielded methods are the engineering responses. The
[RLHF lesson](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models)'s KL
leash and this lesson's cost budget are the same idea pointed at different proxies:
_bound how hard you push on a measure, because the measure stops being right before
the optimizer stops pushing._ As RL systems act in the world with real
consequences, that discipline stops being an add-on and becomes the design problem.

## The through-line: reward is not the whole story

The unconstrained objective assumed the reward said everything. The methods here
each admit that it does not and add the missing structure back.

| Approach | What it adds beyond the reward |
| --- | --- |
| CMDP | an explicit cost signal $c_t$ and a budget $d$; feasibility as a first-class goal |
| Lagrangian / RCPO | a learned price $\lambda$ that self-tunes to hit $J_c = d$ |
| CPO | a trust-region cost bound giving approximate feasibility every step |
| CVaR / risk-sensitive | optimize the lower tail of the return, not its mean |
| Shield / Lyapunov / safety layer | per-action safety enforced _during_ learning |
| Domain randomization | robustness to dynamics the training distribution did not cover |

The lesson's one rule, stated plainly: **real deployments must respect constraints
the reward does not capture.** Reward maximization is a means, not the goal, and a
policy that maximizes reward while violating a limit the reward failed to mention
has not succeeded — it has found the gap between what you wrote and what you meant.
It shares a principle with [offline RL](/reinforcement-learning/modern-deep-rl/offline-rl),
where the discipline was pessimism about values the data cannot verify; here it is
pessimism about behaviors the reward cannot see. In both, the safe move is to
_distrust the parts of the objective you have no way to check._


[^risk]: **Chow, Ghavamzadeh, Janson, Pavone (2017)**, "Risk-Constrained Reinforcement Learning with Percentile Risk Criteria", JMLR — develops policy-gradient and actor-critic methods for CVaR objectives and constraints, using the variational (Rockafellar–Uryasev) form of CVaR.
[^shield]: **Alshiekh, Bloem, Ehlers, Könighofer, Niekum, Topcu (2018)**, "Safe Reinforcement Learning via Shielding", AAAI — synthesizes a reactive shield from a temporal-logic safety specification that overrides unsafe actions, guaranteeing the spec holds throughout learning.
[^lyapunov]: **Chow, Nachum, Duenez-Guzman, Ghavamzadeh (2018)**, "A Lyapunov-based Approach to Safe Reinforcement Learning", NeurIPS — constructs Lyapunov functions for CMDP constraints and restricts the policy to actions that keep the agent inside a certified feasible region during and after training.
[^safelayer]: **Dalal, Dvijotham, Vecerik, Hester, Paduraru, Tassa (2018)**, "Safe Exploration in Continuous Action Spaces", arXiv:1801.08757 — adds a differentiable safety layer that analytically projects each proposed action onto the set satisfying a learned linear per-step cost model.
[^domainrand]: **Tobin, Fong, Ray, Schneider, Zaremba, Abbeel (2017)**, "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World", IROS — randomizes simulator appearance and dynamics during training so a policy robust across the family transfers to the unseen real system.
[^safetygym]: **Ray, Achiam, Amodei (2019)**, "Benchmarking Safe Exploration in Deep Reinforcement Learning" (Safety Gym), OpenAI — introduces a standardized suite of constrained continuous-control tasks with a separate cost signal, and finds a tuned PPO-Lagrangian baseline competitive with or better than CPO on the final reward-cost frontier, establishing the Lagrangian method as the practical default.
[^saferlhf]: **Dai, Pan, Sun, et al. (2023)**, "Safe RLHF: Safe Reinforcement Learning from Human Feedback", arXiv:2310.12773 (ICLR 2024) — decouples helpfulness and harmlessness into a separate reward model and cost model, then optimizes helpfulness subject to a harmlessness constraint via Lagrangian primal-dual updates, applying the CMDP formalism directly to language-model alignment.