---
title: "Safe and Constrained RL: The CMDP and Policy Methods"
module: Modern Deep Reinforcement Learning
moduleNumber: 5
lessonNumber: 20
order: 520
summary: >
  Maximizing a scalar reward is not the same as behaving well: a capable
  optimizer will find and exploit any gap between the reward and what its designer
  actually meant, a failure called specification gaming or reward hacking. The
  remedy is to add explicit cost constraints — the constrained MDP —
  maximizing return subject to an expected-cost budget. This lesson builds the
  core toolkit: the CMDP itself, Lagrangian primal-dual methods that learn a
  multiplier on the constraint (RCPO), and constrained policy optimization (CPO)
  with its trust-region cost bound. A companion lesson covers risk-sensitivity,
  safe exploration, and the alignment framing.
topics: [Deep RL]
sources:
  - book: Sutton & Barto
    ref: "Ch. 13 — Policy Gradient Methods (background for the constrained policy-optimization stage)"
  - book: Grokking Deep RL
    ref: "Ch. 11–12 — policy-based and actor-critic methods (trust-region and penalty background)"
---

Everything in the course so far has taken a single instruction on faith: find the
policy that maximizes expected return. That instruction is complete only if the
reward function is a perfect statement of what you want. It almost never is. A
reward is a compression of a designer's intent into one scalar per step, and a
capable optimizer exploits every gap between that scalar and the intent. It
optimizes what you _wrote_, not what you meant, and the two
diverge exactly where you were not looking. A cleaning robot rewarded for a tidy
room learns to sweep dust under the rug; a boat-racing agent rewarded for the
score circles a lagoon collecting the same power-ups forever instead of finishing
the race.[^gaming] These are not optimizer bugs: the optimizer worked, the
reward was wrong, and nothing in reward maximization penalizes the difference.

This lesson is about the discipline that takes the gap seriously: state, alongside
the reward, the **constraints** a good policy must respect — a collision budget, a
power limit, a rule it may not break — and optimize return _subject to_ them. It
shares a principle with [offline RL](/reinforcement-learning/modern-deep-rl/offline-rl),
where the remedy for unverifiable value estimates was pessimism about what the data
cannot confirm; here the remedy for an incomplete reward is pessimism about what the
reward does not measure. And it sharpens the over-optimization story from
[RLHF](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models): the KL
leash there is one instance of the general move this lesson formalizes — bound how
hard you push on a proxy, because the proxy stops being right before the optimizer
stops pushing.

This lesson builds the core machinery: the constrained MDP that states the problem, and
the two policy-optimization methods that solve it — Lagrangian primal-dual (RCPO) and
constrained policy optimization (CPO). Risk-sensitivity, safe exploration, and the
alignment framing continue in a
[companion lesson](/reinforcement-learning/modern-deep-rl/safe-and-constrained-rl-part-2).

## Why reward maximization alone is dangerous

The trouble is structural, not a matter of careless reward design. Call the
reward you wrote the **proxy** $r$ and the outcome you actually care about the
**true objective** $r^\star$. The two agree on the behaviors you had in mind when
you wrote $r$ — the ones near your demonstrations and your imagination — and they
can diverge arbitrarily on behaviors you did not consider. Optimization is a search
for the highest-reward behavior, so it is drawn precisely toward the regions where
$r$ is high for reasons that have nothing to do with $r^\star$. The better the
optimizer, the more reliably it finds them.[^goodhart]

$$
% caption: Specification gaming. The proxy reward $r$ (solid) agrees with the true
% objective $r^\star$ (dashed) on the intended behaviors near the center, then
% diverges upward on unintended ones; a strong optimizer climbs the proxy straight
% into the gap, where measured reward is high and true value has collapsed.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (-0.2,0) -- (9.4,0) node[anchor=north, black, font=\footnotesize] {policy behavior};
  \draw[->, black] (0,-0.2) -- (0,4.6) node[anchor=east, black, font=\footnotesize] {value};
  \fill[acc!8] (2.0,0) rectangle (4.4,4.4);
  \node[acc, font=\footnotesize, anchor=south] at (3.2,4.35) {intended region};
  % true objective: a bump peaking in the intended region, low outside
  \draw[black, thick, dashed, smooth]
    plot coordinates {(0.4,0.9)(1.4,1.5)(2.4,2.4)(3.2,2.75)(4.0,2.5)(4.8,1.9)(6.0,1.2)(7.2,0.85)(8.6,0.7)};
  \node[black, anchor=west, font=\footnotesize] at (8.65,0.7) {true objective};
  % proxy reward: matches inside, then climbs off to the right
  \draw[red, thick, smooth]
    plot coordinates {(0.4,0.9)(1.4,1.5)(2.4,2.4)(3.2,2.75)(4.0,2.6)(4.8,2.5)(6.0,3.2)(7.2,3.9)(8.6,4.3)};
  \node[red, anchor=west, font=\footnotesize] at (5.7,3.4) {proxy reward};
  % optimizer climbs the proxy into the gap
  \fill[red] (8.5,4.28) circle (2.2pt);
  \draw[red!70, ->] (7.4,4.1) to[bend left=12] (8.45,4.24);
  \node[red, font=\footnotesize, anchor=east] at (7.4,4.05) {optimum of proxy};
\end{tikzpicture}
$$

> **Definition (Specification gaming / reward hacking).** Behavior that scores
> highly under the specified reward while violating the designer's intent, arising
> because the reward is an imperfect proxy for the true objective. A more capable
> optimizer exploits the proxy–objective gap more thoroughly, so gaming tends to
> _worsen_ with capability rather than improve.

The pattern is the same one the [RLHF lesson](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models)
named **over-optimization**: a language policy free to maximize a learned reward
model drifts into degenerate text the reward model mistakenly scores highly. There the fix
was a KL leash back to the reference model — a soft constraint on how far the
policy may move. That is a special case of the general idea. When the thing you
cannot capture in the reward is _how far the policy strays_, a KL bound suffices;
when it is _a physical limit_ (energy, collisions, torque), you need an explicit
constraint on a measured cost. Both limit how far the optimizer can exploit the
proxy. The rest of the lesson builds the machinery for the second, harder case.

A first instinct is to fold the constraint into the reward as a penalty: subtract
$\lambda\,c_t$ from the reward for each unit of cost $c_t$ and tune $\lambda$ by
hand. This works when you can find the right $\lambda$, but it hides the constraint
inside a scalar and gives you no guarantee about the resulting cost. Pick $\lambda$
too small and the policy pays the penalty and violates the limit anyway; too large
and it becomes so cautious it abandons the task. The constrained formulation makes
the limit a first-class object — a number you _specify_ rather than a weight you
tune by trial and error — and derives the right penalty automatically.

## The constrained MDP

The formalism is the **constrained Markov decision process (CMDP)**.[^altman]
It is an ordinary MDP with one addition: alongside the reward $r(s,a)$, each
transition emits one or more **cost** signals $c(s,a)$, and the policy must keep
the expected discounted cost under a threshold $d$ the designer sets.

Write the return as before and define an analogous **cost return**:

$$
J_r(\pi) \;=\; \mathbb{E}_\pi\!\Bigl[\textstyle\sum_{t\ge 0} \gamma^t\, r_t\Bigr],
\qquad
J_c(\pi) \;=\; \mathbb{E}_\pi\!\Bigl[\textstyle\sum_{t\ge 0} \gamma^t\, c_t\Bigr].
$$

The CMDP problem is then to maximize the reward return subject to the cost return
staying within budget:

$$
\max_{\pi}\; J_r(\pi)
\quad\text{subject to}\quad
J_c(\pi) \;\le\; d.
$$

With several constraints there is one $c_i$ and one threshold $d_i$ per
requirement. The cost is not the negative of reward; it is a _separate_
measurement of something the reward does not price — a count of collisions, joules
burned, or times a safety envelope was crossed — and $d$ says how much of it you
will tolerate on average.

> **Definition (Constrained MDP).** A tuple $(\mathcal{S}, \mathcal{A}, P, r, c,
> \gamma, d)$ extending an MDP with a cost function $c : \mathcal{S} \times
> \mathcal{A} \to \mathbb{R}$ and a threshold $d$. A policy is **feasible** if its
> expected discounted cost satisfies $J_c(\pi) \le d$; the objective is the
> highest-return policy among the feasible ones.

The constraint carves the space of all policies into a **feasible set** — those
that satisfy $J_c(\pi) \le d$ — and the goal is the best-return policy _inside_ it,
which is in general different from the unconstrained optimum sitting outside.

$$
% caption: The feasible set in policy space. The cost constraint $J_c(\pi) \le d$
% admits the shaded region; the unconstrained reward optimum lies outside it, so
% the constrained optimum is the highest-reward point on or inside the boundary.
% Reward increases toward the upper right.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ambient policy space
  \draw[black] (-0.2,-0.2) rectangle (8.6,5.0);
  \node[black!70, anchor=north west, font=\scriptsize] at (-0.15,4.95) {policy space};
  % reward contours (increasing toward upper right)
  \foreach \k in {1,2,3,4} {
    \draw[black] (8.6,{0.4*\k}) to[out=200,in=-70] ({0.9*\k},4.95);
  }
  \node[black, anchor=west, font=\scriptsize] at (6.0,3.9) {higher reward};
  % feasible set (shaded blob, lower-left region satisfying the cost budget)
  \fill[acc!10] (0.2,0.1) to[out=40,in=200] (3.2,1.3) to[out=20,in=250] (5.0,3.0)
    to[out=110,in=-30] (3.0,4.4) to[out=210,in=60] (0.6,2.2) to[out=240,in=120] (0.2,0.1);
  \draw[acc, thick] (0.2,0.1) to[out=40,in=200] (3.2,1.3) to[out=20,in=250] (5.0,3.0)
    to[out=110,in=-30] (3.0,4.4) to[out=210,in=60] (0.6,2.2) to[out=240,in=120] (0.2,0.1);
  \node[acc, font=\footnotesize] at (2.0,1.9) {feasible set};
  \node[acc, font=\scriptsize] at (2.4,3.1) {cost $\le$ d};
  % unconstrained optimum outside
  \fill[red] (7.6,4.4) circle (2.4pt);
  \node[red, anchor=east, font=\scriptsize] at (7.45,4.4) {unconstrained optimum};
  % constrained optimum on the boundary
  \fill[acc] (4.85,2.95) circle (2.4pt);
  \node[acc, anchor=west, font=\scriptsize] at (4.95,2.75) {constrained optimum};
\end{tikzpicture}
$$

Because the best feasible policy so often sits _on_ the boundary $J_c(\pi) = d$
(spend the whole budget, no more), the constraint is typically active, and the two
methods that follow are two ways of finding that boundary point.

## Lagrangian methods and RCPO

The textbook tool for a constrained optimization is the **Lagrangian**. Introduce
a multiplier $\lambda \ge 0$ on the constraint and fold it into a single objective,

$$
\mathcal{L}(\pi, \lambda)
\;=\;
J_r(\pi) \;-\; \lambda\,\bigl(J_c(\pi) - d\bigr).
$$

Read the multiplier as a _price_ on cost. For a fixed $\lambda$, maximizing
$\mathcal{L}$ over $\pi$ is an ordinary unconstrained RL problem with the shaped
reward $r_t - \lambda\, c_t$ — exactly the hand-penalty from earlier, except now
$\lambda$ is not something you guess. The theory of Lagrangian duality says the
constrained optimum is a **saddle point** of $\mathcal{L}$: a maximum over $\pi$
and a minimum over $\lambda$. We reach it by **primal-dual** optimization —
alternate improving the policy at the current price and adjusting the price toward
the constraint.

The multiplier update is a gradient step on $\mathcal{L}$ in $\lambda$, and
$\partial \mathcal{L} / \partial \lambda = -(J_c(\pi) - d)$, so it is a simple
feedback law:

$$
\lambda \;\gets\; \bigl[\, \lambda \;+\; \eta_\lambda\,\bigl(J_c(\pi) - d\bigr)\,\bigr]_+ ,
$$

with $[\cdot]_+$ clamping $\lambda$ to be non-negative. When the policy overspends
its cost budget ($J_c > d$), $\lambda$ rises, the penalty grows, and the
next policy update backs off the expensive behavior. When the policy is comfortably
under budget ($J_c < d$), $\lambda$ falls toward zero, the constraint relaxes, and
the policy is free to pursue reward. The price self-tunes until $J_c = d$.

$$
% caption: The primal-dual loop. The multiplier lambda prices the cost; the policy
% is improved against the shaped reward r minus lambda times c, then lambda is
% nudged up when cost exceeds the budget d and down when it falls below, driving
% the pair to the saddle point where cost equals d.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  b/.style={draw, minimum width=34mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[b, draw=acc, text=acc, thick] (pi) at (0,0) {primal step:\\improve policy on\\reward r - lambda c};
  \node[b] (lam) at (7.2,0) {dual step:\\lambda $\gets$ lambda + eta (cost - d)};
  \draw[->, acc, thick] (pi.north) .. controls (2.4,1.8) and (4.8,1.8) .. (lam.north)
    node[midway, above, black!70, font=\scriptsize] {measured cost};
  \draw[->, red, thick] (lam.south) .. controls (4.8,-1.8) and (2.4,-1.8) .. (pi.south)
    node[midway, below, black!70, font=\scriptsize] {updated price lambda};
\end{tikzpicture}
$$

**Reward Constrained Policy Optimization (RCPO)**[^rcpo] is the actor-critic
realization of this idea, with one refinement that makes it practical. Rather than
wait for a full episode to measure $J_c(\pi)$, RCPO treats $\lambda$ as a slowly
adapting parameter on a separate, slower timescale than the policy and value
updates. On the fast timescale the actor and critic optimize the penalized reward
$r_t - \lambda\, c_t$ with any standard policy-gradient method; on the slow
timescale $\lambda$ drifts by the feedback law above. The timescale separation is
what makes the saddle-point search converge: the policy sees a nearly-fixed price
long enough to respond to it, and the price sees a nearly-converged policy long
enough to read its true cost.

```algorithm
caption: $\textsc{RCPO}$ — reward-constrained actor-critic via primal-dual optimization
$\lambda \gets 0$ // initialize the cost price
initialize policy $\theta$ and value/cost critics
repeat
  collect trajectories under $\pi_\theta$, recording rewards $r_t$ and costs $c_t$
  $\tilde r_t \gets r_t - \lambda\, c_t$ // penalized reward on the fast timescale
  update critics and $\theta$ by a policy-gradient step on $\tilde r_t$
  $\hat J_c \gets$ estimate of $\mathbb{E}_\pi[\sum_t \gamma^t c_t]$ from the trajectories
  $\lambda \gets [\, \lambda + \eta_\lambda\,(\hat J_c - d)\,]_+$ // slow dual step
until policy and $\lambda$ both converge
return $\pi_\theta$
```

> **Definition (Lagrangian / primal-dual method).** A constrained-optimization
> scheme that adjoins the constraint to the objective through a multiplier
> $\lambda \ge 0$, forming $\mathcal{L}(\pi,\lambda) = J_r(\pi) - \lambda(J_c(\pi) -
> d)$, then finds its saddle point by alternating a **primal** step (improve the
> policy against the shaped reward) with a **dual** step (raise $\lambda$ when the
> constraint is violated, lower it when slack). RCPO is the actor-critic instance
> with $\lambda$ on a slower timescale.

The appeal is that RCPO reuses any policy-gradient method wholesale and adds one
scalar and one update line. The cost is that the guarantees are asymptotic: during
training, before $\lambda$ has caught up, the policy can and does violate the
constraint. For a simulator that is harmless. For a physical system where every
constraint violation is a real collision, it is not — which motivates methods that
bound the cost at _every_ update, not just in the limit.

#### Worked example: the multiplier finding the budget

Trace $\lambda$ over a few dual steps to see the feedback settle. Set the cost
budget $d = 25$ (say, an average of 25 collisions-worth of cost per episode is
tolerated) and the dual step $\eta_\lambda = 0.01$, starting from $\lambda = 0$.
With $\lambda = 0$ the policy ignores cost entirely, so it overspends badly —
measure $\hat J_c = 60$. The dual step raises

$$
\lambda \gets [\,0 + 0.01\,(60 - 25)\,]_+ = 0.35.
$$

Now cost is priced at $0.35$ per unit; the next policy backs off and measures
$\hat J_c = 32$, still over budget, so

$$
\lambda \gets [\,0.35 + 0.01\,(32 - 25)\,]_+ = 0.42.
$$

The policy tightens further to $\hat J_c = 22$, now _under_ budget, and the
multiplier falls:

$$
\lambda \gets [\,0.42 + 0.01\,(22 - 25)\,]_+ = 0.39.
$$

The overshoot-and-correct pattern — $0 \to 0.35 \to 0.42 \to 0.39$ — is the
feedback loop converging on the price at which $\hat J_c = d$. It
oscillates around the budget and, on the slow timescale, damps toward the
$\lambda^\ast$ where the constraint is exactly met. That equilibrium $\lambda^\ast$
is the _shadow price_ of the constraint: the marginal reward the policy gives up per
unit of cost budget, and had you set the hand-penalty of the opening section to
exactly $\lambda^\ast$ you would have recovered the same policy — the primal-dual
loop is what finds that number for you instead of making you guess it.

$$
% caption: The multiplier lambda converging to the budget. Starting from zero (cost
% ignored, badly over budget), each dual step nudges lambda up while measured cost
% exceeds d and down once it dips below, so lambda overshoots then damps toward the
% shadow price where measured cost equals the budget d. The cost trace (right axis,
% dashed) settles onto d from above.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (7.4,0) node[anchor=north, black] {dual steps};
  \draw[->, black] (0,0) -- (0,4.0) node[anchor=east, black] {lambda};
  % lambda trajectory: 0, 0.35, 0.42, 0.39, 0.40 (scaled *8 for height)
  \coordinate (l0) at (0.4,0);
  \coordinate (l1) at (1.8,2.8);
  \coordinate (l2) at (3.2,3.36);
  \coordinate (l3) at (4.6,3.12);
  \coordinate (l4) at (6.0,3.2);
  \draw[acc, very thick] (l0) -- (l1) -- (l2) -- (l3) -- (l4);
  \foreach \p in {(l0),(l1),(l2),(l3),(l4)} { \fill[acc] \p circle (2.2pt); }
  \node[acc, anchor=north, font=\scriptsize] at (0.4,-0.1) {0};
  \node[acc, anchor=south east, font=\scriptsize] at (1.8,2.8) {0.35};
  \node[acc, anchor=south, font=\scriptsize] at (3.2,3.4) {0.42};
  \node[acc, anchor=north, font=\scriptsize] at (4.6,3.0) {0.39};
  % shadow-price dashed target
  \draw[red, dashed] (0,3.2) -- (6.4,3.2);
  \node[red, anchor=south west, font=\scriptsize] at (6.15,3.28) {shadow price};
\end{tikzpicture}
$$

## Constrained policy optimization (CPO)

**Constrained Policy Optimization (CPO)**[^cpo] delivers a per-update guarantee by
grafting the constraint onto a trust-region method. Recall the trust-region idea
from [policy-gradient methods](/reinforcement-learning/approximation/policy-gradient-methods):
improve the policy within a small KL ball around the current one, small enough that
a local linear model of the return is trustworthy. TRPO solves, each step,

$$
\max_{\theta}\; \hat A_r(\theta)
\quad\text{subject to}\quad
\overline{\mathrm{KL}}(\theta, \theta_k) \le \delta,
$$

where $\hat A_r$ is the estimated reward advantage of the new policy over the old
and $\delta$ is the trust-region radius. CPO adds a second constraint inside the
same trust region: a linear model of the _cost_ return must stay under budget.
Using the surrogate $J_c(\theta_k) + \hat A_c(\theta)$ for the new policy's cost,
the CPO subproblem is

$$
\max_{\theta}\; \hat A_r(\theta)
\quad\text{s.t.}\quad
J_c(\theta_k) + \hat A_c(\theta) \le d,
\quad
\overline{\mathrm{KL}}(\theta, \theta_k) \le \delta.
$$

Because both the objective and the cost constraint are linearized and the KL
constraint is quadratic, the subproblem is a small convex program solvable in
closed form (up to a line search). Achiam et al. prove a **worst-case bound**: the
true cost return of the updated policy exceeds the surrogate by at most a term that
shrinks with $\delta$, so with a small enough trust region each CPO update keeps
the policy _approximately feasible_ throughout training, not merely at
convergence. That near-monotonic feasibility is the property RCPO lacks and the
reason CPO is the reference method when violations during learning are costly.

$$
% caption: One CPO update. Inside the KL trust region (circle) around the current
% policy, the cost constraint cuts off the half that would overspend the budget d;
% CPO takes the highest-reward point in the surviving feasible wedge, so the step
% improves reward while keeping the policy approximately within cost.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % feasible side of the trust region (lower-left of the cost line), clipped to the circle
  \begin{scope}
    \clip (0,0) circle (2.4);
    \fill[acc!10] (-2.8,-2.8) -- (2.8,-2.8) -- (2.8,-1.4) -- (-2.8,1.9) -- cycle;
  \end{scope}
  % trust region
  \draw[acc, thick] (0,0) circle (2.4);
  \node[acc, anchor=south, font=\scriptsize] at (0,2.5) {KL trust region};
  % feasible-set label inside shaded wedge
  \node[acc, font=\scriptsize] at (-1.3,-1.35) {feasible wedge};
  % current policy at center
  \fill[black] (0,0) circle (2.2pt);
  \node[black, anchor=north east, font=\scriptsize] at (-0.1,-0.1) {current policy};
  % reward-improving direction (arrow up-right, toward the infeasible corner)
  \draw[->, black, thick] (0,0) -- (1.6,1.6);
  \node[black, anchor=south, font=\scriptsize] at (1.35,1.7) {reward gradient};
  % cost constraint line cutting the region (feasible below-left)
  \draw[red, thick] (-2.35,1.7) -- (2.35,-1.1);
  \node[red, anchor=north west, font=\scriptsize] at (1.7,-0.7) {cost = d};
  % chosen CPO step: highest reward inside feasible region, on the cost boundary
  \coordinate (step) at (0.95,0.35);
  \fill[acc] (step) circle (2.4pt);
  \draw[->, acc, very thick] (0,0) -- (step);
  \node[acc, anchor=west, font=\scriptsize] at (1.05,0.4) {CPO step};
\end{tikzpicture}
$$

> **Definition (Constrained policy optimization).** A trust-region policy update
> that maximizes a linear model of the reward advantage subject to two constraints
> inside a KL ball: a linear model of the cost return must stay under budget $d$,
> and the KL step size must stay under $\delta$. The resulting update carries a
> worst-case bound on constraint violation, giving approximate feasibility at every
> step rather than only at convergence.

When a proposed update finds no feasible point at all — the current policy already
violates the budget and no step within the trust region fixes it — CPO falls back
to a pure **recovery** step that decreases cost as fast as the trust region allows,
ignoring reward until the policy is feasible again. That recovery mode is what lets
CPO start from an infeasible policy and steer it into the feasible set.

## Where this leaves us

The honest formulation of "behave well" is now precise. A **constrained MDP** keeps the
reward but adds one or more cost signals, each with a budget, and asks for the policy
that maximizes return _subject to_ keeping expected cost under budget. That single move
turns the vague demand for safety into an optimization problem with a feasible set.

Two policy-optimization methods solve it:

- **Lagrangian primal-dual (RCPO)** turns the constraint into a penalty with a learned
  multiplier $\lambda$: raise $\lambda$ when the cost budget is exceeded, lower it when
  there is slack, so the penalty self-tunes toward the boundary.
- **CPO** builds the cost budget directly into a trust-region update, guaranteeing (to
  first order) that each policy step stays feasible rather than only converging to
  feasibility in the limit.

Both control the _mean_ cost. But a policy safe on average can still be catastrophic in
the tail, and a policy that respects a constraint at convergence may violate it wildly
_while learning_. Optimizing the tail (risk-sensitive RL), staying safe during
exploration (shields, Lyapunov methods, safety layers), and the connection to alignment
continue in
[Safe RL: Risk, Safe Exploration, and Alignment](/reinforcement-learning/modern-deep-rl/safe-and-constrained-rl-part-2).

[^gaming]: **Krakovna, Uesato, Mikulik, et al. (2020)**, "Specification gaming: the flip side of AI ingenuity", DeepMind — catalogues dozens of documented cases, including the CoastRunners boat that laps a lagoon collecting power-ups instead of finishing the race. See also **Clark & Amodei (2016)**, "Faulty Reward Functions in the Wild", OpenAI, for the original boat-racing example.
[^goodhart]: A restatement of Goodhart's law for optimization: when a measure becomes a target it ceases to be a good measure. **Manheim & Garrabrant (2018)**, "Categorizing Variants of Goodhart's Law", arXiv:1803.04585, formalizes the failure modes; the proxy–objective gap is the mechanism behind specification gaming.
[^altman]: **Altman (1999)**, _Constrained Markov Decision Processes_, Chapman & Hall — the standard reference formalizing the CMDP: an MDP with additional cost functions and expected-cost constraints, and the linear-program and Lagrangian characterizations of its optimal policy.
[^rcpo]: **Tessler, Mankowitz, Mannor (2019)**, "Reward Constrained Policy Optimization", ICLR — the multi-timescale actor-critic that learns a penalty multiplier $\lambda$ on a slow timescale while optimizing the shaped reward $r - \lambda c$ on a fast one, converging to a constraint-satisfying policy.
[^cpo]: **Achiam, Held, Tamar, Abbeel (2017)**, "Constrained Policy Optimization", ICML — the first general-purpose policy-search method with near-monotonic constraint satisfaction, adding a linearized cost constraint to a TRPO trust-region step with a worst-case bound on violation.