Safe RL: Risk, Safe Exploration, and Alignment

This builds on Safe and Constrained RL: The CMDP and Policy Methods, which argued that reward maximization alone invites specification gaming, formalized the honest alternative as a constrained MDP (maximize return subject to a cost budget), and solved it two ways: Lagrangian primal-dual with a learned multiplier (RCPO), and constrained policy optimization with a trust-region cost bound (CPO).

Those methods bound the expected cost at convergence. Two gaps remain. A policy safe on average can still be ruinous in its worst episodes, and a policy that ends up safe can behave dangerously while it is still learning. This lesson closes both gaps and then connects safety to the alignment framing.

Risk-sensitive RL: optimizing the tail

Constraints bound the expected cost, but expectation is blind to variance. A policy that is safe on average can still visit a catastrophe on a small fraction of episodes, and for high-stakes deployments the tail is the whole point: a self-driving policy with a great mean and a rare fatal outcome is unacceptable, and no bound on $\mathbb{E}[\text{cost}]$ rules it out. Risk-sensitive RL optimizes a functional of the return distribution that weights bad outcomes more heavily than the mean does.¹

This connects directly to distributional RL, which already models the full distribution $Z^{\pi }(s,a)$ of returns rather than only its mean $Q^{\pi }(s,a)=\mathbb{E}[Z^{\pi }]$. Once you have the distribution, you can optimize any risk measure of it. The standard choice is Conditional Value at Risk. Let $Z$ be the return (a random variable) and fix a tail level $\alpha \in (0,1)$. The value-at-risk ${\mathrm{VaR}}_{\alpha }$ is the $\alpha$-quantile of $Z$ — the return you do at least as well as with probability $1-\alpha$ — and ${\mathrm{CVaR}}_{\alpha }$ is the mean of the returns below that quantile:

It is the average outcome over the worst $\alpha$ fraction of episodes. Maximizing ${\mathrm{CVaR}}_{\alpha }$ makes the policy improve its bad cases, not its average one. As $\alpha \to 1$ the whole distribution is in the tail and CVaR reduces to the ordinary expectation (risk-neutral); as $\alpha \to 0$ it concentrates on the single worst outcome (worst-case, robust). The knob $\alpha$ dials continuously between average-case and worst-case optimization.

CVaR has two properties that make it the default risk measure. It is a coherent risk measure (it respects diversification and monotonicity in a precise sense that variance does not), and it has a variational form,

that turns CVaR optimization into a tractable stochastic program — the auxiliary variable $\nu$ converges to ${\mathrm{VaR}}_{\alpha }$, and the objective is differentiable in the policy parameters, so it plugs into a policy-gradient method. A distributional critic that already estimates the quantiles of $Z^{\pi }$ supplies everything the CVaR objective needs; risk-sensitivity is then almost free on top of a distributional agent.

Worked example: CVaR on a discrete return distribution

Compute CVaR by hand to see how differently it ranks policies from the mean. Take two policies whose returns are ten equally-likely outcomes each:

• Policy A (steady): returns $\{6,7,7,8,8,8,9,9,10,8\}$, mean $=8.0$.
• Policy B (gamble): returns $\{-40,12,12,13,13,14,14,14,15,13\}$, mean $=8.0$.

Both have the same mean $8.0$, so a risk-neutral objective is indifferent. Now take $\alpha =0.1$, the worst $10\%$ — one outcome out of ten. For A the single worst outcome is $6$, so ${\mathrm{CVaR}}_{0.1}(A)=6$. For B it is $-40$, so ${\mathrm{CVaR}}_{0.1}(B)=-40$. The tail measure ranks A vastly above B — $6$ versus $-40$ — even though their means are identical, because B pays for its higher typical return with a rare catastrophe. Widen the tail to $\alpha =0.3$ (the worst three of ten): A averages $(6+7+7)/3\approx 6.67$; B averages $(-40+12+12)/3\approx -5.33$. As $\alpha \to 1$ both CVaRs climb back to the shared mean $8.0$ and the distinction vanishes. The knob $\alpha$ sets how much of the bad tail you insist on optimizing: a self-driving policy graded at $\alpha =0.01$ is forced to fix its one-in-a-hundred worst episode, which is the one that matters.

Safe exploration

Constraints and risk measures shape the policy the agent converges to. They say nothing about the actions it takes while learning, and that is where physical systems get damaged. A robot that must not exceed a joint torque cannot afford to discover the limit by exceeding it once. Safe exploration enforces constraints during learning, on every action, not merely on the final policy. Three mechanisms dominate.

Shielding

A shield is a reactive safety layer synthesized from a formal specification of what must never happen — a temporal-logic formula like never enter a collision state, never let the tank run dry.² At each step the shield inspects the action the agent proposes and, if that action could lead to a specification violation within its horizon, it overrides it with a safe alternative. The agent learns normally, receiving the state and reward as usual, but its actions are filtered through a correctness monitor that is provably safe by construction. The shield ignores reward entirely; it encodes only the safety automaton, and it guarantees the spec is never violated regardless of what the still-learning policy proposes. The limitation is that you must be able to write the specification and model the dynamics well enough to check it — feasible for discrete or well-understood systems, hard for high-dimensional continuous ones.

Lyapunov methods

A second approach borrows the Lyapunov function from control theory. A Lyapunov function $L(s)$ is a scalar energy that certifies stability: if you can guarantee $L$ never increases along the agent's trajectories, the agent is trapped inside a sublevel set $\{s:L(s)\le \ell \}$ — a certified safe region it can never leave. Lyapunov-based safe RL³ constructs such a function for the cost constraint and restricts the policy at each state to the actions that keep $L$ from growing,

which is a per-state constraint on the action set rather than a bound on the episodic cost. The guarantee is that the closed-loop system stays within the certified region for all time, giving the same during-learning safety a shield does but derived from the cost dynamics instead of a logical spec. The difficulty is constructing a valid Lyapunov function; much of the work is in learning or approximating one that is both safe and not so conservative that it blocks the task.

Safety layers

The most direct mechanism assumes a differentiable model of the cost and projects each proposed action onto the safe set. A safety layer⁴ sits between the policy and the environment: the policy proposes an action $a_{\text{raw}}$, and the layer solves a small optimization to find the closest action that satisfies the per-step cost constraint,

where $\hat{c}(s)+g(s{)}^{\top }a$ is a linear model of the next-step cost learned from data. With a single linear constraint this projection has a closed-form solution — the layer either passes $a_{\text{raw}}$ through unchanged (if it is already safe) or slides it along the constraint's gradient just far enough to satisfy the budget. Because the correction is analytic and differentiable, gradients still flow back to the policy, which learns to propose actions that need little correction. The safety layer is the continuous-control counterpart to the shield: where the shield vetoes discrete actions against a logical spec, the layer projects continuous actions against a learned cost model.

The three mechanisms trade off generality against assumptions. A shield needs a logical spec and a checkable model; Lyapunov methods need a certified energy function; a safety layer needs a differentiable cost model. Each provides what the constrained and risk-sensitive methods do not: safety during learning, enforced action by action.

Benchmarks and safe RLHF

The methods above are a decade of theory; two later developments ground them in practice and connect the thread back to the RLHF lesson.

Standardized safe-RL benchmarks. For years constrained RL lacked a common testbed, and every paper reported on its own tasks. Safety Gym fixed this: a suite of continuous-control navigation tasks where an agent must reach goals while a separate cost signal counts contacts with hazards, and every method is measured on the same reward-versus-cost frontier.⁵ Its empirical finding is worth stating plainly, because it tempers the theory: a well-tuned PPO-Lagrangian baseline — plain PPO with the primal-dual multiplier of this lesson bolted on — is hard to beat, and CPO's per-step guarantee often does not yield a better final frontier despite its stronger theory. The practical default in constrained deep RL is the Lagrangian method, not the trust-region one; CPO is preferred only when violations during training must be bounded, not merely at convergence.

Safe RLHF. The constrained formalism came back to language models directly. The RLHF lesson optimized a single reward that had to bundle helpfulness and harmlessness together, which pits the two against each other inside one scalar. Safe RLHF splits them: it trains a separate helpfulness reward model and a cost model for harm, then optimizes helpfulness subject to a harmlessness constraint using exactly the Lagrangian primal-dual machinery of this lesson — a learned multiplier trading helpfulness for safety on a slow timescale.⁶ The over-optimization the RLHF lesson controlled with a KL leash is, in this framing, one constraint among several, and the CMDP is the right container for all of them. The boat spinning in the lagoon and a language model that is helpful but unsafe are the same failure, and the same constrained-optimization discipline answers both.

Robustness and the alignment framing

Two threads sit just outside the CMDP formalism but belong to the same concern: making a policy behave when the world is not the training distribution.

The first is sim-to-real robustness. Policies trained in simulation exploit the simulator's quirks and fail on the real system whose friction, latency, and mass differ from the model — a distribution shift that shows up as a safety failure. Domain randomization⁷ addresses it directly: randomize the simulator's physics parameters (masses, friction, delays, sensor noise) across a wide range during training, so the policy must succeed on a whole family of dynamics rather than one. A policy robust across the randomized family tends to treat the real system as just another sample from it. This is a form of worst-case optimization — optimizing for the hardest dynamics you might face is a robust objective, related to the CVaR tail, and it is why domain randomization and risk-sensitive RL are two answers to one question: how to be safe under uncertainty you cannot fully model.

The second is the broader AI-safety / alignment framing. Every technique here is a response to the same fact: a scalar reward is an incomplete statement of intent, and a competent optimizer will exploit the incompleteness. Specification gaming, reward hacking, and over-optimization are one phenomenon seen at different scales — a boat spinning in a lagoon, a language model padding an answer, a cleaning robot hiding the mess — and the constrained, risk-sensitive, and shielded methods are the engineering responses. The RLHF lesson's KL leash and this lesson's cost budget are the same idea pointed at different proxies: bound how hard you push on a measure, because the measure stops being right before the optimizer stops pushing. As RL systems act in the world with real consequences, that discipline stops being an add-on and becomes the design problem.

The through-line: reward is not the whole story

The unconstrained objective assumed the reward said everything. The methods here each admit that it does not and add the missing structure back.

The lesson's one rule, stated plainly: real deployments must respect constraints the reward does not capture. Reward maximization is a means, not the goal, and a policy that maximizes reward while violating a limit the reward failed to mention has not succeeded — it has found the gap between what you wrote and what you meant. It shares a principle with offline RL, where the discipline was pessimism about values the data cannot verify; here it is pessimism about behaviors the reward cannot see. In both, the safe move is to distrust the parts of the objective you have no way to check.