---
title: "Average-Reward Control for Continuing Tasks"
module: Approximate Solution Methods
moduleNumber: 3
lessonNumber: 4
order: 304
summary: >
  With function approximation, discounting has no effect on a
  continuing task: averaged over the on-policy distribution, the discounted
  objective equals the average reward times a policy-independent constant, so
  $\gamma$ cannot change which policy is best. This lesson replaces discounting
  with the average-reward setting — the long-run reward rate $r(\pi)$, the
  differential return that measures each state's transient advantage over that
  rate, differential value functions and TD error, and differential
  semi-gradient Sarsa, the control method for continuing tasks that never
  invokes a discount factor.
topics: [Approximation]
sources:
  - book: Sutton & Barto
    ref: "Ch. 10 — On-policy Control with Approximation; §10.3 Average Reward: A New Problem Setting for Continuing Tasks"
  - book: Sutton & Barto
    ref: "§10.4 Deprecating the Discounted Setting"
---

This builds on
[on-policy control with approximation](/reinforcement-learning/approximation/on-policy-control),
which carried the semi-gradient update over to action values $\hat q(s,a,\mathbf{w})$
and solved Mountain Car — an episodic, discounted task. That lesson ended on a
warning: once the state space is too large to enumerate, discounting the future
starts to come apart. This lesson makes that precise and supplies the
replacement, the average-reward formulation of continuing control.

## Average reward: a new problem setting

So far everything has been _episodic_ and _discounted_. There is a third
classical setting for the goal of an MDP, standing alongside them: the
**average-reward** setting, which applies to _continuing_ problems — interaction
that goes on forever with no terminal state and no restarts. Unlike the
discounted case there is no discounting; the agent cares just as much about
delayed reward as immediate reward. This setting is central to the classical
theory of dynamic programming and less common in reinforcement learning, but with
function approximation it becomes necessary — for reasons the next section makes
precise.[^sb-avg]

The idea in one line: instead of the total reward collected from a state (which
needs discounting to stay finite), the objective is the rate of reward the
policy sustains forever, and states are measured by how much better or worse
than that rate they start out.

$$
% caption: The three problem settings for an MDP's objective. Episodic tasks sum
% undiscounted reward to a terminal state; discounted continuing tasks weight
% future reward by $\gamma^k$; average-reward continuing tasks maximize the
% long-run reward rate $r(\pi)$ with no discounting. Function approximation pushes
% continuing control from the middle column to the right.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  col/.style={draw, minimum width=28mm, minimum height=17mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[col] (ep) at (0,0) {episodic\\sum R to a\\terminal state\\(no discount)};
  \node[col] (disc) at (4.0,0) {discounted\\continuing\\sum gamma-to-the-k R\\(discount gamma $<$ 1)};
  \node[col, draw=acc, text=acc] (avg) at (8.4,0) {average reward\\maximize rate r(pi)\\(no discount)};
  \draw[->, acc, thick] (disc) -- (avg) node[midway, above, font=\scriptsize, text=black] {with approx.};
\end{tikzpicture}
$$

The quality of a policy $\pi$ is its long-run **average reward** per step, denoted
$r(\pi)$:

$$
r(\pi) \;\doteq\; \lim_{h \to \infty} \frac{1}{h} \sum_{t=1}^{h} \mathbb{E}\!\left[\, R_t \mid S_0, A_{0:t-1} \sim \pi \,\right]
\;=\; \lim_{t \to \infty} \mathbb{E}\!\left[\, R_t \mid S_0, A_{0:t-1} \sim \pi \,\right]
\;=\; \sum_s \mu_\pi(s) \sum_a \pi(a \mid s) \sum_{s', r} p(s', r \mid s, a)\, r,
$$

where the expectations are conditioned on the start state $S_0$ and on actions
following $\pi$ thereafter. Here $\mu_\pi$ is the **steady-state distribution**,
$\mu_\pi(s) \doteq \lim_{t \to \infty} \Pr\{S_t = s \mid A_{0:t-1} \sim \pi\}$,
assumed to exist for any $\pi$ and to be independent of the start state. That
independence is the assumption of **ergodicity**: where the MDP starts, and any
early decision, has only a temporary effect; in the long run the probability of
being in a state depends only on the policy and the transition dynamics.

> **Definition (Average reward $r(\pi)$).** The long-run rate of reward under
> $\pi$, $r(\pi) = \sum_s \mu_\pi(s) \sum_a \pi(a \mid s) \sum_{s',r} p(s', r \mid
> s, a)\, r$, the reward per step averaged over the steady-state distribution
> $\mu_\pi$. In the average-reward setting, a policy is optimal when it attains
> the maximal $r(\pi)$; policies are ordered simply by their reward per time step.

The steady-state distribution $\mu_\pi$ is the special distribution under which,
if you select actions according to $\pi$, you _stay_ in the same distribution:

$$
\sum_s \mu_\pi(s) \sum_a \pi(a \mid s)\, p(s' \mid s, a) \;=\; \mu_\pi(s').
$$

Selecting an action under $\pi$ from a state drawn from $\mu_\pi$ leaves the
next-state distribution equal to $\mu_\pi$ again — the distribution is stationary
under the policy, and that stationarity is what makes the average rate of reward
well defined.

$$
% caption: Ergodicity and the steady-state distribution. Starting from any of
% several initial states (left), the state distribution under a fixed policy $\pi$
% mixes over time and converges to the same steady-state $\mu_\pi$ (right),
% independent of where it started. This mixing is what makes the average reward
% $r(\pi)$ well defined.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bar/.style={draw, fill=acc!15, minimum width=3mm}]
  \definecolor{acc}{HTML}{2348F2}
  % left: several different starting distributions
  \node[anchor=south, font=\scriptsize] at (1.0,2.5) {start (any state)};
  \draw[black] (0.2,0) -- (2.0,0);
  \fill[acc] (0.5,0) rectangle (0.7,1.9);
  \fill[black] (1.5,0) rectangle (1.7,0.3);
  \fill[black] (1.0,0) rectangle (1.2,0.5);
  % arrow: mixing under pi
  \draw[->, black, thick] (2.3,1.0) -- (3.9,1.0)
    node[midway, above, font=\scriptsize] {follow pi};
  \node[anchor=north, font=\scriptsize, black] at (3.1,0.95) {mix over time};
  % right: steady-state, same regardless of start
  \node[anchor=south, font=\scriptsize] at (5.7,2.5) {steady-state mu(pi)};
  \draw[black] (4.3,0) -- (7.1,0);
  \foreach \x/\h in {4.6/0.6, 5.0/1.1, 5.4/1.5, 5.8/1.3, 6.2/0.9, 6.6/0.5} {
    \fill[acc!55] (\x,0) rectangle (\x+0.2,\h);
  }
\end{tikzpicture}
$$

### Why discounting fails with approximation

Why abandon the discounted setting we have used all along? Because with function
approximation it stops carrying information. Consider a continuing problem
with no beginning or end and no clearly identified states — states represented
only by feature vectors that may do little to distinguish one from another. In the
extreme, imagine _all_ the feature vectors are the same, so every state looks
identical to the approximator. Then the only thing the problem can be judged on is
the reward sequence, and performance must be assessed purely from that.

Now try to use discounting. At each step you could measure the discounted return;
some returns are big and some small, so you would average them over a long
interval — which is just the average-reward idea again. The decisive fact is what
that average equals. For any policy $\pi$, the average of the discounted returns is
_proportional_ to the average reward:

$$
\text{average of discounted returns under } \pi \;=\; \frac{r(\pi)}{1 - \gamma}.
$$

The discount rate $\gamma$ enters only through the constant factor
$1/(1 - \gamma)$, which is the _same_ for every policy. So it cancels out of any
comparison: the _ordering_ of all policies under the discounted objective is
identical to their ordering under the undiscounted average-reward objective. The
discount rate has no effect on which policy is best — it could be set to zero and
the ranking would be unchanged.

$$
% caption: The futility-of-discounting argument. Averaging the discounted return
% over the on-policy distribution yields $r(\pi)/(1-\gamma)$, so the factor
% $1/(1-\gamma)$ is a policy-independent constant: two policies keep the same
% relative order whatever $\gamma$ is, and $\gamma$ carries no information about
% which policy is better.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=34mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (obj) at (0,0) {discounted objective\\(over on-policy dist.)};
  \node[box, draw=acc, text=acc] (avg) at (5.4,0) {average reward r(pi)\\times 1/(1 - gamma)};
  \draw[->, acc, thick] (obj) -- (avg) node[midway, above, font=\scriptsize, text=black] {equals};
  % below: the constant factor is policy-independent
  \node[align=center, font=\scriptsize, anchor=north] at (2.7,-1.05)
    {1/(1 - gamma) is the same constant for every policy,\\so gamma cannot change which policy ranks higher};
  \draw[black, ->] (2.7,-0.62) -- (2.7,-0.95);
\end{tikzpicture}
$$

Two arguments make this equality solid. The first is a **symmetry** count. Every
time step is identical to every other, so with discounting each reward appears once
in each position across the family of returns: the reward at step $t$ shows up
undiscounted in the return that begins at $t-1$, discounted once in the return that
begins at $t-2$, discounted $k$ times in the return that begins $k$ steps earlier.
Its total weight across all returns is $1 + \gamma + \gamma^2 + \cdots = 1/(1-\gamma)$.
Since every reward carries that same weight, the average of the discounted returns is
$1/(1-\gamma)$ times the average reward, $r(\pi)/(1-\gamma)$.

The second is a **direct derivation** that pins it exactly: the same fact
follows from the Bellman equation in four lines. Define the objective as
the discounted value averaged over the on-policy distribution, $J(\pi) = \sum_s
\mu_\pi(s)\, v_\pi(s)$, with $v_\pi$ the ordinary discounted value function. Expand
$v_\pi$ by its Bellman equation and use the steady-state property of $\mu_\pi$:

$$
\begin{aligned}
J(\pi)
&= \sum_s \mu_\pi(s) \sum_a \pi(a\mid s) \sum_{s',r} p(s',r\mid s,a)\,[\,r + \gamma\, v_\pi(s')\,] \\
&= r(\pi) + \gamma \sum_{s'} v_\pi(s') \sum_s \mu_\pi(s) \sum_a \pi(a\mid s)\, p(s'\mid s,a) \\
&= r(\pi) + \gamma \sum_{s'} v_\pi(s')\, \mu_\pi(s') \\
&= r(\pi) + \gamma\, J(\pi).
\end{aligned}
$$

The first reward term collapsed to $r(\pi)$ because $\sum_s \mu_\pi(s) \sum_a \pi
\sum_{s',r} p\, r$ is precisely the definition of average reward; the inner double sum
over $s, a$ in the second term collapsed to $\mu_\pi(s')$ because $\mu_\pi$ is
stationary under $\pi$. Unrolling $J(\pi) = r(\pi) + \gamma J(\pi)$ gives $J(\pi) =
r(\pi) + \gamma r(\pi) + \gamma^2 r(\pi) + \cdots = r(\pi)/(1-\gamma)$. The discount
$\gamma$ appears only in that policy-independent constant, so it cannot change which
policy maximizes $J$.

The general version of this argument shows that optimizing discounted value over
the on-policy distribution is identical to optimizing _undiscounted_ average
reward — the value of $\gamma$ has no effect. So discounting has no role to play
in the definition of the control problem with function approximation. One can
still use $\gamma$ inside a _solution method_, turning it from a problem parameter
into an algorithm parameter, but there is a deeper loss behind this.

The root cause of the difficulty is the **policy improvement theorem**. In the
tabular case, improving the discounted value of one state is guaranteed to improve
the overall policy; with function approximation, that guarantee is gone. Changing
the weights to raise $\hat q$ at one state can lower it at others, because the
states share parameters — so a greedy step is no longer certain to help. Without
the improvement theorem the discounted control setting loses the property that
justified it, and the average-reward setting is the natural replacement.

## Differential returns and value functions

If there is no discounting, what plays the role of the return? In the
average-reward setting the return is defined in terms of _differences_ between
each reward and the average reward $r(\pi)$. This is the **differential return**:

$$
G_t \;\doteq\; \big(R_{t+1} - r(\pi)\big) + \big(R_{t+2} - r(\pi)\big) + \big(R_{t+3} - r(\pi)\big) + \cdots.
$$

Subtracting $r(\pi)$ from every reward is what makes this infinite sum converge:
a policy earning exactly its own average rate contributes zero on balance, and the
differential return measures only the _transient_ advantage or deficit of starting
in a particular state. The corresponding value functions are the **differential
value functions**, defined with the same expectations as before,
$v_\pi(s) \doteq \mathbb{E}_\pi[G_t \mid S_t = s]$ and
$q_\pi(s, a) \doteq \mathbb{E}_\pi[G_t \mid S_t = s, A_t = a]$, now with the
differential return inside.

> **Definition (Differential return).** The undiscounted sum of reward minus
> average reward, $G_t = \sum_{k=1}^{\infty} \big(R_{t+k} - r(\pi)\big)$. It
> converges because each term is centered at zero in the long run, and it measures
> the extra reward a state earns _relative to_ the policy's steady rate. The
> differential value functions are its state and state–action expectations.

The differential value functions satisfy Bellman equations of the familiar shape,
obtained from the discounted ones by two edits: drop every $\gamma$, and replace
each reward $r$ by $r - r(\pi)$. For state values,

$$
v_\pi(s) \;=\; \sum_a \pi(a \mid s) \sum_{r, s'} p(s', r \mid s, a) \big[\, r - r(\pi) + v_\pi(s') \,\big],
$$

and for action values,

$$
q_\pi(s, a) \;=\; \sum_{r, s'} p(s', r \mid s, a) \Big[\, r - r(\pi) + \sum_{a'} \pi(a' \mid s')\, q_\pi(s', a') \,\Big].
$$

The optimality equations replace $r(\pi)$ by its maximum over policies,
$\max_\pi r(\pi)$, and take the greedy max over next actions — the same edits
applied to $v_\ast$ and $q_\ast$. There is also a **differential TD error**, again the
discounted TD error with $\gamma$ removed and reward replaced by reward-minus-average.
For action values,

$$
\delta_t \;\doteq\; R_{t+1} - \bar R_{t+1} + \hat q(S_{t+1}, A_{t+1}, \mathbf{w}_t) - \hat q(S_t, A_t, \mathbf{w}_t),
$$

where $\bar R_{t+1}$ is the current estimate at time $t$ of the average reward
$r(\pi)$. (The state-value differential TD error is the same with $\hat v$ in
place of $\hat q$.) With these definitions, most of the algorithms and theoretical
results carry over to the average-reward setting unchanged.

$$
% caption: The two edits from the discounted to the differential Bellman/TD form.
% Every discount factor $\gamma$ is dropped, and every reward $r$ is replaced by
% $r - \bar R$, the reward minus the current estimate of the average reward. The
% differential TD error $\delta_t = R_{t+1} - \bar R_{t+1} + \hat q(S_{t+1},
% A_{t+1}) - \hat q(S_t, A_t)$ is the result.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=36mm, minimum height=12mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (disc) at (0,0) {discounted error\\R + gamma q(S',A') - q(S,A)};
  \node[box, draw=acc, text=acc] (diff) at (6.0,0) {di\/f\/ferential error\\R - Rbar + q(S',A') - q(S,A)};
  \draw[->, acc, thick] (disc) -- (diff);
  \node[align=center, font=\scriptsize, anchor=south] at (3.0,0.75)
    {drop gamma;\\reward $\to$ reward - Rbar};
\end{tikzpicture}
$$

## Differential semi-gradient Sarsa

Assembling the pieces gives the control algorithm for the continuing,
average-reward setting. The weight update is the average-reward version of
semi-gradient Sarsa: exactly equation for one-step Sarsa, but with the
differential TD error in place of the discounted one,

$$
\mathbf{w}_{t+1} \;\doteq\; \mathbf{w}_t + \alpha\, \delta_t\, \nabla \hat q(S_t, A_t, \mathbf{w}_t),
$$

with $\delta_t$ the differential action-value TD error above. Two things are
learned at once: the weights $\mathbf{w}$, as always, and the average-reward
estimate $\bar R$, which is nudged by the _same_ TD error. Because $\delta_t$
measures how much better the last transition was than expected, its running total
signals whether $\bar R$ is set too low or too high.

```algorithm
caption: $\textsc{Differential-Semi-Gradient-Sarsa}$ — estimate $\hat q \approx q_\ast$
input: a differentiable $\hat q : \mathcal{S} \times \mathcal{A} \times \mathbb{R}^d \to \mathbb{R}$
parameters: step sizes $\alpha, \beta > 0$
initialize $\mathbf{w} \in \mathbb{R}^d$ arbitrarily (e.g. $\mathbf{w} = \mathbf{0}$)
initialize average-reward estimate $\bar R \in \mathbb{R}$ arbitrarily (e.g. $\bar R = 0$)
initialize state $S$ and action $A$
loop // for each step
  take action $A$, observe $R, S'$
  choose $A'$ as a function of $\hat q(S', \cdot, \mathbf{w})$ (e.g. $\varepsilon$-greedy)
  $\delta \gets R - \bar R + \hat q(S', A', \mathbf{w}) - \hat q(S, A, \mathbf{w})$
  $\bar R \gets \bar R + \beta\, \delta$
  $\mathbf{w} \gets \mathbf{w} + \alpha\, \delta\, \nabla \hat q(S, A, \mathbf{w})$
  $S \gets S'$
  $A \gets A'$
```

There is no episode loop — the interaction is one unbroken stream of steps, which
is what "continuing" means. The average-reward estimate uses its own step size
$\beta$, typically smaller than $\alpha$, so $\bar R$ tracks the slow drift of the
policy's reward rate while $\mathbf{w}$ adapts faster to individual states.

### Worked example: the access-control queuing task

For example: ten servers face a single queue. Customers arrive with one of four
priorities, paying $1$, $2$, $4$, or $8$ if admitted to a free server and $0$ if
rejected. Each step the agent looks at the customer at the head of the queue and
either accepts (assigns a free server, if one exists) or rejects (drops the customer
for zero reward). Each busy server frees itself with probability $p = 0.06$ per step.
The queue never empties and priorities are uniform, so the state is just the pair
(number of free servers, priority of the head customer) — $11 \times 4 = 44$ states —
and there are two actions. The goal is to maximize long-run reward with no discounting,
which is what forces the average-reward setting.

Run differential semi-gradient Sarsa with $\alpha = 0.01$, $\beta = 0.01$,
$\varepsilon = 0.1$, all action values and $\bar R$ starting at zero. The policy it
finds is intuitive. When many servers are free, accept every customer —
there is no reason to hold capacity. As free servers grow scarce, start rejecting the
low-priority customers to hold capacity open for the high-priority ones still to come:
with one or two free servers the agent rejects priority-$1$ and often priority-$2$
customers while still admitting priority-$4$ and priority-$8$. The threshold is a
tradeoff — rejecting a priority-$1$ customer forfeits $1$ now but keeps a server open
for a possible priority-$8$ later, and that gamble only pays when free servers are few
enough that the wait for a replacement is short. As a byproduct $\bar R$ converges to
the policy's true average reward, the per-step rate this accept/reject rule sustains.

$$
% caption: The policy found by differential semi-gradient Sarsa on the access-control
% queuing task, as a grid over (free servers, head-customer priority). With many free
% servers every priority is accepted; as servers grow scarce the agent rejects the
% low-priority customers first to reserve capacity for high-priority ones. A = accept,
% R = reject.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum size=6.5mm, inner sep=0pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % column headers = free servers
  \foreach \c/\lab in {1/0, 2/2, 3/4, 4/6, 5/8, 6/10}
    \node[font=\scriptsize] at ({\c*0.75},3.1) {\lab};
  \node[anchor=south, font=\scriptsize] at (2.6,3.4) {free servers};
  % row headers = priority
  \foreach \r/\lab in {1/8, 2/4, 3/2, 4/1}
    \node[anchor=east, font=\scriptsize] at (0.35,{\r*0.7 - 0.35}) {\lab};
  \node[rotate=90, anchor=south, font=\scriptsize] at (-0.55,1.4) {priority};
  % grid of decisions: reject (R) when few servers + low priority
  \foreach \r/\p in {1/8, 2/4, 3/2, 4/1} {
    \foreach \c/\free in {1/0, 2/2, 3/4, 4/6, 5/8, 6/10} {
      \pgfmathsetmacro\thresh{\p < 2 ? 3 : (\p < 4 ? 1 : 0)}
      \ifdim \free pt < \thresh pt
        \node[cell, fill=red!12, text=red] at ({\c*0.75},{\r*0.7 - 0.35}) {R};
      \else
        \node[cell, fill=acc!10, text=acc] at ({\c*0.75},{\r*0.7 - 0.35}) {A};
      \fi
    }
  }
\end{tikzpicture}
$$

## The average-reward revival and continuous control

The differential machinery of this lesson has become more, not less, relevant as
reinforcement learning has scaled to deep networks.

**Average reward comes back for deep RL.** This chapter argues the average-reward
setting is the appropriate formulation for continuing problems under function
approximation, and recent deep-RL work has taken that seriously. Naik et al. (2019),
"Discounted reinforcement learning is not an optimization problem", _NeurIPS Optimization
Foundations of RL workshop_, sharpened the point that discounted control with function
approximation is not optimizing a well-defined objective, and average-reward deep
methods such as RVI-Q and differential Q-learning descendants (Wan, Naik & Sutton,
2021, "Learning and planning in average-reward Markov decision processes", _ICML_)
carry the differential-return machinery of this lesson into deep networks.[^avg-reward]
The $\bar R$ update — one scalar tracking the reward rate, nudged by the same TD error
that trains the weights — carries over essentially unchanged.

**Deterministic policy gradients for continuous control.** When actions are continuous
and the critic is differentiable, one can push the gradient of $\hat q$ through the
action into a deterministic policy: DDPG (Lillicrap et al., 2016, "Continuous control
with deep reinforcement learning", _ICLR_) and its successor TD3 (Fujimoto et al.,
2018, "Addressing function approximation error in actor-critic methods", _ICML_) do
exactly this, solving the Mountain-Car-like problem of continuous throttle that the
discrete $\{-1, 0, +1\}$ action set sidesteps.[^ddpg] They inherit the off-policy
instabilities the next lesson names, which is why they rely on target networks and
replay.

## What changed, and what comes next

The mechanical part of average-reward control is small — swap the discounted TD
error for the differential one, track a running estimate $\bar R$ of the reward
rate with its own step size, and drop the episode loop. What changed underneath is
the whole justification. Function approximation broke discounting: the discounted
objective ranks policies identically to the average-reward objective, so $\gamma$
carries no information, and the policy improvement theorem that justified greedy
control is lost. The average-reward setting replaces it — the differential
return, the differential value functions, and differential semi-gradient Sarsa give
a control method for continuing tasks that never invokes a discount factor at all.

One assumption remains: all of this was _on-policy_. The data came
from the very policy being evaluated. The
[next lesson](/reinforcement-learning/approximation/off-policy-and-the-deadly-triad)
lets the behavior and target policies diverge, and finds that off-policy learning,
bootstrapping, and function approximation together form a _deadly triad_ that can
make the weights diverge — the sharpest instability in all of approximate
reinforcement learning.

[^sb-avg]: **Sutton & Barto**, _Reinforcement Learning: An Introduction_ (2nd ed.), §10.3 — Average Reward: A New Problem Setting for Continuing Tasks: the average reward $r(\pi)$ (10.6)–(10.7), steady-state distribution (10.8), the differential return (10.9), the differential Bellman equations, the differential TD errors (10.10)–(10.11), and differential semi-gradient Sarsa (10.12). §10.4 — Deprecating the Discounted Setting: the futility-of-discounting result that the average of the discounted returns is $r(\pi)/(1-\gamma)$, the symmetry argument and the boxed direct derivation $J(\pi) = r(\pi) + \gamma J(\pi)$, and the loss of the policy improvement theorem under function approximation. Example 10.2 — the access-control queuing task (10 servers, four priorities, $p = 0.06$, parameters $\alpha = \beta = 0.01$, $\varepsilon = 0.1$).
[^avg-reward]: **Naik, A. et al.** (2019), "Discounted reinforcement learning is not an optimization problem", NeurIPS Optimization Foundations of RL workshop — the argument that discounted control with function approximation lacks a well-defined objective. **Wan, Y., Naik, A. & Sutton, R. S.** (2021), "Learning and planning in average-reward Markov decision processes", _ICML_ — differential Q-learning and average-reward planning.
[^ddpg]: **Lillicrap, T. et al.** (2016), "Continuous control with deep reinforcement learning", _ICLR_ — DDPG, deterministic policy gradients through a differentiable critic. **Fujimoto, S. et al.** (2018), "Addressing function approximation error in actor-critic methods", _ICML_ — TD3, correcting overestimation bias in continuous-control critics.