---
title: Activation Functions
module: Neural Networks
moduleNumber: 2
lessonNumber: 2
order: 202
summary: >
  The activation is the only nonlinear part of a layer, and the reason
  depth adds expressive power. We catalog the standard hidden units (sigmoid, tanh,
  ReLU and its descendants, plus GELU, softplus, swish and maxout), derive each
  unit's derivative in full, make the vanishing-gradient problem
  quantitative with the chain-rule product, work numeric examples, and explain
  why the saturating units gave way to ReLU and why ReLU's own dead-unit failure
  gave way to Leaky/PReLU/ELU/GELU.
topics: [Neural Networks]
sources:
  - book: Goodfellow
    ref: "§6.3 — Hidden Units"
  - book: Goodfellow
    ref: "§6.2.2 — Output Units; §8.2.5 Saturation"
  - book: Chollet
    ref: "Ch. 4 — Getting Started with Neural Networks (activations)"
  - book: Stevens
    ref: "Ch. 6 — Using a Neural Network to Fit the Data (nn.Tanh, nn.ReLU)"
---

A layer is an affine map followed by a pointwise nonlinearity,
$h = g(Wx + b)$. The affine part carries all the parameters; the
[nonlinearity $g$ supplies the entire nonlinear effect](/deep-learning/neural-networks/the-multilayer-perceptron):
strip it out and a deep stack collapses to a single linear map. The choice of $g$
therefore matters: it sets how gradients flow back through depth, whether units
saturate, and how fast the network trains. This lesson catalogs the standard
choices and their derivatives.

> **Definition (Activation function).** A function $g : \mathbb{R} \to \mathbb{R}$
> applied elementwise to the pre-activation $z = Wx + b$ to produce a layer's
> output $h = g(z)$. It must be nonlinear (else depth collapses) and almost
> everywhere differentiable (so [backpropagation](/deep-learning/neural-networks/backpropagation)
> can pass a gradient through it).

To make the requirement precise, suppose two
layers with $g$ the identity: $h_2 = W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x +
(W_2 b_1 + b_2)$. The composite is one affine map with weight $W_2 W_1$ and bias
$W_2 b_1 + b_2$, and no amount of stacking escapes the affine class. A single
nonlinear $g$ between the two multiplications is what lets a network represent a
function that no single affine map can. Everything below is about which $g$ to
pick and what each choice costs in the backward pass.

## The catalog

Every hidden unit in common use is one scalar function and its derivative. The
derivative column matters most: during the backward pass the gradient at a
unit is multiplied by $g'(z)$, so the _size_ of $g'$ controls how much signal
passes through. The following table is the reference the rest of the lesson expands.

| name | $g(z)$ | $g'(z)$ | range | saturates? | notes |
| --- | --- | --- | --- | --- | --- |
| sigmoid | $\dfrac{1}{1+e^{-z}}$ | $\sigma(1-\sigma)$ | $(0,1)$ | both tails | $g'\le\tfrac14$; not zero-centered |
| tanh | $\dfrac{e^{z}-e^{-z}}{e^{z}+e^{-z}}$ | $1-\tanh^2 z$ | $(-1,1)$ | both tails | zero-centered; $g'\le 1$ |
| ReLU | $\max(0,z)$ | $\mathbf{1}[z>0]$ | $[0,\infty)$ | left only | cheap; can die |
| Leaky ReLU | $\max(\alpha z, z)$ | $\mathbf{1}[z>0]+\alpha\,\mathbf{1}[z\le 0]$ | $(-\infty,\infty)$ | no | $\alpha\!\approx\!0.01$ keeps a trickle |
| PReLU | $\max(\alpha z, z)$ | as Leaky, $\alpha$ learned | $(-\infty,\infty)$ | no | $\alpha$ a trained parameter |
| ELU | $z$ if $z>0$ else $\alpha(e^{z}-1)$ | $1$ if $z>0$ else $\alpha e^{z}$ | $(-\alpha,\infty)$ | left soft | smooth; pushes mean toward $0$ |
| GELU | $z\,\Phi(z)$ | $\Phi(z)+z\,\phi(z)$ | $\approx[-0.17,\infty)$ | left soft | $\Phi$ standard-normal CDF |
| Softplus | $\log(1+e^{z})$ | $\sigma(z)$ | $(0,\infty)$ | no | smooth ReLU; $g'=\sigma$ |
| Swish / SiLU | $z\,\sigma(z)$ | $\sigma(z)+z\,\sigma(z)(1-\sigma(z))$ | $\approx[-0.28,\infty)$ | left soft | self-gated; non-monotone |
| Maxout | $\max_j (w_j^\top x + b_j)$ | $w_{j^\star}$ of the winning piece | $\mathbb{R}$ | no | learns its own shape; $k\times$ params |

Here $\sigma$ is the logistic sigmoid, $\Phi$ and $\phi$ are the standard-normal
CDF and PDF, and $\alpha$ is a small positive constant (or a learned parameter for
PReLU). Two families fall out of the table: the **saturating** units (sigmoid,
tanh) whose derivative vanishes at both extremes, and the **non-saturating**
rectified units (ReLU onward) whose derivative stays bounded away from zero on at
least the positive half-line.[^gf-hidden]

## Every derivative, derived

Backpropagation never needs the activation itself in the backward pass, only its
derivative. Each derivative below takes one or two lines to derive.

### Sigmoid

Write $\sigma(z) = (1+e^{-z})^{-1}$ and differentiate by the chain rule:

$$
\sigma'(z) = -\,(1+e^{-z})^{-2}\cdot(-e^{-z})
= \frac{e^{-z}}{(1+e^{-z})^2}
= \frac{1}{1+e^{-z}}\cdot\frac{e^{-z}}{1+e^{-z}}.
$$

The first factor is $\sigma(z)$; the second is $1-\sigma(z)$, since
$1-\sigma = 1 - \frac{1}{1+e^{-z}} = \frac{e^{-z}}{1+e^{-z}}$. Hence

$$
\;\sigma'(z) = \sigma(z)\,\parens{1-\sigma(z)}\;
$$

Two consequences read straight off this form. The derivative is expressed in the
_output_ $\sigma$, so a forward pass that cached $h=\sigma(z)$ needs no re-exp in
the backward pass, only $h(1-h)$. And because $\sigma \in (0,1)$, the product
$\sigma(1-\sigma)$ is a downward parabola in $\sigma$ capped at its vertex, which
is the source of the $\tfrac14$ ceiling derived below.

### Tanh

The tanh derivative follows the same pattern. With $\tanh z = \frac{e^z-e^{-z}}{e^z+e^{-z}}$,
the quotient rule gives numerator $(e^z+e^{-z})^2 - (e^z-e^{-z})^2$ over
$(e^z+e^{-z})^2$, and expanding the two squares,

$$
(e^z+e^{-z})^2 - (e^z-e^{-z})^2
= \parens{e^{2z}+2+e^{-2z}} - \parens{e^{2z}-2+e^{-2z}} = 4,
$$

so

$$
\tanh'(z) = \frac{(e^z+e^{-z})^2-(e^z-e^{-z})^2}{(e^z+e^{-z})^2}
= 1 - \frac{(e^z-e^{-z})^2}{(e^z+e^{-z})^2}
= \,1-\tanh^2 z\,.
$$

Since $\tanh = 2\sigma(2z)-1$, tanh is a rescaled, recentered sigmoid; the
zero-centering lifts its peak derivative to $1$ (at $z=0$) versus the sigmoid's
$\tfrac14$, and makes tanh the better of the two saturating choices.

### ReLU

ReLU is differentiable everywhere except the kink at $z=0$, and its derivative is
just the indicator of the active region:

$$
\frac{d}{dz}\max(0,z) =
\begin{cases} 1 & z > 0 \\ 0 & z < 0 \end{cases}
= \mathbf{1}[z>0],
$$

with the value at $z=0$ taken by convention to be $0$ (frameworks pick $0$ or $1$;
the measure-zero choice is irrelevant in practice). This step-function derivative,
a hard gate of $1$ or $0$, is the source of both ReLU's strength and its dead-unit
pathology.

### Leaky ReLU and PReLU

Leaky ReLU replaces the flat left branch with a shallow line of slope $\alpha$,

$$
g(z) = \max(\alpha z, z) =
\begin{cases} z & z \ge 0 \\ \alpha z & z < 0, \end{cases}
\qquad
g'(z) =
\begin{cases} 1 & z > 0 \\ \alpha & z < 0. \end{cases}
$$

The derivative is now $\alpha > 0$ on the entire negative half-line rather than
$0$, so a unit sitting in the negative region still receives an $\alpha$-scaled
gradient. PReLU is identical except $\alpha$ is a learned parameter (one per
channel), so the negative slope adapts during training rather than being fixed at
$0.01$.

### ELU

ELU keeps the identity for $z>0$ but replaces the kink with a smooth exponential
that saturates gently toward $-\alpha$ on the left:

$$
g(z) =
\begin{cases} z & z > 0 \\ \alpha\,(e^{z}-1) & z \le 0, \end{cases}
\qquad
g'(z) =
\begin{cases} 1 & z > 0 \\ \alpha\,e^{z} & z \le 0. \end{cases}
$$

The left derivative $\alpha e^{z}$ is strictly positive and continuous, and at
$z=0$ both branches agree ($g'=1$ from the right, $\alpha e^{0}=\alpha$ from the
left; with the common choice $\alpha=1$ the derivative is continuous too). Because
the negative branch bottoms out at $-\alpha$ instead of $0$, ELU can output
negative values, which pulls each layer's mean activation toward zero and keeps
the next layer's pre-activations nearer the high-gradient region.

### Softplus

Softplus is a smooth ReLU; its derivative is the sigmoid, which is why it appears
whenever a strictly positive, differentiable output is wanted:

$$
g(z) = \log(1+e^{z}),
\qquad
g'(z) = \frac{e^{z}}{1+e^{z}} = \frac{1}{1+e^{-z}} = \sigma(z).
$$

As $z \to +\infty$, $\log(1+e^z) \to z$ (it hugs the ReLU line); as $z \to
-\infty$ it decays to $0$ smoothly. Its derivative never reaches $0$ or $1$
exactly, so it has no dead region but also no perfectly flat gradient of $1$.

### GELU and swish/SiLU

GELU gates the input by the probability that a standard normal falls below it,
$g(z) = z\,\Phi(z)$ with $\Phi$ the standard-normal CDF and $\phi$ its density.
The product rule gives

$$
g'(z) = \Phi(z) + z\,\phi(z),
\qquad \phi(z) = \tfrac{1}{\sqrt{2\pi}}e^{-z^2/2}.
$$

Swish (a.k.a. SiLU) is the same idea with a logistic gate, $g(z) = z\,\sigma(z)$,
so

$$
g'(z) = \sigma(z) + z\,\sigma(z)\parens{1-\sigma(z)} = \sigma(z)\parens{1 + z\,(1-\sigma(z))}.
$$

Both are smooth, both are the identity in the large-$z$ limit, and both dip
slightly _below_ zero for moderately negative $z$ before returning to $0$ — a
non-monotone shape that the empirical record favors in deep Transformers. Their
minimum sits near $z\approx -1.28$ for swish (value $\approx -0.28$) and near
$z\approx -0.75$ for GELU (value $\approx -0.17$).

### Maxout

Maxout is the outlier: it is not a fixed scalar shape but a piecewise-linear unit
that takes the max over $k$ affine pieces,

$$
g(x) = \max_{j=1,\dots,k}\parens{w_j^\top x + b_j},
\qquad
\nabla_x g = w_{j^\star},\quad j^\star = \arg\max_j\parens{w_j^\top x + b_j}.
$$

The gradient flows entirely through the winning piece, so maxout has no flat
region at all as long as one piece is increasing; with $k=2$ and $w_2=0$ it
recovers ReLU exactly. The cost is $k\times$ the parameters, which is why it is
rarely the default.

$$
% caption: Three activations on shared axes. Sigmoid and tanh flatten in both tails;
% ReLU is the identity for $z>0$ and exactly $0$ for $z<0$.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (-3.6,0) -- (3.8,0) node[right, font=\footnotesize] {input};
  \draw[->, thick] (0,-1.9) -- (0,2.3) node[above, font=\footnotesize] {output};
  \draw[black, dashed] (-3.4,1.5) -- (3.4,1.5);
  \node[black, font=\footnotesize, anchor=west] at (3.45,1.5) {1};
  \draw[black, dashed] (-3.4,-1.5) -- (3.4,-1.5);
  \node[black, font=\footnotesize, anchor=west] at (3.45,-1.5) {-1};
  % sigmoid, scaled so 1 -> 1.5
  \draw[acc, very thick] plot[domain=-3.3:3.3, samples=80] (\x, {1.5/(1+exp(-2*\x))});
  % tanh, scaled so 1 -> 1.5
  \draw[green, very thick] plot[domain=-3.3:3.3, samples=80] (\x, {1.5*tanh(\x)});
  % ReLU (clip at axis range), scaled so input 1 -> 1.5
  \draw[red, very thick] (-3.3,0) -- (0,0) -- (1.27,1.9);
  \node[acc, font=\footnotesize, anchor=east] at (-1.3,1.0) {sigmoid};
  \node[green, font=\footnotesize, anchor=west] at (1.85,1.2) {tanh};
  \node[red, font=\footnotesize, anchor=south east] at (1.27,1.95) {ReLU};
\end{tikzpicture}
$$

The rectified family shares one shape below, all identity on the right and
differing only in how they treat the negative side: hard zero (ReLU), a shallow
line (Leaky/PReLU), or a smooth curve that dips and returns (ELU, GELU, swish).

$$
% caption: The rectified family on shared axes. All are the identity for $z>0$.
% ReLU is flat on the left; Leaky keeps a shallow line; ELU and swish curve
% smoothly, and swish dips below $0$ before returning.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{orange}{HTML}{D97A17}
  % axes
  \draw[->, thick] (-2.8,0) -- (3.7,0) node[right, font=\footnotesize] {input};
  \draw[->, thick] (0,-1.15) -- (0,2.4) node[above, font=\footnotesize] {output};
  % ReLU (blue): flat then identity
  \draw[acc, very thick] (-2.6,0) -- (0,0) -- (2.1,2.1);
  % Leaky (red): shallow slope on the left
  \draw[red, very thick] (-2.6,-0.52) -- (0,0);
  % ELU (green): smooth saturating left branch toward -1
  \draw[green, very thick] plot[domain=-2.6:0, samples=60] (\x, {exp(\x)-1});
  % swish (orange): dips below 0 then returns; z*sigmoid(z)
  \draw[orange, very thick] plot[domain=-2.6:0, samples=60] (\x, {\x/(1+exp(-\x))});
  \node[acc, font=\footnotesize, anchor=south east] at (2.05,2.0) {ReLU};
  \node[red, font=\footnotesize, anchor=south east] at (-2.6,-0.5) {Leaky};
  \node[green, font=\footnotesize, anchor=north west] at (-2.55,-0.9) {ELU};
  \node[orange, font=\footnotesize, anchor=north] at (-0.55,-0.8) {swish};
\end{tikzpicture}
$$

## The vanishing-gradient problem, made quantitative

The backward pass through an $L$-layer network multiplies one Jacobian factor per
layer. Along a single path of units the gradient that reaches layer $k$ carries a
product of activation derivatives,

$$
\frac{\partial \mathcal{L}}{\partial z^{(k)}}
\;=\;
\parens{\textstyle\prod_{\ell=k+1}^{L} g'\!\parens{z^{(\ell)}}\, w^{(\ell)}}\,
\frac{\partial \mathcal{L}}{\partial h^{(L)}},
$$

so the magnitude of the gradient at depth $k$ scales like the **product** of the
per-layer derivatives $g'(z^{(\ell)})$ (times the weights). If each factor is
below $1$, the product shrinks geometrically with depth, and the deep layers stop
receiving signal.[^gf-vanish]

> **Definition (Saturating unit).** A unit whose derivative $g'(z) \to 0$ as
> $\abs{z} \to \infty$. Sigmoid and tanh saturate in both tails; a unit is
> **non-saturating** if $g'$ stays bounded away from $0$ on an unbounded region
> (ReLU: $g'=1$ for all $z>0$).

The sigmoid is the worst case: its derivative is small even at its
maximum. Maximizing $\sigma'(z) = \sigma(1-\sigma)$ over $\sigma \in (0,1)$: set
$p = \sigma$ and differentiate $p(1-p) = p - p^2$, giving $1 - 2p = 0$, so the
peak is at $p = \tfrac12$, i.e. $z = 0$:

$$
\max_z \sigma'(z) = \tfrac12\cdot\tfrac12 = \tfrac14 = 0.25 .
$$

So a sigmoid contributes **at most** $0.25$ per layer, and far less in the tails.
Work the tail out numerically: at $z=4$, $\sigma(4) = \frac{1}{1+e^{-4}} \approx
0.982$, and $\sigma'(4) = 0.982\,(1-0.982) \approx 0.0177$. The tanh is better (it
peaks at $\tanh'(0)=1$) but still vanishes in the tails, with
$\tanh'(2) = 1-\tanh^2 2 = 1 - 0.964^2 \approx 0.071$.

| depth $L$ | $0.25^{L}$ (sigmoid peak) | $1.0^{L}$ (tanh peak) | $1^{L}$ (ReLU active) |
| --- | --- | --- | --- |
| $1$ | $0.25$ | $1$ | $1$ |
| $5$ | $9.8\times 10^{-4}$ | $1$ | $1$ |
| $10$ | $9.5\times 10^{-7}$ | $1$ | $1$ |
| $20$ | $9.1\times 10^{-13}$ | $1$ | $1$ |

Ten sigmoid layers, each at their _best_ point, already
attenuate the gradient by a factor of $10^{6}$, and the realistic factor is worse
because most units sit off their peak. This is why deep sigmoid/tanh stacks were
nearly untrainable before ReLU, and why initialization that keeps pre-activations
near $z=0$ (where $g'$ is largest) matters so much; see
[initialization](/deep-learning/optimization/initialization).[^gf-saturate]

$$
% caption: Derivatives on shared axes: the sigmoid derivative peaks at $0.25$,
% tanh at $1$, and the ReLU derivative is a unit step.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (-3.3,0) -- (3.5,0) node[right, font=\footnotesize] {input};
  \draw[->, thick] (0,-0.2) -- (0,2.2) node[above, font=\footnotesize] {\texttt{derivative}};
  \draw[black, dashed] (-3.3,2.0) -- (3.3,2.0) node[black, right, font=\footnotesize] {1};
  \draw[black, dashed] (-3.3,0.5) -- (3.3,0.5) node[black, right, font=\footnotesize] {0.25};
  % sigmoid' = s(1-s), peak 0.25 -> scaled to 0.5
  \draw[acc, very thick] plot[domain=-3.2:3.2, samples=90]
    (\x, {2*(1/(1+exp(-2*\x)))*(1-1/(1+exp(-2*\x)))});
  % tanh' = 1 - tanh^2, peak 1 -> scaled to 2
  \draw[green, very thick] plot[domain=-3.2:3.2, samples=90]
    (\x, {2*(1-tanh(\x)*tanh(\x))});
  % ReLU' step: 0 then 1 -> scaled to 2
  \draw[red, very thick] (-3.2,0) -- (0,0);
  \draw[red, very thick] (0,2.0) -- (3.2,2.0);
  \draw[red, very thick, dotted] (0,0) -- (0,2.0);
  \node[acc, font=\footnotesize, anchor=south] at (1.7,0.55) {sigmoid'};
  \node[green, font=\footnotesize, anchor=west] at (0.7,1.55) {tanh'};
  \node[red, font=\footnotesize, anchor=south] at (2.2,2.0) {ReLU'};
\end{tikzpicture}
$$

The product-of-derivatives view makes the depth penalty geometric, not additive.
The next figure shows the same gradient magnitude collapsing layer by layer as it
propagates backward through a saturating stack.

$$
% caption: Gradient magnitude shrinking backward through a saturating stack: each
% layer multiplies by $g'\le 0.25$, so early layers receive almost nothing.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % baseline
  \draw[->, thick] (-0.4,0) -- (10.4,0) node[right, font=\footnotesize] {\texttt{layer (output at right)}};
  \draw[->, thick] (0,-0.1) -- (0,3.4) node[above, font=\footnotesize] {\texttt{gradient size}};
  % bars: heights ~ 0.25^k from the output side, drawn left=early to right=output
  % use explicit heights so the decay reads cleanly
  \draw[red, very thick, fill=red!15]  (0.4,0)  rectangle (1.2,0.10);
  \draw[red, very thick, fill=red!15]  (1.8,0)  rectangle (2.6,0.16);
  \draw[acc, very thick, fill=acc!15]  (3.2,0)  rectangle (4.0,0.30);
  \draw[acc, very thick, fill=acc!15]  (4.6,0)  rectangle (5.4,0.60);
  \draw[acc, very thick, fill=acc!15]  (6.0,0)  rectangle (6.8,1.30);
  \draw[acc, very thick, fill=acc!15]  (7.4,0)  rectangle (8.2,3.00);
  \node[font=\scriptsize, anchor=north] at (0.8,-0.12)  {early};
  \node[font=\scriptsize, anchor=north] at (7.8,-0.12)  {output};
  \node[red, font=\footnotesize, anchor=south west] at (2.0,0.55)
    {\texttt{early layers starve}};
  \draw[->, black, thick] (7.2,2.7) .. controls (4.4,3.0) and (2.4,2.2) .. (1.1,1.4);
  \node[black, font=\scriptsize, anchor=south] at (4.9,3.0)
    {\texttt{each step scales by the derivative}};
\end{tikzpicture}
$$

## ReLU and the dead-unit problem

ReLU sidesteps vanishing gradients on its active side outright: for $z>0$ the
derivative is exactly $1$, so the product across active layers is $1^{L}=1$, with no
attenuation at all. That single property, more than any other, made very deep
networks trainable.

> **Theorem (ReLU does not saturate on the right).** For $z>0$, $\,\frac{d}{dz}\max(0,z)=1$
> for all $z$, so a path of active ReLUs multiplies the backward gradient by
> $\prod_\ell 1 = 1$, independent of depth.

> **Proof.** $\max(0,z)=z$ on $z>0$, whose derivative is the constant $1$; the
> product of $L$ ones is $1$ regardless of $L$. $\qed$

The flat left half causes a different failure. A unit whose pre-activation is negative for
_every_ training example outputs $0$ and has derivative $0$, so it receives no
gradient and never updates: it is **dead**.

> **Definition (Dead ReLU).** A ReLU unit for which $z = w^\top x + b < 0$ on the
> entire data distribution. Then $g(z)=0$ and $g'(z)=0$ for all inputs, so
> $\nabla_w \mathcal{L}=0$: the unit is frozen and contributes nothing. A large
> negative bias, or a too-large gradient step that drives the weights negative,
> can kill a unit permanently.

The mechanism is worth tracing through the chain rule. The gradient on a weight
$w_i$ of a ReLU unit is

$$
\frac{\partial \mathcal{L}}{\partial w_i}
= \frac{\partial \mathcal{L}}{\partial h}\cdot g'(z)\cdot x_i
= \frac{\partial \mathcal{L}}{\partial h}\cdot \mathbf{1}[z>0]\cdot x_i .
$$

Once $z<0$ on every example, the factor $\mathbf{1}[z>0]$ is $0$ for every one, so
$\partial\mathcal{L}/\partial w_i = 0$ regardless of the upstream gradient or the
inputs. Nothing in the update can move $w$ or $b$, so the unit cannot recover:
the failure is irreversible.

$$
% caption: A dead ReLU: its pre-activation stays in the flat region $z<0$ where
% the derivative is $0$, so no gradient reaches its weights and it never recovers.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \definecolor{green}{HTML}{1F9D4D}
  % axes
  \draw[->, thick] (-3.7,0) -- (3.8,0) node[right, font=\footnotesize] {input};
  \draw[->, thick] (0,-1.1) -- (0,2.6) node[above, font=\footnotesize] {output};
  % shaded dead region (left of 0)
  \fill[red!14] (-3.5,-1.0) rectangle (0,2.45);
  % ReLU curve (drawn over the shading)
  \draw[acc, very thick] (-3.5,0) -- (0,0) -- (2.1,2.1);
  \node[red, font=\footnotesize, anchor=south] at (-1.75,1.85) {\texttt{flat: output 0}};
  \node[red, font=\footnotesize, anchor=south] at (-1.75,1.25) {\texttt{derivative 0}};
  % the data cloud sitting entirely in z<0, below the axis to keep text clear
  \foreach \p in {(-2.7,-0.55),(-2.2,-0.55),(-1.7,-0.55),(-1.2,-0.55),(-0.7,-0.55)}
    \fill[red] \p circle (2.4pt);
  \node[red, font=\footnotesize, anchor=north] at (-1.75,-0.75) {\texttt{all inputs land here}};
  \node[green, font=\footnotesize, anchor=west] at (0.55,2.25) {\texttt{alive: slope 1}};
\end{tikzpicture}
$$

The fixes all do the same thing: give the negative side a nonzero slope so a stuck
unit still receives a gradient and can recover.

| unit | negative branch | derivative at $z<0$ | recovers? |
| --- | --- | --- | --- |
| ReLU | $0$ | $0$ | no — gradient is $0$ |
| Leaky ReLU | $\alpha z$, $\alpha\!=\!0.01$ | $\alpha$ | yes — small constant slope |
| PReLU | $\alpha z$, $\alpha$ learned | $\alpha$ | yes — slope adapts per channel |
| ELU | $\alpha(e^{z}-1)$ | $\alpha e^{z}$ | yes — smooth, mean toward $0$ |
| GELU | $z\,\Phi(z)$ | $\Phi(z)+z\phi(z)$ | yes — smooth, near-$0$ slope |

Leaky ReLU and PReLU keep a literal straight line of slope $\alpha$ on the left so
$g'(z)=\alpha>0$ everywhere; a dead-looking unit still receives an $\alpha$-scaled
gradient and can recover. ELU and GELU replace the kink with a smooth curve whose
negative-side derivative is small but strictly positive, which additionally pushes
the activation mean toward zero, a self-normalizing effect that helps the next
layer's pre-activations stay near the high-gradient region.[^chollet-act]

$$
% caption: ReLU versus Leaky ReLU. Both are the identity for $z>0$; for $z<0$ ReLU
% is flat (zero gradient) while Leaky ReLU keeps a small negative slope $\alpha$.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (-3.0,0) -- (4.4,0) node[right, font=\footnotesize] {input};
  \draw[->, thick] (0,-1.1) -- (0,2.3) node[above, font=\footnotesize] {output};
  % ReLU (blue): flat then slope 1
  \draw[acc, very thick] (-2.9,0) -- (0,0) -- (2.0,2.0);
  % Leaky ReLU (red): slight negative slope then slope 1
  \draw[red, very thick] (-2.9,-0.58) -- (0,0) -- (2.0,2.0);
  \node[acc, font=\footnotesize, anchor=south east] at (-1.4,0.08) {\texttt{ReLU (flat)}};
  \node[red, font=\footnotesize, anchor=north east] at (-1.9,-0.42) {\texttt{Leaky (slope a)}};
  \node[font=\footnotesize, anchor=west] at (2.4,1.4) {both: slope $1$};
\end{tikzpicture}
$$

## Output units, not hidden units

The hidden-layer rules above optimize for gradient flow. The **output** unit is
chosen instead to match the target's range and the loss, and the saturating units
that fail in hidden layers become the right choice at the output, where they pair
with a log-loss that cancels the saturation. The output choice is tied to the
[loss](/deep-learning/neural-networks/loss-functions-and-output-units); the table
below is the standard pairing.

| task | output activation | range | paired loss |
| --- | --- | --- | --- |
| regression | none (linear) | $\mathbb{R}$ | mean squared error |
| binary classification | sigmoid | $(0,1)$ | binary cross-entropy |
| multiclass | softmax | simplex, sums to $1$ | categorical cross-entropy |
| count / nonnegative | softplus or exp | $(0,\infty)$ | Poisson / MSE |

> **Definition (Softmax).** The vector activation
> $\softmax(z)_i = e^{z_i}/\sum_j e^{z_j}$, mapping a vector of logits
> to a probability distribution over $K$ classes. It is the multiclass
> generalization of the sigmoid, and pairs with cross-entropy so the
> log cancels the exponential and leaves the clean gradient $\hat{y}-y$.[^gf-output]

The cancellation is worth seeing once. With softmax output $\hat{y} =
\softmax(z)$ and cross-entropy loss $\mathcal{L} = -\sum_i y_i
\log \hat{y}_i$ against a one-hot target $y$, the gradient with respect to the
logits collapses to

$$
\frac{\partial \mathcal{L}}{\partial z_i} = \hat{y}_i - y_i .
$$

The exponential of the softmax and the logarithm of the loss cancel, leaving a
gradient that is just the prediction error, bounded and never vanishing even when
$\hat{y}$ saturates near $0$ or $1$. Pairing the saturating output with the wrong
loss (say squared error on a sigmoid) reintroduces a $\sigma'(z)$ factor and the
saturation stalls learning — the reason the pairing in the table is not optional.

## Which activation to use where

Defaults, in the order you should reach for them:

| layer / task | first choice | why | fall back to |
| --- | --- | --- | --- |
| hidden (MLP, CNN) | ReLU | cheap, no right-side saturation | Leaky ReLU / ELU if units die |
| hidden (Transformers) | GELU | smooth, strong empirically | Swish / SiLU |
| hidden, self-normalizing | SELU / ELU | drives mean toward $0$ | — |
| output, regression | linear | unbounded targets | — |
| output, binary | sigmoid | gives a probability | — |
| output, multiclass | softmax | distribution over classes | — |
| recurrent gates | sigmoid + tanh | gating needs $(0,1)$; state needs $(-1,1)$ | — |

> **Remark (The practical default).** Start every hidden layer with ReLU. If you
> watch the activation histograms and a large fraction of units sit dead at $0$,
> switch to Leaky ReLU or ELU; if you are training a Transformer, use GELU. Reserve
> sigmoid and tanh for output and gating, where their bounded range is what is
> wanted, and pair sigmoid/softmax outputs with cross-entropy so the
> saturation cancels.[^stevens-act]

## Activations after ReLU

Goodfellow's catalog predates the activations that now dominate large models. Two
later developments matter, and both were settled empirically rather than from
first principles.

**GELU and swish were found empirically.** Hendrycks & Gimpel's
_Gaussian Error Linear Unit_ (2016) motivated $z\,\Phi(z)$ as multiplying the input
by the probability a standard normal falls below it — a smooth, stochastic-looking
gate rather than a hard ReLU switch. Around the same time Ramachandran, Zoph & Le
(2017) ran an automated _search_ over candidate activation formulas and the winner,
$z\,\sigma(\beta z)$, they named **Swish**; it is the SiLU of the table with a
learnable or fixed gain $\beta$. Neither unit is dramatically better than ReLU on
small networks, but on deep Transformers the smoothness and the small negative dip
consistently shave a little off the loss, which is why GELU became the default in
BERT and GPT-family models.

**Gated linear units reshaped the Transformer's MLP.** Dauphin et al.
(2017) introduced the **gated linear unit** (GLU), which splits a projection into
two halves and lets one gate the other, $g(x) = (xW_1) \odot \sigma(xW_2)$. Shazeer
(2020) swapped the sigmoid gate for a swish gate and found **SwiGLU**,
$(xW_1)\odot\swish(xW_2)$, improved Transformer feed-forward blocks
enough that it is now standard in models such as LLaMA and PaLM. The lesson's
scalar-activation view still holds — the nonlinearity is where the depth comes
from — but the modern feed-forward block gates two linear projections against each
other rather than applying one pointwise $g$.

A quantitative footnote on the vanishing-gradient section: the modern answer is not
only "use ReLU" but "**normalize**." Batch normalization (Ioffe & Szegedy, 2015)
and layer normalization (Ba, Kiros & Hinton, 2016) rescale each layer's
pre-activations to keep them near the high-gradient region, so even saturating
units stay trainable at depth. Normalization and residual connections together are
what keep deep stacks trainable despite the $0.25^{L}$ arithmetic above.[^beyond-act]

## Takeaways

- The activation is the only nonlinearity in a layer; its **derivative** is what
  backpropagation multiplies, so the size of $g'$ governs gradient flow.
- $\sigma' = \sigma(1-\sigma)$ peaks at $0.25$ and $\tanh'=1-\tanh^2$ peaks at $1$;
  both **saturate** in the tails, and a product of $L$ small factors vanishes
  geometrically with depth — the **vanishing-gradient** problem.
- **ReLU** has derivative $\mathbf{1}[z>0]$: no right-side saturation, so deep
  stacks train, but a unit driven entirely negative is **dead** ($g=g'=0$ forever).
- **Leaky ReLU / PReLU / ELU / GELU / swish** give the negative side a nonzero
  slope so no unit can be permanently frozen; ELU and GELU additionally center the
  mean; **softplus** and **maxout** round out the smooth and piecewise-linear ends.
- Hidden layers optimize for gradient flow (ReLU/GELU); **output** units match the
  target's range and loss (linear / sigmoid / softmax), where saturation is desired
  and cancelled by cross-entropy.

[^gf-hidden]: **Goodfellow**, _Deep Learning_, §6.3 — Hidden Units: the catalog of ReLU and its generalizations (Leaky/PReLU/ELU/Maxout) versus the saturating sigmoid and tanh, and why rectified units are the modern default.
[^gf-vanish]: **Goodfellow**, _Deep Learning_, §8.2.5 / §10.7 — the vanishing- and exploding-gradient problem: backprop multiplies per-layer Jacobians, so derivatives below $1$ shrink the gradient geometrically with depth.
[^gf-saturate]: **Goodfellow**, _Deep Learning_, §6.3.2 / §8.4 — saturation and initialization: careful weight initialization buys precisely this, pre-activations near $0$, where $g'$ is largest.
[^gf-output]: **Goodfellow**, _Deep Learning_, §6.2.2 — Output Units: sigmoid and softmax paired with the negative log-likelihood, so the loss's $\log$ undoes the unit's $\exp$ and the saturation that would otherwise stall learning.
[^chollet-act]: **Chollet**, _Deep Learning with Python_, Ch. 4 — practical guidance on choosing activations: ReLU as the hidden-layer default, sigmoid/softmax at the output, and the failure modes that motivate the alternatives.
[^stevens-act]: **Stevens et al.**, _Deep Learning with PyTorch_, Ch. 6 — activations in practice: `nn.Tanh` and `nn.ReLU` in a fitted network, and the effect of the choice on the training curve.
[^beyond-act]: Primary sources: Hendrycks & Gimpel, "Gaussian Error Linear Units (GELUs)" (2016); Ramachandran, Zoph & Le, "Searching for Activation Functions" (2017) for Swish/SiLU; Dauphin et al., "Language Modeling with Gated Convolutional Networks" (2017) for the GLU; Shazeer, "GLU Variants Improve Transformer" (2020) for SwiGLU; Ioffe & Szegedy, "Batch Normalization" (2015) and Ba, Kiros & Hinton, "Layer Normalization" (2016) for keeping pre-activations in the high-gradient region.