Activation Functions

A layer is an affine map followed by a pointwise nonlinearity, $h=g(Wx+b)$. The affine part carries all the parameters; the nonlinearity $g$ supplies the entire nonlinear effect: strip it out and a deep stack collapses to a single linear map. The choice of $g$ therefore matters: it sets how gradients flow back through depth, whether units saturate, and how fast the network trains. This lesson catalogs the standard choices and their derivatives.

To make the requirement precise, suppose two layers with $g$ the identity: $h_2=W_2(W_{1}x+b_1)+b_2=(W_2{W}_1)x+(W_2{b}_1+b_2)$. The composite is one affine map with weight $W_2{W}_1$ and bias $W_2{b}_1+b_2$, and no amount of stacking escapes the affine class. A single nonlinear $g$ between the two multiplications is what lets a network represent a function that no single affine map can. Everything below is about which $g$ to pick and what each choice costs in the backward pass.

The catalog

Every hidden unit in common use is one scalar function and its derivative. The derivative column matters most: during the backward pass the gradient at a unit is multiplied by $g^{\prime }(z)$, so the size of $g^{\prime }$ controls how much signal passes through. The following table is the reference the rest of the lesson expands.

Here $\sigma$ is the logistic sigmoid, $\Phi$ and $\varphi$ are the standard-normal CDF and PDF, and $\alpha$ is a small positive constant (or a learned parameter for PReLU). Two families fall out of the table: the saturating units (sigmoid, tanh) whose derivative vanishes at both extremes, and the non-saturating rectified units (ReLU onward) whose derivative stays bounded away from zero on at least the positive half-line.¹

Every derivative, derived

Backpropagation never needs the activation itself in the backward pass, only its derivative. Each derivative below takes one or two lines to derive.

Sigmoid

Write $\sigma (z)=(1+e^{-z}{)}^{-1}$ and differentiate by the chain rule:

The first factor is $\sigma (z)$; the second is $1-\sigma (z)$, since $1-\sigma =1-\frac{1}{1+e^{-z}}=\frac{e^{-z}}{1+e^{-z}}$. Hence

Two consequences read straight off this form. The derivative is expressed in the output $\sigma$, so a forward pass that cached $h=\sigma (z)$ needs no re-exp in the backward pass, only $h(1-h)$. And because $\sigma \in (0,1)$, the product $\sigma (1-\sigma )$ is a downward parabola in $\sigma$ capped at its vertex, which is the source of the $\frac{1}{4}$ ceiling derived below.

Tanh

The tanh derivative follows the same pattern. With $\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}}$, the quotient rule gives numerator $(e^z+e^{-z}{)}^2-(e^z-e^{-z}{)}^2$ over $(e^z+e^{-z}{)}^2$, and expanding the two squares,

so

Since $\tanh =2\sigma (2z)-1$, tanh is a rescaled, recentered sigmoid; the zero-centering lifts its peak derivative to $1$ (at $z=0$) versus the sigmoid's $\frac{1}{4}$, and makes tanh the better of the two saturating choices.

ReLU

ReLU is differentiable everywhere except the kink at $z=0$, and its derivative is just the indicator of the active region:

with the value at $z=0$ taken by convention to be $0$ (frameworks pick $0$ or $1$; the measure-zero choice is irrelevant in practice). This step-function derivative, a hard gate of $1$ or $0$, is the source of both ReLU's strength and its dead-unit pathology.

Leaky ReLU and PReLU

Leaky ReLU replaces the flat left branch with a shallow line of slope $\alpha$,

The derivative is now $\alpha >0$ on the entire negative half-line rather than $0$, so a unit sitting in the negative region still receives an $\alpha$-scaled gradient. PReLU is identical except $\alpha$ is a learned parameter (one per channel), so the negative slope adapts during training rather than being fixed at $0.01$.

ELU

ELU keeps the identity for $z>0$ but replaces the kink with a smooth exponential that saturates gently toward $-\alpha$ on the left:

The left derivative $\alpha e^z$ is strictly positive and continuous, and at $z=0$ both branches agree ($g^{\prime }=1$ from the right, $\alpha e^0=\alpha$ from the left; with the common choice $\alpha =1$ the derivative is continuous too). Because the negative branch bottoms out at $-\alpha$ instead of $0$, ELU can output negative values, which pulls each layer's mean activation toward zero and keeps the next layer's pre-activations nearer the high-gradient region.

Softplus

Softplus is a smooth ReLU; its derivative is the sigmoid, which is why it appears whenever a strictly positive, differentiable output is wanted:

As $z\to +\infty$, $\log (1+e^z)\to z$ (it hugs the ReLU line); as $z\to -\infty$ it decays to $0$ smoothly. Its derivative never reaches $0$ or $1$ exactly, so it has no dead region but also no perfectly flat gradient of $1$.

GELU and swish/SiLU

GELU gates the input by the probability that a standard normal falls below it, $g(z)=z\Phi (z)$ with $\Phi$ the standard-normal CDF and $\varphi$ its density. The product rule gives

Swish (a.k.a. SiLU) is the same idea with a logistic gate, $g(z)=z\sigma (z)$, so

Both are smooth, both are the identity in the large-$z$ limit, and both dip slightly below zero for moderately negative $z$ before returning to $0$ — a non-monotone shape that the empirical record favors in deep Transformers. Their minimum sits near $z\approx -1.28$ for swish (value $\approx -0.28$) and near $z\approx -0.75$ for GELU (value $\approx -0.17$).

Maxout

Maxout is the outlier: it is not a fixed scalar shape but a piecewise-linear unit that takes the max over $k$ affine pieces,

The gradient flows entirely through the winning piece, so maxout has no flat region at all as long as one piece is increasing; with $k=2$ and $w_2=0$ it recovers ReLU exactly. The cost is $k\times$ the parameters, which is why it is rarely the default.

The rectified family shares one shape below, all identity on the right and differing only in how they treat the negative side: hard zero (ReLU), a shallow line (Leaky/PReLU), or a smooth curve that dips and returns (ELU, GELU, swish).

The vanishing-gradient problem, made quantitative

The backward pass through an $L$-layer network multiplies one Jacobian factor per layer. Along a single path of units the gradient that reaches layer $k$ carries a product of activation derivatives,

so the magnitude of the gradient at depth $k$ scales like the product of the per-layer derivatives $g^{\prime }(z^{(\ell )})$ (times the weights). If each factor is below $1$, the product shrinks geometrically with depth, and the deep layers stop receiving signal.²

The sigmoid is the worst case: its derivative is small even at its maximum. Maximizing ${\sigma }^{\prime }(z)=\sigma (1-\sigma )$ over $\sigma \in (0,1)$: set $p=\sigma$ and differentiate $p(1-p)=p-p^2$, giving $1-2p=0$, so the peak is at $p=\frac{1}{2}$, i.e. $z=0$:

So a sigmoid contributes at most $0.25$ per layer, and far less in the tails. Work the tail out numerically: at $z=4$, $\sigma (4)=\frac{1}{1+e^{-4}}\approx 0.982$, and ${\sigma }^{\prime }(4)=0.982(1-0.982)\approx 0.0177$. The tanh is better (it peaks at ${\tanh }^{\prime }(0)=1$) but still vanishes in the tails, with ${\tanh }^{\prime }(2)=1-{\tanh }^{2}2=1-0.964^2\approx 0.071$.

Ten sigmoid layers, each at their best point, already attenuate the gradient by a factor of $10^6$, and the realistic factor is worse because most units sit off their peak. This is why deep sigmoid/tanh stacks were nearly untrainable before ReLU, and why initialization that keeps pre-activations near $z=0$ (where $g^{\prime }$ is largest) matters so much; see initialization.³

The product-of-derivatives view makes the depth penalty geometric, not additive. The next figure shows the same gradient magnitude collapsing layer by layer as it propagates backward through a saturating stack.

ReLU and the dead-unit problem

ReLU sidesteps vanishing gradients on its active side outright: for $z>0$ the derivative is exactly $1$, so the product across active layers is $1^L=1$, with no attenuation at all. That single property, more than any other, made very deep networks trainable.

The flat left half causes a different failure. A unit whose pre-activation is negative for every training example outputs $0$ and has derivative $0$, so it receives no gradient and never updates: it is dead.

The mechanism is worth tracing through the chain rule. The gradient on a weight $w_i$ of a ReLU unit is

Once $z<0$ on every example, the factor $1[z>0]$ is $0$ for every one, so $\partial \mathcal{L}/\partial w_i=0$ regardless of the upstream gradient or the inputs. Nothing in the update can move $w$ or $b$, so the unit cannot recover: the failure is irreversible.

The fixes all do the same thing: give the negative side a nonzero slope so a stuck unit still receives a gradient and can recover.

Leaky ReLU and PReLU keep a literal straight line of slope $\alpha$ on the left so $g^{\prime }(z)=\alpha >0$ everywhere; a dead-looking unit still receives an $\alpha$-scaled gradient and can recover. ELU and GELU replace the kink with a smooth curve whose negative-side derivative is small but strictly positive, which additionally pushes the activation mean toward zero, a self-normalizing effect that helps the next layer's pre-activations stay near the high-gradient region.⁴

Output units, not hidden units

The hidden-layer rules above optimize for gradient flow. The output unit is chosen instead to match the target's range and the loss, and the saturating units that fail in hidden layers become the right choice at the output, where they pair with a log-loss that cancels the saturation. The output choice is tied to the loss; the table below is the standard pairing.

The cancellation is worth seeing once. With softmax output $\hat{y}=\mathrm{softmax}(z)$ and cross-entropy loss $\mathcal{L}=-{\sum }_i{y}_i\log {\hat{y}}_i$ against a one-hot target $y$, the gradient with respect to the logits collapses to

The exponential of the softmax and the logarithm of the loss cancel, leaving a gradient that is just the prediction error, bounded and never vanishing even when $\hat{y}$ saturates near $0$ or $1$. Pairing the saturating output with the wrong loss (say squared error on a sigmoid) reintroduces a ${\sigma }^{\prime }(z)$ factor and the saturation stalls learning — the reason the pairing in the table is not optional.

Which activation to use where

Defaults, in the order you should reach for them:

Activations after ReLU

Goodfellow's catalog predates the activations that now dominate large models. Two later developments matter, and both were settled empirically rather than from first principles.

GELU and swish were found empirically. Hendrycks & Gimpel's Gaussian Error Linear Unit (2016) motivated $z\Phi (z)$ as multiplying the input by the probability a standard normal falls below it — a smooth, stochastic-looking gate rather than a hard ReLU switch. Around the same time Ramachandran, Zoph & Le (2017) ran an automated search over candidate activation formulas and the winner, $z\sigma (\beta z)$, they named Swish; it is the SiLU of the table with a learnable or fixed gain $\beta$. Neither unit is dramatically better than ReLU on small networks, but on deep Transformers the smoothness and the small negative dip consistently shave a little off the loss, which is why GELU became the default in BERT and GPT-family models.

Gated linear units reshaped the Transformer's MLP. Dauphin et al. (2017) introduced the gated linear unit (GLU), which splits a projection into two halves and lets one gate the other, $g(x)=(x{W}_1)\odot \sigma (x{W}_2)$. Shazeer (2020) swapped the sigmoid gate for a swish gate and found SwiGLU, $(x{W}_1)\odot \mathrm{swish}(x{W}_2)$, improved Transformer feed-forward blocks enough that it is now standard in models such as LLaMA and PaLM. The lesson's scalar-activation view still holds — the nonlinearity is where the depth comes from — but the modern feed-forward block gates two linear projections against each other rather than applying one pointwise $g$.

A quantitative footnote on the vanishing-gradient section: the modern answer is not only use ReLU but normalize. Batch normalization (Ioffe & Szegedy, 2015) and layer normalization (Ba, Kiros & Hinton, 2016) rescale each layer's pre-activations to keep them near the high-gradient region, so even saturating units stay trainable at depth. Normalization and residual connections together are what keep deep stacks trainable despite the $0.25^L$ arithmetic above.⁷

Takeaways

• The activation is the only nonlinearity in a layer; its derivative is what backpropagation multiplies, so the size of $g^{\prime }$ governs gradient flow.
• ${\sigma }^{\prime }=\sigma (1-\sigma )$ peaks at $0.25$ and ${\tanh }^{\prime }=1-{\tanh }^2$ peaks at $1$; both saturate in the tails, and a product of $L$ small factors vanishes geometrically with depth — the vanishing-gradient problem.
• ReLU has derivative $1[z>0]$: no right-side saturation, so deep stacks train, but a unit driven entirely negative is dead ($g=g^{\prime }=0$ forever).
• Leaky ReLU / PReLU / ELU / GELU / swish give the negative side a nonzero slope so no unit can be permanently frozen; ELU and GELU additionally center the mean; softplus and maxout round out the smooth and piecewise-linear ends.
• Hidden layers optimize for gradient flow (ReLU/GELU); output units match the target's range and loss (linear / sigmoid / softmax), where saturation is desired and cancelled by cross-entropy.