LSTM & GRU

A recurrent network folds a sequence into a single hidden state by applying the same transition at every step,

The defect is structural, not a tuning problem: information from step $t$ reaches step $T$ only by passing through $T-t$ multiplications by $W$. Training backpropagates the loss along that same chain, and the chain's Jacobian is a product of $T-t$ matrices. Products of matrices do not stay bounded; they collapse to zero or blow up, so the gradient linking step $t$ to a much later target is effectively lost.¹ This lesson builds the standard fix: a gated, additive memory path whose Jacobian is near the identity.

Why gating: the vanishing Jacobian product

Differentiate the plain recurrence. Write $a_t=W{h}_{t-1}+U{x}_t+b$ so that $h_t=\tanh (a_t)$, and let $D_t=\mathrm{diag}\left(1-h_t^2\right)$ be the diagonal Jacobian of the $\tanh$ at step $t$. One step back is

and the loss $\mathcal{L}$ at the final step depends on the early state $h_t$ only through the long chain of these factors:

The whole problem lives in that product of $T-t$ matrices. Diagonalize $W=Q\Lambda Q^{-1}$ and ignore the bounded $D_k$ factors for the magnitude argument: the product scales like ${\Lambda }^{T-t}$, so each eigenvalue ${\lambda }_i$ is raised to the power $T-t$.

As a scalar caricature: with $W$ a single number $w$ and $D_k\approx 1$, the factor is $w^{T-t}$: at $w=0.9$ over $100$ steps it is $0.9^{100}\approx 3\times 10^{-5}$; at $w=1.1$ it is $1.1^{100}\approx 1.4\times 10^4$. Either way the signal linking distant steps is destroyed.

The fix is to replace the multiplicative carry with an additive one. If the state is updated by adding rather than multiplying, the one-step Jacobian becomes (near) the identity, and a product of identities neither shrinks nor grows. That additive carry is the cell state, and the gates are what decide, per coordinate and per step, how much to add and how much to keep.

The LSTM cell

The long short-term memory (LSTM) cell threads a cell state $c_t$ through time along a near-uninterrupted highway, and surrounds it with three multiplicative gates — each a sigmoid-squashed linear map producing values in $(0,1)$ that act as soft, per-coordinate switches.² Writing $[h_{t-1},x_t]$ for the concatenation of the previous hidden state and the current input, the cell is six equations:

Here $\odot$ is the elementwise (Hadamard) product and $\sigma$ the logistic sigmoid. The single load-bearing line is the cell update: the old cell $c_{t-1}$ survives, scaled by the forget gate $f_t$, and a candidate ${\tilde{c}}_t$ is added in, scaled by the input gate $i_t$. The structure is the signature diagram below.

Fix the dimensions once and every shape follows. Let the input be $x_t\in {\mathbb{R}}^n$ and the hidden and cell states be $h_t,c_t\in {\mathbb{R}}^d$. The concatenation $[h_{t-1},x_t]\in {\mathbb{R}}^{d+n}$ stacks the two. Every gate and the candidate read this same vector through its own weight block, so each weight matrix maps ${\mathbb{R}}^{d+n}\to {\mathbb{R}}^d$ and each bias lives in ${\mathbb{R}}^d$:

Every intermediate — $f_t$, $i_t$, ${\tilde{c}}_t$, $o_t$, $c_t$, $h_t$ — is a vector in ${\mathbb{R}}^d$, and the three products $f_t\odot c_{t-1}$, $i_t\odot {\tilde{c}}_t$, $o_t\odot \tanh (c_t)$ are elementwise, so shapes never leave ${\mathbb{R}}^d$ after the concatenation is consumed. In practice the four blocks are stacked into one matrix $W\in {\mathbb{R}}^{4d\times (d+n)}$ and applied in a single matmul, then sliced into the four pre-activations — one dense layer of width $4d$ does the whole gate computation.

A count follows immediately. Each of the four blocks holds $d(d+n)$ weights and $d$ biases, so an LSTM layer has

parameters. For $n=100$ inputs and $d=256$ hidden units that is $4\cdot 256\cdot 357=365,568$ weights — dominated by the $4d^2$ recurrent term, which is why hidden width, not input width, sets an LSTM's cost.

The structure is the signature diagram below.

The three gates differ only in what they multiply, not in form. Each is a learned, input-dependent decision applied elementwise to one signal in the cell.

For a worked step, take a two-dimensional cell, $d=2$, carried state $c_{t-1}=(2.0,-1.0)$, and suppose the gates and candidate come out as

The cell update runs coordinate by coordinate:

The first coordinate kept almost all of its old value ($f=0.9$) and added a little; the second nearly forgot its old value ($f=0.1$) and was overwritten by the candidate. The read-out then squashes and gates:

Two coordinates, two independent decisions — this is the per-coordinate control the gates provide.

The forward pass is one sweep of these six lines across the sequence, carrying $(c,h)$ from step to step.

The cell-state highway

The reason gating works is one derivative. Differentiate the cell update $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$ with respect to $c_{t-1}$. The second term does not contain $c_{t-1}$, and to first order the gates are functions of $h_{t-1}$, not of $c_{t-1}$ directly, so

There is no weight matrix in this Jacobian, no repeated $W$ to raise to a power. Backpropagating the cell state across many steps multiplies these diagonal gate factors:

When the forget gates stay open, $f_k\approx 1$, the product is $\approx 1$: the gradient passes through unattenuated, no matter how large $T-t$ is. This is the constant error carousel made precise.³

The contrast is the whole point of the architecture: two paths from $c_t$ back to $c_{t-1}$, one near-identity and one decaying.

A practical corollary: LSTM forget-gate biases $b_f$ are initialized positive (often $+1$), so that $f_t=\sigma (b_f)\approx 0.73$ or higher at the start of training: the highway begins open, and the cell remembers by default until the network learns when to forget.

Gate behavior

The gate is a soft switch. Because it is a sigmoid, its value slides continuously in $(0,1)$, but the two ends are what carry meaning: near $0$ it blocks its signal, near $1$ it passes it. Applied to the forget gate, this is precisely the choice between erasing and preserving a memory.

The gate values are not hand-set; each is the output of a learned linear map of $[h_{t-1},x_t]$, so the network discovers from data when a coordinate of the cell should be held, overwritten, or read out.⁴

The GRU cell

The gated recurrent unit (GRU) is a streamlined gating with two gates instead of three and a single state vector: it drops the separate cell state, letting $h_t$ serve as both memory and output.⁵ An update gate $z_t$ interpolates between carrying the old state and writing a candidate; a reset gate $r_t$ controls how much past state enters that candidate:

The convex combination in the last line is the GRU's highway: when $z_t\approx 0$ the state is copied forward, $h_t\approx h_{t-1}$, giving the same near-identity carry as the LSTM cell. A single gate $z_t$ does the job the LSTM splits between forget and input — here keep and write are tied, $f\equiv 1-i$.

The dimensions mirror the LSTM with one block fewer. With $x_t\in {\mathbb{R}}^n$ and $h_t\in {\mathbb{R}}^d$, the three weight blocks each map the concatenation $[h_{t-1},x_t]\in {\mathbb{R}}^{d+n}$ into ${\mathbb{R}}^d$:

The candidate reads $[r_t\odot h_{t-1},x_t]$ rather than $[h_{t-1},x_t]$ — the reset gate masks the past before it enters — but the shape is unchanged, still ${\mathbb{R}}^{d+n}$. Three blocks instead of four gives

parameters, exactly three-quarters of the LSTM's $4d(d+n+1)$. For the same $n=100$, $d=256$ that is $3\cdot 256\cdot 357=274,176$ weights against the LSTM's $365,568$.

The update-gate interpolation reads like the LSTM's cell update with the two gates fused. Take $d=2$ again, $h_{t-1}=(0.4,-0.2)$, and

Then

The first coordinate mostly carried its old value ($z=0.2$, so $1-z=0.8$ on the past); the second mostly rewrote to the candidate ($z=0.9$). The single gate $z_t$ must trade keeping against writing — the LSTM, with separate $f_t$ and $i_t$, can keep and write the same coordinate, or do neither.

GRU and LSTM share the additive-highway principle; they differ in bookkeeping. The GRU has fewer parameters (three weight blocks rather than four) and one state to carry, which makes it lighter and often faster to train; the LSTM's separate cell and explicit output gate give it a finer-grained, sometimes stronger memory.

Persistence across time

Unrolled, both architectures are the same cell applied at every step, with the memory carried along the top from start to finish. The forget/update gate is what lets a coordinate survive intact across the whole span while the input rewrites others — the network holds some facts and revises the rest.

This is the architectural payoff. The plain RNN's hidden state was the only channel, and it decayed; the gated cell adds a second channel whose default behavior is to remember, and whose forgetting is a learned operation rather than an unavoidable side effect of repeated matrix multiplication. Gated RNNs carried most sequence modeling for years, until the parallelism and longer effective context of attention and transformers displaced recurrence for the largest models.

Which gates earn their keep

The LSTM in Goodfellow's Chapter 10 is the 1997 Hochreiter–Schmidhuber cell plus the forget gate (Gers et al., 2000). A few results since then sharpen which parts of it actually matter.

• Which gates matter. Greff et al., LSTM: A Search Space Odyssey (IEEE TNNLS, 2017), ablated every component across thousands of runs. The finding is blunt: the forget gate and the output activation are essential — removing either collapses performance — while peephole connections and the input-gate/forget- gate coupling have little effect. This is the empirical justification for the GRU's design: it keeps the forget-like update gate and drops nearly everything the study found expendable.
• The highway in numbers. With the forget gate held at $f_t=0.98$, the cell-state Jacobian across $100$ steps is ${\prod }_t{f}_t=0.98^{100}\approx 0.13$ — the gradient shrinks by less than an order of magnitude. Compare the plain RNN from the recurrent-networks lesson, where $\rho =0.9$ gave $0.9^{100}\approx 2.7\times 10^{-5}$. The gate's job is to keep that per-step multiplier as close to $1$ as the task allows; at $f_t=1$ the product is exactly $1$ and memory is perfect.
• Normalization inside the cell. Batch normalization does not fit a recurrence (statistics differ per step), but layer normalization applied to the gate pre-activations (Ba et al., 2016) stabilizes LSTM training and became standard for deep recurrent stacks — the same per-token normalization that later anchors the Transformer block.

The modern verdict is mixed rather than a clean replacement. The Transformer won the largest models on parallelism, but gated recurrence never fully left: it remains the default for strict streaming and low-latency inference (constant memory per step, no growing context), and recent work such as the xLSTM (Beck et al., 2024) revisits the cell with matrix memory and parallel training, so the additive-highway idea is still live.

Takeaways

• The plain RNN propagates state through a repeated weight-matrix multiply; the backward Jacobian is a product ${\prod }_k{D}_{k}W\sim \rho (W{)}^{T-t}$ that vanishes ($\rho <1$) or explodes ($\rho >1$), killing long-range gradients.
• Gating replaces the multiplicative carry with an additive one. The LSTM cell update $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$ has Jacobian $\partial c_t/\partial c_{t-1}=\mathrm{diag}(f_t)$ — no weight matrix, so with $f_t\approx 1$ the gradient rides a near-identity highway.
• The LSTM has three gates (forget, input, output) and two state vectors (cell $c_t$, hidden $h_t$); the GRU merges these into two gates (update, reset) and one state, with fewer parameters and a tied keep/write.
• Both are the same cell unrolled across time; the forget/update gate is the learned switch that lets a memory persist intact or be overwritten, step by step.