---
title: LSTM & GRU
module: Architectures
moduleNumber: 5
lessonNumber: 4
order: 504
summary: >
  A plain recurrent network propagates its hidden state through a repeated
  weight-matrix multiply, and the Jacobian product that results vanishes or
  explodes long before a useful gradient can reach the early steps. Gated RNNs
  fix this with an additive memory path: a cell state that is carried forward
  almost unchanged, past which the gradient flows along a near-identity
  highway. We derive that highway, give the full LSTM and GRU equations, and
  compare the two.
topics: [Architectures]
sources:
  - book: Goodfellow
    ref: "§10.10 — The Long Short-Term Memory and Other Gated RNNs"
  - book: Goodfellow
    ref: "§10.7 — The Challenge of Long-Term Dependencies"
  - book: Chollet
    ref: "§6.2 — Understanding Recurrent Neural Networks (LSTM/GRU)"
---

A [recurrent network](/deep-learning/architectures/recurrent-networks) folds a
sequence into a single hidden state by applying the _same_ transition at every
step,

$$
h_t = \tanh\!\parens{W h_{t-1} + U x_t + b},
\qquad t = 1, \dots, T.
$$

The defect is structural, not a tuning problem: information from step $t$ reaches
step $T$ only by passing through $T - t$ multiplications by $W$. Training
backpropagates the loss along that same chain, and the chain's Jacobian is a
_product_ of $T-t$ matrices. Products of matrices do not stay bounded; they
collapse to zero or blow up, so the gradient linking step $t$ to a
much later target is effectively lost.[^gf-longterm] This lesson builds the standard fix: a
gated, _additive_ memory path whose Jacobian is near the identity.

## Why gating: the vanishing Jacobian product

Differentiate the plain recurrence. Write $a_t = W h_{t-1} + U x_t + b$ so that
$h_t = \tanh(a_t)$, and let $D_t = \diag\!\parens{1 - h_t^2}$ be the
diagonal Jacobian of the $\tanh$ at step $t$. One step back is

$$
\frac{\partial h_t}{\partial h_{t-1}}
= D_t\,W,
$$

and the loss $\mathcal{L}$ at the final step depends on the early state
$h_{t}$ only through the long chain of these factors:

$$
\frac{\partial \mathcal{L}}{\partial h_t}
= \frac{\partial \mathcal{L}}{\partial h_T}
\prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}}
= \frac{\partial \mathcal{L}}{\partial h_T}
\prod_{k=t+1}^{T} D_k\,W.
$$

The whole problem lives in that product of $T - t$ matrices. Diagonalize
$W = Q \Lambda Q^{-1}$ and ignore the bounded $D_k$ factors for the magnitude
argument: the product scales like $\Lambda^{\,T-t}$, so each eigenvalue
$\lambda_i$ is raised to the power $T - t$.

> **Theorem (Vanishing and exploding gradients).** For the plain recurrence
> $h_t = \tanh(W h_{t-1} + \dots)$, the gradient norm of the loss with respect to
> an early state $h_t$ obeys, for spectral radius $\rho(W)$ and the bound
> $\lVert D_k \rVert \le 1$,
> $$
> \norm{ \frac{\partial \mathcal{L}}{\partial h_t} }
> \;\lesssim\;
> \rho(W)^{\,T-t}\,
> \norm{ \frac{\partial \mathcal{L}}{\partial h_T} }.
> $$
> Hence the gradient decays exponentially when $\rho(W) < 1$ (**vanishing**) and
> grows exponentially when $\rho(W) > 1$ (**exploding**); only the measure-zero
> knife-edge $\rho(W) = 1$ is stable.

> **Proof.** Backpropagation through time multiplies the per-step Jacobians, so
> $\partial \mathcal{L}/\partial h_t = (\partial \mathcal{L}/\partial h_T)\prod_{k=t+1}^{T} D_k W$.
> Taking norms and using submultiplicativity with $\lVert D_k\rVert \le 1$ (the
> $\tanh$ derivative $1 - h^2 \in (0,1]$) gives
> $\lVert \partial\mathcal{L}/\partial h_t\rVert \le \lVert \partial\mathcal{L}/\partial h_T\rVert\,\lVert W\rVert^{T-t}$.
> Aligning with the dominant eigenvector replaces $\lVert W\rVert$ by the spectral
> radius $\rho(W)$ in the leading term, and $\rho(W)^{T-t}$ tends to $0$ for
> $\rho < 1$ and to $\infty$ for $\rho > 1$ as $T - t \to \infty$. $\qed$

As a scalar caricature: with $W$ a single number $w$ and
$D_k \approx 1$, the factor is $w^{T-t}$: at $w = 0.9$ over $100$ steps it is
$0.9^{100} \approx 3 \times 10^{-5}$; at $w = 1.1$ it is $1.1^{100} \approx
1.4 \times 10^{4}$. Either way the signal linking distant steps is destroyed.

| Quantity | Plain RNN | What goes wrong |
| --- | --- | --- |
| state update | $h_t = \tanh(W h_{t-1} + \dots)$ | multiplicative, nonlinear |
| one-step Jacobian | $\partial h_t / \partial h_{t-1} = D_t\,W$ | repeated factor $W$ |
| $T-t$-step Jacobian | $\prod_{k} D_k W \sim \Lambda^{T-t}$ | eigenvalues raised to a power |
| long-range gradient | $\propto \rho(W)^{\,T-t}$ | vanishes or explodes |

The fix is to replace the multiplicative carry with an **additive** one. If the
state is updated by _adding_ rather than _multiplying_, the one-step Jacobian
becomes (near) the identity, and a product of identities neither shrinks nor
grows. That additive carry is the **cell state**, and the gates are what decide,
per coordinate and per step, how much to add and how much to keep.

> **Definition (Constant error carousel).** A memory path whose state is updated
> additively, $c_t = c_{t-1} + (\text{increment})$, so that gradient flows
> backward through it with Jacobian $\partial c_t/\partial c_{t-1} \approx 1$: a
> near-identity channel that carries error across many steps without decay.

## The LSTM cell

The **long short-term memory** (LSTM) cell threads a cell state $c_t$ through time
along a near-uninterrupted highway, and surrounds it with three multiplicative
**gates** — each a sigmoid-squashed linear map producing values in $(0,1)$ that
act as soft, per-coordinate switches.[^gf-lstm] Writing $[h_{t-1}, x_t]$ for the
concatenation of the previous hidden state and the current input, the cell is six
equations:

$$
f_t = \sigma\!\parens{W_f [h_{t-1}, x_t] + b_f}
\qquad\text{(forget gate)}
$$

$$
i_t = \sigma\!\parens{W_i [h_{t-1}, x_t] + b_i}
\qquad\text{(input gate)}
$$

$$
\tilde c_t = \tanh\!\parens{W_c [h_{t-1}, x_t] + b_c}
\qquad\text{(candidate cell)}
$$

$$
c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t
\qquad\text{(cell update)}
$$

$$
o_t = \sigma\!\parens{W_o [h_{t-1}, x_t] + b_o}
\qquad\text{(output gate)}
$$

$$
h_t = o_t \odot \tanh(c_t)
\qquad\text{(hidden state)}
$$

Here $\odot$ is the elementwise (Hadamard) product and $\sigma$ the logistic
sigmoid. The single load-bearing line is the cell update: the old cell $c_{t-1}$
survives, scaled by the forget gate $f_t$, and a candidate $\tilde c_t$ is added
in, scaled by the input gate $i_t$. The structure is the signature diagram below.

Fix the dimensions once and every shape follows. Let the input be $x_t \in
\mathbb{R}^{n}$ and the hidden and cell states be $h_t, c_t \in \mathbb{R}^{d}$.
The concatenation $[h_{t-1}, x_t] \in \mathbb{R}^{d+n}$ stacks the two. Every gate
and the candidate read this same vector through its own weight block, so each
weight matrix maps $\mathbb{R}^{d+n} \to \mathbb{R}^{d}$ and each bias lives in
$\mathbb{R}^{d}$:

$$
W_f, W_i, W_c, W_o \in \mathbb{R}^{d \times (d+n)},
\qquad
b_f, b_i, b_c, b_o \in \mathbb{R}^{d}.
$$

Every intermediate — $f_t$, $i_t$, $\tilde c_t$, $o_t$, $c_t$, $h_t$ — is a vector
in $\mathbb{R}^{d}$, and the three products $f_t \odot c_{t-1}$, $i_t \odot
\tilde c_t$, $o_t \odot \tanh(c_t)$ are elementwise, so shapes never leave
$\mathbb{R}^{d}$ after the concatenation is consumed. In practice the four blocks
are stacked into one matrix $W \in \mathbb{R}^{4d \times (d+n)}$ and applied in a
single matmul, then sliced into the four pre-activations — one dense layer of
width $4d$ does the whole gate computation.

A count follows immediately. Each of the four blocks holds $d(d+n)$ weights and
$d$ biases, so an LSTM layer has

$$
4\,\big[\,d(d+n) + d\,\big] = 4d(d+n) + 4d = 4d(d + n + 1)
$$

parameters. For $n = 100$ inputs and $d = 256$ hidden units that is
$4 \cdot 256 \cdot 357 = 365{,}568$ weights — dominated by the $4d^2$ recurrent
term, which is why hidden width, not input width, sets an LSTM's cost.

The structure is the signature diagram below.

$$
% caption: The LSTM cell. The cell state runs along the top highway: scaled by the
% forget gate, written by the input gate, and read out by the output gate.
\begin{tikzpicture}[>=stealth, font=\footnotesize, x=1cm, y=1cm,
  gate/.style={draw, thick, minimum width=13mm, minimum height=9mm, align=center, font=\scriptsize},
  op/.style={circle, draw, black, thick, minimum size=7mm, inner sep=0pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % --- the cell-state highway across the top (y = 4) ---
  \node[green] (cin)  at (-1.0,4) {\texttt{c(t-1)}};
  \node[op] (mult) at (2.4,4) {\texttt{x}};
  \node[op] (add)  at (6.0,4) {$+$};
  \node[green] (cout) at (11.4,4) {\texttt{c(t)}};
  \draw[->, green, very thick] (cin) -- (mult);
  \draw[->, green, very thick] (mult) -- (add);
  \draw[->, green, very thick] (add) -- (cout);
  \node[green, anchor=south, font=\footnotesize] at (4.2,4.3) {\texttt{cell highway}};
  % --- the four gate units (y = 1.4) ---
  \node[gate, draw=acc, text=acc] (fg) at (2.4,1.4) {\texttt{forget}};
  \node[gate, draw=acc, text=acc] (ig) at (4.6,1.4) {\texttt{input}};
  \node[gate, draw=black, text=black] (cg) at (6.8,1.4) {\texttt{cand-}\\\texttt{idate}};
  \node[gate, draw=acc, text=acc] (og) at (9.4,1.4) {\texttt{output}};
  % --- concatenated input bus at the bottom (y = -1) ---
  \node[green] (hin) at (0.2,-1) {\texttt{h(t-1)}};
  \node[black] (xin) at (1.8,-1) {\texttt{x(t)}};
  \draw[green, very thick] (hin) -- (1.0,-0.1);
  \draw[black, very thick] (xin) -- (1.0,-0.1);
  \node[op, fill=acc!12, draw=acc, text=acc, minimum size=5mm] (bus) at (1.0,-0.1) {};
  \node[acc, anchor=west, font=\footnotesize] at (1.35,-0.3) {\texttt{[h(t-1), x(t)]}};
  % feed the bus up into each gate
  \draw[->, black, thick] (bus) -- (fg);
  \draw[->, black, thick] (bus) -- (ig);
  \draw[->, black, thick] (bus) -- (cg);
  \draw[->, black, thick] (bus) -- (og);
  % gates act on the highway
  \draw[->, acc, thick] (fg) -- (mult);                       % forget scales c_{t-1}
  \draw[->, acc, thick] (ig.north) |- (4.6,3.1) -| (add.south west); % input gate into add
  \draw[->, black, thick] (cg) -- (add.south);                % candidate into add
  % --- output read-out on the right ---
  \node[op] (tanh) at (11.4,2.4) {\texttt{tanh}};
  \node[op] (omult) at (11.4,1.4) {\texttt{x}};
  \node[acc] (hout) at (13.4,1.4) {$h_t$};
  \draw[->, green, very thick] (11.4,3.65) -- (tanh);  % c_t down into tanh
  \draw[->, black, thick] (tanh) -- (omult);
  \draw[->, acc, thick] (og) -- (omult);
  \draw[->, acc, very thick] (omult) -- (hout);
\end{tikzpicture}
$$

> **Definition (Gate).** A vector $g_t = \sigma(W_g[h_{t-1}, x_t] + b_g) \in (0,1)^d$
> used as a multiplicative mask: $g_t \odot v$ passes a coordinate of $v$ when the
> corresponding gate value is near $1$ and blocks it when near $0$. The sigmoid
> makes the switch soft and differentiable.

The three gates differ only in what they multiply, not in form. Each is a learned,
input-dependent decision applied elementwise to one signal in the cell.

| Gate | Formula | Role |
| --- | --- | --- |
| forget $f_t$ | $\sigma(W_f [h_{t-1}, x_t] + b_f)$ | how much of $c_{t-1}$ to keep |
| input $i_t$ | $\sigma(W_i [h_{t-1}, x_t] + b_i)$ | how much of the candidate to write |
| candidate $\tilde c_t$ | $\tanh(W_c [h_{t-1}, x_t] + b_c)$ | the new content proposed for the cell |
| output $o_t$ | $\sigma(W_o [h_{t-1}, x_t] + b_o)$ | how much of $\tanh(c_t)$ to expose as $h_t$ |

For a worked step, take a two-dimensional cell, $d = 2$,
carried state $c_{t-1} = (2.0,\ -1.0)$, and suppose the gates and candidate come
out as

$$
f_t = (0.9,\ 0.1),
\quad
i_t = (0.5,\ 0.8),
\quad
\tilde c_t = (1.0,\ -0.5),
\quad
o_t = (0.6,\ 0.9).
$$

The cell update runs coordinate by coordinate:

$$
c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t
= (0.9 \cdot 2.0,\ \ 0.1 \cdot (-1.0)) + (0.5 \cdot 1.0,\ \ 0.8 \cdot (-0.5))
= (2.3,\ -0.5).
$$

The first coordinate kept almost all of its old value ($f = 0.9$) and added a
little; the second nearly forgot its old value ($f = 0.1$) and was overwritten by
the candidate. The read-out then squashes and gates:

$$
h_t = o_t \odot \tanh(c_t)
= (0.6 \cdot \tanh 2.3,\ \ 0.9 \cdot \tanh(-0.5))
\approx (0.58,\ -0.42).
$$

Two coordinates, two independent decisions — this is the per-coordinate control
the gates provide.

The forward pass is one sweep of these six lines across the sequence, carrying
$(c, h)$ from step to step.

```algorithm
caption: $\textsc{LstmStep}(x_t, h_{t-1}, c_{t-1})$ — one cell update
$f \gets \sigma(W_f [h_{t-1}, x_t] + b_f)$ // forget gate
$i \gets \sigma(W_i [h_{t-1}, x_t] + b_i)$ // input gate
$g \gets \tanh(W_c [h_{t-1}, x_t] + b_c)$ // candidate cell
$c_t \gets f \odot c_{t-1} + i \odot g$ // additive cell update
$o \gets \sigma(W_o [h_{t-1}, x_t] + b_o)$ // output gate
$h_t \gets o \odot \tanh(c_t)$ // read-out
return $(h_t, c_t)$
```

## The cell-state highway

The reason gating works is one derivative. Differentiate the cell update
$c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$ with respect to $c_{t-1}$. The
second term does not contain $c_{t-1}$, and to first order the gates are functions
of $h_{t-1}$, not of $c_{t-1}$ directly, so

$$
\frac{\partial c_t}{\partial c_{t-1}} = \diag(f_t).
$$

There is **no weight matrix** in this Jacobian, no repeated $W$ to raise to a
power. Backpropagating the cell state across many steps multiplies these diagonal
gate factors:

$$
\frac{\partial c_T}{\partial c_t}
= \prod_{k=t+1}^{T} \diag(f_k)
= \diag\!\parens{\textstyle\prod_{k=t+1}^{T} f_k}.
$$

When the forget gates stay open, $f_k \approx 1$, the product is $\approx 1$: the
gradient passes through unattenuated, no matter how large $T - t$ is. This is the
constant error carousel made precise.[^gf-carousel]

> **Theorem (Near-identity cell gradient).** Along the LSTM cell path, the
> long-range Jacobian is the elementwise product of forget gates,
> $\partial c_T / \partial c_t = \diag\parens{\prod_{k=t+1}^{T} f_k}$.
> If every $f_k \approx 1$ then $\partial c_T/\partial c_t \approx I$, so the
> gradient neither vanishes nor explodes; contrast the plain RNN's
> $\prod_k D_k W$, which scales as $\rho(W)^{T-t}$.

> **Proof.** From $c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$, treating the
> gates as constants with respect to $c_{t-1}$ (the dominant dependence path), the
> Jacobian of the first term is $\diag(f_t)$ and the second term
> contributes nothing, giving $\partial c_t/\partial c_{t-1} = \diag(f_t)$.
> Chaining over $k = t+1, \dots, T$ multiplies diagonal matrices, which commute and
> multiply entrywise, yielding $\diag(\prod_k f_k)$. With each
> $f_k \to 1$ the product $\to 1$ and the matrix $\to I$. No factor of a shared
> weight matrix appears, so the spectral-radius blow-up of the plain RNN is absent.
> $\qed$

The contrast is the whole point of the architecture: two paths from $c_t$ back to
$c_{t-1}$, one near-identity and one decaying.

$$
% caption: Gradient highway. Along the cell path the Jacobian is near the identity, carrying the signal almost unchanged; along the plain-RNN path it decays toward zero.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  nd/.style={circle, draw, minimum size=6mm, inner sep=0pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % --- cell highway (top): thick, undecaying ---
  \foreach \i/\x in {0/0, 1/2.2, 2/4.4, 3/6.6, 4/8.8} { \node[nd, draw=green] (c\i) at (\x,1.8) {}; }
  \foreach \i/\j in {0/1, 1/2, 2/3, 3/4} {
    \draw[->, green, very thick] (c\i) -- (c\j) node[midway, above, font=\footnotesize, text=green] {\texttt{gate open}};
  }
  \node[green, anchor=east] at (-0.4,1.8) {\texttt{cell}};
  \node[green, anchor=north, font=\footnotesize] at (4.4,1.45) {\texttt{gradient carried (near identity)}};
  % --- plain RNN (bottom): thinning arrows, decaying ---
  \foreach \i/\x in {0/0, 1/2.2, 2/4.4, 3/6.6, 4/8.8} { \node[nd, draw=red] (r\i) at (\x,0) {}; }
  \draw[->, red, very thick]   (r0) -- (r1);
  \draw[->, red, very thick]   (r1) -- (r2);
  \draw[->, red, thick]        (r2) -- (r3);
  \draw[->, red, thick]        (r3) -- (r4);
  \node[red, anchor=east] at (-0.4,0) {\texttt{plain}};
  \node[red, anchor=north, font=\footnotesize] at (4.4,-0.45) {\texttt{gradient decays toward zero}};
\end{tikzpicture}
$$

A practical corollary: LSTM forget-gate biases $b_f$ are initialized **positive**
(often $+1$), so that $f_t = \sigma(b_f) \approx 0.73$ or higher at the start of
training: the highway begins open, and the cell remembers by default until the
network learns when to forget.

## Gate behavior

The gate is a soft switch. Because it is a sigmoid, its value slides continuously
in $(0,1)$, but the two ends are what carry meaning: near $0$ it **blocks** its
signal, near $1$ it **passes** it. Applied to the forget gate, this is precisely
the choice between erasing and preserving a memory.

$$
% caption: A gate near $0$ blocks its signal (memory erased) and near $1$ passes it (preserved); the sigmoid makes the switch soft, so it can pass a fraction.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=9mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % --- BLOCK (gate near 0) ---
  \node[cell, draw=red, text=red] (m0) at (0,0) {\texttt{mem}};
  \node[draw=red, text=red, minimum width=8mm, minimum height=7mm, font=\footnotesize] (g0) at (2.4,0) {\texttt{gate}};
  \node[red, font=\footnotesize, anchor=south] at (2.4,0.55) {\texttt{value near 0}};
  \node[cell, draw=black, text=black] (n0) at (4.8,0) {\texttt{empty}};
  \draw[->, red, thick] (m0) -- (g0);
  \draw[->, red, thick, dotted] (g0) -- (n0) node[midway, above, font=\footnotesize, text=red] {\texttt{block}};
  \node[red, anchor=north, font=\footnotesize] at (2.4,-0.7) {\texttt{memory erased}};
  % --- PASS (gate near 1) ---
  \begin{scope}[yshift=-2.6cm]
    \node[cell, draw=green, text=green] (m1) at (0,0) {\texttt{mem}};
    \node[draw=green, text=green, minimum width=8mm, minimum height=7mm, font=\footnotesize] (g1) at (2.4,0) {\texttt{gate}};
    \node[green, font=\footnotesize, anchor=south] at (2.4,0.55) {\texttt{value near 1}};
    \node[cell, draw=green, text=green] (n1) at (4.8,0) {\texttt{mem}};
    \draw[->, green, thick] (m1) -- (g1);
    \draw[->, green, very thick] (g1) -- (n1) node[midway, above, font=\footnotesize, text=green] {\texttt{pass}};
    \node[green, anchor=north, font=\footnotesize] at (2.4,-0.7) {\texttt{memory preserved}};
  \end{scope}
\end{tikzpicture}
$$

The gate values are not hand-set; each is the output of a learned linear map of
$[h_{t-1}, x_t]$, so the network discovers _from data_ when a coordinate of the
cell should be held, overwritten, or read out.[^chollet-lstm]

## The GRU cell

The **gated recurrent unit** (GRU) is a streamlined gating with two gates instead
of three and a _single_ state vector: it drops the separate cell state, letting
$h_t$ serve as both memory and output.[^gf-gru] An **update gate** $z_t$ interpolates
between carrying the old state and writing a candidate; a **reset gate** $r_t$
controls how much past state enters that candidate:

$$
z_t = \sigma\!\parens{W_z [h_{t-1}, x_t] + b_z}
\qquad\text{(update gate)}
$$

$$
r_t = \sigma\!\parens{W_r [h_{t-1}, x_t] + b_r}
\qquad\text{(reset gate)}
$$

$$
\tilde h_t = \tanh\!\parens{W_h [\,r_t \odot h_{t-1},\ x_t\,] + b_h}
\qquad\text{(candidate state)}
$$

$$
h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde h_t
\qquad\text{(state update)}
$$

The convex combination in the last line is the GRU's highway: when $z_t \approx 0$
the state is copied forward, $h_t \approx h_{t-1}$, giving the same near-identity
carry as the LSTM cell. A single gate $z_t$ does the job the LSTM splits between
forget and input — here keep and write are tied, $f \equiv 1 - i$.

The dimensions mirror the LSTM with one block fewer. With $x_t \in \mathbb{R}^{n}$
and $h_t \in \mathbb{R}^{d}$, the three weight blocks each map the concatenation
$[h_{t-1}, x_t] \in \mathbb{R}^{d+n}$ into $\mathbb{R}^{d}$:

$$
W_z, W_r, W_h \in \mathbb{R}^{d \times (d+n)},
\qquad
b_z, b_r, b_h \in \mathbb{R}^{d}.
$$

The candidate reads $[r_t \odot h_{t-1},\ x_t]$ rather than $[h_{t-1}, x_t]$ — the
reset gate masks the past before it enters — but the shape is unchanged, still
$\mathbb{R}^{d+n}$. Three blocks instead of four gives

$$
3\,\big[\,d(d+n) + d\,\big] = 3d(d + n + 1)
$$

parameters, exactly three-quarters of the LSTM's $4d(d+n+1)$. For the same
$n = 100$, $d = 256$ that is $3 \cdot 256 \cdot 357 = 274{,}176$ weights against
the LSTM's $365{,}568$.

$$
% caption: The GRU cell. The update gate mixes the carried state with a candidate; the
% reset gate gates how much past state feeds it. One vector is memory and output.
\begin{tikzpicture}[>=stealth, font=\footnotesize, x=1cm, y=1cm,
  gate/.style={draw, thick, minimum width=14mm, minimum height=9mm, align=center, font=\scriptsize},
  op/.style={circle, draw, black, thick, minimum size=8mm, inner sep=0pt, font=\small}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % --- carried-state highway across the top ---
  \node[green] (hin)  at (-0.5,3.3) {\texttt{h(t-1)}};
  \node[op] (mix) at (9.0,3.3) {$+$};
  \node[green] (hout) at (10.7,3.3) {$h_t$};
  \draw[->, green, very thick] (hin) -- (mix);
  \draw[->, green, very thick] (mix) -- (hout);
  \node[green, anchor=south, font=\footnotesize] at (3.8,3.5) {\texttt{carried state}};
  % --- gate row ---
  \node[gate, draw=acc, text=acc]  (rg) at (2.2,1.4) {\texttt{reset}};
  \node[gate, draw=black, text=black] (cg) at (5.0,1.4) {\texttt{cand-}\\\texttt{idate}};
  \node[gate, draw=acc, text=acc]  (zg) at (7.7,1.4) {\texttt{update}};
  % --- input bus along the bottom with vertical risers to each gate ---
  \node[green] (hb)  at (1.4,-1.2) {\texttt{h(t-1)}};
  \node[black] (xin) at (3.2,-1.2) {\texttt{x(t)}};
  \draw[green, thick] (hb)  |- (2.2,-0.45);
  \draw[black, thick] (xin) |- (2.2,-0.45);
  \draw[black, thick] (2.2,-0.45) -- (7.7,-0.45);          % the [h_prev, x_t] bus
  \draw[->, black, thick] (2.2,-0.45) -- (rg.south);
  \draw[->, black, thick] (5.0,-0.45) -- (cg.south);
  \draw[->, black, thick] (7.7,-0.45) -- (zg.south);
  % --- reset modulates the candidate ---
  \draw[->, acc, thick] (rg) -- (cg);
  % --- candidate and update feed the interpolation (orthogonal routes) ---
  \draw[->, black, thick] (cg.north) -- (5.0,2.5) -| (mix.south west);
  \draw[->, acc, thick]   (zg.north) -- (7.7,2.1) -| (mix.south);
\end{tikzpicture}
$$

The update-gate interpolation reads like the LSTM's cell update with the two gates
fused. Take $d = 2$ again, $h_{t-1} = (0.4,\ -0.2)$, and

$$
z_t = (0.2,\ 0.9),
\qquad
\tilde h_t = (0.8,\ 0.5).
$$

Then

$$
h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde h_t
= (0.8 \cdot 0.4,\ \ 0.1 \cdot (-0.2)) + (0.2 \cdot 0.8,\ \ 0.9 \cdot 0.5)
= (0.48,\ 0.43).
$$

The first coordinate mostly carried its old value ($z = 0.2$, so $1 - z = 0.8$ on
the past); the second mostly rewrote to the candidate ($z = 0.9$). The single gate
$z_t$ must trade keeping against writing — the LSTM, with separate
$f_t$ and $i_t$, can keep _and_ write the same coordinate, or do neither.

GRU and LSTM share the additive-highway principle; they differ in bookkeeping. The
GRU has fewer parameters (three weight blocks rather than four) and one state to
carry, which makes it lighter and often faster to train; the LSTM's separate cell
and explicit output gate give it a finer-grained, sometimes stronger memory.

$$
% caption: LSTM vs GRU. The LSTM keeps a separate cell state $c_t$ and gates the read-out with $o_t$; the GRU folds memory and output into one $h_t$ and ties keep-vs-write to a single update gate $z_t$. Both carry an additive highway (thick green).
\begin{tikzpicture}[>=stealth, font=\footnotesize, x=1cm, y=1cm,
  box/.style={draw, thick, minimum width=11mm, minimum height=8mm, align=center, font=\scriptsize},
  op/.style={circle, draw, black, thick, minimum size=6mm, inner sep=0pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % ===== LSTM (left) =====
  \node[acc, font=\footnotesize\bfseries] at (2.1,3.7) {\texttt{LSTM}};
  % cell highway
  \node[green] (lc0) at (0,2.6) {\texttt{c(t-1)}};
  \node[op] (lca) at (2.1,2.6) {$+$};
  \node[green] (lc1) at (4.2,2.6) {\texttt{c(t)}};
  \draw[->, green, very thick] (lc0) -- (lca);
  \draw[->, green, very thick] (lca) -- (lc1);
  % gates
  \node[box, draw=acc, text=acc] (lf) at (1.0,0.8) {\texttt{f,i}};
  \node[box, draw=black]         (lg) at (2.6,0.8) {\texttt{cand}};
  \node[box, draw=acc, text=acc] (lo) at (4.2,0.8) {\texttt{o}};
  \draw[->, acc, thick] (lf) -- (lca);
  \draw[->, black, thick] (lg) -- (lca);
  % read-out
  \node[green] (lh) at (4.2,-0.8) {$h_t$};
  \draw[->, green, thick] (lc1) -- (4.2,1.2);
  \draw[->, acc, thick] (lo) -- (lh);
  \node[black, anchor=north, font=\footnotesize] at (2.1,-0.5) {\texttt{cell + gated output}};
  % divider
  \draw[black, thick] (5.6,-1.0) -- (5.6,3.9);
  % ===== GRU (right) =====
  \node[acc, font=\footnotesize\bfseries] at (8.6,3.7) {\texttt{GRU}};
  \node[green] (gh0) at (6.3,2.6) {\texttt{h(t-1)}};
  \node[op] (gha) at (8.7,2.6) {$+$};
  \node[green] (gh1) at (11.0,2.6) {$h_t$};
  \draw[->, green, very thick] (gh0) -- (gha);
  \draw[->, green, very thick] (gha) -- (gh1);
  \node[box, draw=acc, text=acc] (gr) at (7.0,0.8) {\texttt{r}};
  \node[box, draw=black]         (gg) at (8.7,0.8) {\texttt{cand}};
  \node[box, draw=acc, text=acc] (gz) at (10.4,0.8) {\texttt{z}};
  \draw[->, acc, thick] (gr) -- (gg);
  \draw[->, black, thick] (gg) -- (gha);
  \draw[->, acc, thick] (gz) -- (gha);
  \node[black, anchor=north, font=\footnotesize] at (8.7,-0.5) {\texttt{one state, tied gates}};
\end{tikzpicture}
$$

| | LSTM | GRU |
| --- | --- | --- |
| gates | forget, input, output (3) | update, reset (2) |
| state | cell $c_t$ + hidden $h_t$ | single $h_t$ |
| highway | $c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$ | $h_t = (1-z_t)\odot h_{t-1} + z_t \odot \tilde h_t$ |
| weight blocks | $W_f, W_i, W_c, W_o$ (4) | $W_z, W_r, W_h$ (3) |
| params (state dim $d$, input dim $n$) | $4d(d+n) + 4d$ | $3d(d+n) + 3d$ |
| output exposure | gated read-out $o_t \odot \tanh(c_t)$ | full state $h_t$ |
| when to use | long, complex dependencies; capacity to spare | smaller data / compute; speed matters |

## Persistence across time

Unrolled, both architectures are the same cell applied at every step, with the
memory carried along the top from start to finish. The forget/update gate is what
lets a coordinate survive intact across the whole span while the input rewrites
others — the network holds some facts and revises the rest.

$$
% caption: An LSTM unrolled over four steps. The cell state runs along the top, carried near-unchanged when forget gates are open; each step also emits a hidden state.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=12mm, minimum height=10mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \foreach \i/\x in {1/0, 2/2.6, 3/5.2, 4/7.8} {
    \node[cell, draw=acc, text=acc] (L\i) at (\x,0) {\texttt{LSTM}};
    \node (x\i) at (\x,-1.6) {$x_\i$};
    \node (h\i) at (\x,1.6)  {$h_\i$};
    \draw[->, black, thick] (x\i) -- (L\i);
    \draw[->, acc, thick] (L\i) -- (h\i);
  }
  % cell-state highway along the top of the cells
  \node (c0) at (-1.6,0.0) {$c_0$};
  \draw[->, green, very thick] (c0) -- (L1);
  \foreach \i/\j in {1/2, 2/3, 3/4}
    \draw[->, green, very thick] (L\i) -- (L\j) node[midway, above, font=\scriptsize, text=green] {$c_\i$};
  \node (cT) at (9.4,0.0) {$c_4$};
  \draw[->, green, very thick] (L4) -- (cT);
  \node[green, anchor=north, font=\footnotesize] at (3.9,-0.7) {\texttt{persistent cell state}};
\end{tikzpicture}
$$

This is the architectural payoff. The plain RNN's hidden state was the _only_
channel, and it decayed; the gated cell adds a second channel whose default
behavior is to remember, and whose forgetting is a learned operation rather
than an unavoidable side effect of repeated matrix multiplication. Gated RNNs carried
most sequence modeling for years, until the parallelism and longer
effective context of
[attention and transformers](/deep-learning/architectures/attention-and-transformers)
displaced recurrence for the largest models.

## Which gates earn their keep

The LSTM in Goodfellow's Chapter 10 is the 1997 Hochreiter–Schmidhuber cell plus
the forget gate (Gers et al., 2000). A few results since then sharpen which parts
of it actually matter.

- **Which gates matter.** Greff et al., "LSTM: A Search Space Odyssey"
  (IEEE TNNLS, 2017), ablated every component across thousands of runs. The finding
  is blunt: the **forget gate** and the **output activation** are essential — removing
  either collapses performance — while peephole connections and the input-gate/forget-
  gate coupling have little effect. This is the empirical justification for the
  GRU's design: it keeps the forget-like update gate and drops nearly everything the
  study found expendable.
- **The highway in numbers.** With the forget gate held at $f_t = 0.98$, the
  cell-state Jacobian across $100$ steps is $\prod_t f_t = 0.98^{100} \approx 0.13$ —
  the gradient shrinks by less than an order of magnitude. Compare the plain RNN from
  the [recurrent-networks lesson](/deep-learning/architectures/recurrent-networks),
  where $\rho = 0.9$ gave $0.9^{100} \approx 2.7\times10^{-5}$. The gate's job is to
  keep that per-step multiplier as close to $1$ as the task allows; at $f_t = 1$ the
  product is exactly $1$ and memory is perfect.
- **Normalization inside the cell.** Batch normalization does not fit a recurrence
  (statistics differ per step), but **layer normalization** applied to the gate
  pre-activations (Ba et al., 2016) stabilizes LSTM training and became standard for
  deep recurrent stacks — the same per-token normalization that later anchors the
  Transformer block.

The modern verdict is mixed rather than a clean replacement. The
[Transformer](/deep-learning/architectures/the-transformer-architecture) won the
largest models on parallelism, but gated recurrence never fully left: it remains the
default for strict streaming and low-latency inference (constant memory per step, no
growing context), and recent work such as the "xLSTM" (Beck et al., 2024) revisits
the cell with matrix memory and parallel training, so the additive-highway idea is
still live.

## Takeaways

- The plain RNN propagates state through a **repeated weight-matrix multiply**;
  the backward Jacobian is a product $\prod_k D_k W \sim \rho(W)^{T-t}$ that
  **vanishes** ($\rho < 1$) or **explodes** ($\rho > 1$), killing long-range
  gradients.
- Gating replaces the multiplicative carry with an **additive** one. The LSTM cell
  update $c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$ has Jacobian
  $\partial c_t/\partial c_{t-1} = \diag(f_t)$ — **no weight matrix**,
  so with $f_t \approx 1$ the gradient rides a near-identity highway.
- The **LSTM** has three gates (forget, input, output) and two state vectors
  (cell $c_t$, hidden $h_t$); the **GRU** merges these into two gates (update,
  reset) and one state, with fewer parameters and a tied keep/write.
- Both are the same cell unrolled across time; the **forget/update gate** is the
  learned switch that lets a memory persist intact or be overwritten, step by step.

[^gf-longterm]: **Goodfellow**, _Deep Learning_, §10.7 — The Challenge of Long-Term Dependencies: the repeated weight multiply whose Jacobian product vanishes or explodes, the failure gating is built to fix.
[^gf-lstm]: **Goodfellow**, _Deep Learning_, §10.10 — The Long Short-Term Memory and Other Gated RNNs: the cell state and the three sigmoid gates (forget, input, output) of the LSTM.
[^gf-carousel]: **Goodfellow**, _Deep Learning_, §10.10.1 — LSTM: the self-loop on the cell state (the "constant error carousel") whose forget-gated Jacobian $\diag(f_t)$ stays near the identity.
[^gf-gru]: **Goodfellow**, _Deep Learning_, §10.10.2 — Other Gated RNNs: the GRU's single update gate that merges the LSTM's forget and input gates onto one state vector.
[^chollet-lstm]: **Chollet**, _Deep Learning with Python_, §6.2 — Understanding Recurrent Neural Networks: a practitioner's reading of LSTM/GRU gates as learned, data-driven carry/write/read switches in Keras.