---
title: Recurrent Networks
module: Architectures
moduleNumber: 5
lessonNumber: 3
order: 503
summary: >
  A recurrent network folds a sequence into a fixed-size hidden state, reusing one
  set of weights at every time step, the architectural prior that the same rule
  applies wherever it lands in time. Unrolling the recurrence exposes a deep
  feed-forward graph; backpropagation through it sums gradient contributions across
  all steps and chains a product of Jacobians, and that product is why long-range
  gradients vanish or explode. That failure motivates gated architectures.
topics: [Architectures]
sources:
  - book: Goodfellow
    ref: "Ch. 10 — Sequence Modeling: Recurrent and Recursive Nets"
  - book: Goodfellow
    ref: "§10.2 Recurrent Neural Networks; §10.2.2 Computing the Gradient (BPTT)"
  - book: Goodfellow
    ref: "§10.7 The Challenge of Long-Term Dependencies"
---

A [convolutional network](/deep-learning/architectures/convolutional-networks) bakes
in the prior that the same feature detector should slide across _space_. A **recurrent
network** bakes in the dual prior for _time_: the same transition rule should apply at
every position in a sequence. It processes $x_1, x_2, \dots, x_T$ one element at a time,
carrying a fixed-size **hidden state** $h_t$ that summarizes everything seen so far,
a learned, lossy memory of the past.[^gf-rnn]

## The recurrence

A vanilla RNN is one nonlinear map applied repeatedly. At step $t$ it folds the new
input $x_t$ into the running state $h_{t-1}$ and emits an output $y_t$:

$$
h_t = g\!\parens{W_{hh}\,h_{t-1} + W_{xh}\,x_t + b},
\qquad
y_t = W_{hy}\,h_t,
$$

with $x_t \in \mathbb{R}^{d}$, $h_t \in \mathbb{R}^{m}$, the activation $g$ usually
$\tanh$, and the state seeded at $h_0 = 0$. Three weight matrices carry the whole model:

| matrix | shape | role |
| --- | --- | --- |
| $W_{xh}$ | $m \times d$ | maps the current input into the state |
| $W_{hh}$ | $m \times m$ | propagates the previous state forward |
| $W_{hy}$ | $k \times m$ | reads an output off the state |

The defining structural fact is in the subscripts: $W_{hh}, W_{xh}, W_{hy}$ carry **no
time index**. The same three matrices act at $t = 1$ and at $t = T$.

To make the sizing concrete, chase the shapes through one step. Take a concrete
$d = 3$ input, an $m = 4$ state, and a $k = 2$ output. Then $W_{xh}$ is $4 \times 3$,
$W_{hh}$ is $4 \times 4$, $b$ is $4 \times 1$, and $W_{hy}$ is $2 \times 4$. The two
matrix-vector products both land in $\mathbb{R}^{4}$, add elementwise, pass through
$\tanh$ componentwise, and read out through $W_{hy}$:

$$
\underbrace{W_{xh}}_{4\times 3}\underbrace{x_t}_{3\times 1}
+ \underbrace{W_{hh}}_{4\times 4}\underbrace{h_{t-1}}_{4\times 1}
+ \underbrace{b}_{4\times 1}
\;=\; \underbrace{a_t}_{4\times 1},
\qquad
h_t = \tanh(a_t) \in \mathbb{R}^{4},
\qquad
y_t = \underbrace{W_{hy}}_{2\times 4}\,h_t \in \mathbb{R}^{2}.
$$

The parameter count is $md + m^2 + m + km = 4\cdot 3 + 16 + 4 + 2\cdot 4 = 40$, and it
stays $40$ whether the sequence is ten steps or ten thousand. The figure below tracks
those shapes through one step; the same block is what the unrolled graph copies.

$$
% caption: One recurrence step as a shape flow ($d{=}3$ input, $m{=}4$ state, $k{=}2$ output). The two matrix-vector products land in $\mathbb{R}^{4}$, sum with the bias, pass through $\tanh$, and read out through $W_{hy}$ to $\mathbb{R}^{2}$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  vec/.style={draw, thick, minimum height=7mm, align=center},
  op/.style={draw, thick, circle, inner sep=0.6pt, minimum size=5mm},
  mat/.style={draw=acc, text=acc, thick, minimum height=7mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  % inputs on the left
  \node[vec] (x)  at (0,0)    {$x_t$\\\texttt{[3]}};
  \node[vec] (hp) at (0,-1.9) {$h$\texttt{(t-1)}\\\texttt{[4]}};
  % weight matrices
  \node[mat] (wx) at (2.6,0)    {$W_{xh}$\\\texttt{[4x3]}};
  \node[mat] (wh) at (2.6,-1.9) {$W_{hh}$\\\texttt{[4x4]}};
  \draw[->, thick] (x)  -- (wx);
  \draw[->, thick] (hp) -- (wh);
  % sum node
  \node[op] (sum) at (5.2,-0.95) {$+$};
  \node[vec, minimum width=6mm] (b) at (5.2,-2.7) {$b$\\\texttt{[4]}};
  \draw[->, thick] (wx) -- (sum);
  \draw[->, thick] (wh) -- (sum);
  \draw[->, thick] (b)  -- (sum);
  % pre-activation and tanh
  \node[vec] (a) at (7.2,-0.95) {$a_t$\\\texttt{[4]}};
  \node[op]  (t) at (8.9,-0.95) {\scriptsize$\tanh$};
  \node[vec, draw=acc, text=acc] (h) at (10.7,-0.95) {$h_t$\\\texttt{[4]}};
  \draw[->, thick] (sum) -- (a);
  \draw[->, thick] (a) -- (t);
  \draw[->, thick] (t) -- (h);
  % readout
  \node[mat] (wy) at (10.7,1.4) {$W_{hy}$\\\texttt{[2x4]}};
  \node[vec] (y)  at (12.6,1.4) {$y_t$\\\texttt{[2]}};
  \draw[->, acc, thick] (h) -- (wy);
  \draw[->, thick] (wy) -- (y);
\end{tikzpicture}
$$

> **Definition (Parameter sharing across time).** A recurrent network applies one fixed
> parameter set $\{W_{hh}, W_{xh}, W_{hy}, b\}$ at every time step. The model size is
> independent of the sequence length $T$, and a pattern learned at one position
> generalizes to the same pattern at any other position.

This is the temporal analogue of a convolution's shared
[filter](/deep-learning/architectures/convolutional-networks): it lets one network
ingest sequences of _any_ length, and it is a strong regularizer: the count of free
parameters does not grow with $T$.

> **Definition (Hidden state as memory).** The vector $h_t$ is the only channel through
> which information at step $t' < t$ can influence step $t$. The recurrence
> $h_t = g(W_{hh} h_{t-1} + W_{xh} x_t + b)$ forces the network to compress the entire
> prefix $x_1, \dots, x_t$ into the fixed-dimension $h_t$: a lossy summary, not a
> verbatim transcript.

## Unrolling through time

The self-loop is a compact fiction. To compute anything we **unroll** the recurrence:
copy the cell once per time step and feed each copy's state into the next. The result
is an ordinary feed-forward graph, $T$ layers deep, in which every layer shares weights.[^gf-unfold]

$$
% caption: Left: the recurrent cell with its self-loop. Right: the same cell unrolled across time, one copy per step with hidden state $h_t$ threaded along and weights shared.
\begin{tikzpicture}[>=stealth, font=\small,
  cell/.style={draw, thick, minimum width=11mm, minimum height=11mm, align=center},
  io/.style={draw, black, minimum width=8mm, minimum height=7mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{accmid}{HTML}{6A82F6}
  % --- folded cell (left) ---
  \node[cell, draw=acc, text=acc] (h) at (0,0) {$h$};
  \node[io] (x) at (0,-1.9) {$x_t$};
  \node[io] (y) at (0,1.9) {$y_t$};
  \draw[->, thick] (x) -- (h);
  \draw[->, thick] (h) -- (y);
  % self loop
  \draw[->, acc, thick] (h.east) .. controls (1.5,0.9) and (1.5,-0.9) .. (h.east)
    node[right=2mm, midway, font=\footnotesize, text=acc] {$W_{hh}$};
  \node[font=\footnotesize, black] at (0,-2.9) {recurrent cell};
  % --- unrolled (right) ---
  \begin{scope}[xshift=42mm]
    \foreach \i/\lab in {1/1,2/2,3/3} {
      \node[cell, draw=acc, text=acc] (u\i) at (\i*2.2,0) {$h_{\lab}$};
      \node[io] (ux\i) at (\i*2.2,-1.9) {$x_{\lab}$};
      \node[io] (uy\i) at (\i*2.2,1.9) {$y_{\lab}$};
      \draw[->, thick] (ux\i) -- (u\i);
      \draw[->, thick] (u\i) -- (uy\i);
    }
    \node[black, font=\footnotesize] (start) at (0.3,0) {$h_0$};
    \draw[->, acc, thick] (start) -- (u1) node[midway, above, font=\footnotesize, text=acc] {$W_{hh}$};
    \draw[->, acc, thick] (u1) -- (u2) node[midway, above, font=\footnotesize, text=acc] {$W_{hh}$};
    \draw[->, acc, thick] (u2) -- (u3) node[midway, above, font=\footnotesize, text=acc] {$W_{hh}$};
    \draw[->, black, thick] (u3) -- ++(1.4,0) node[right, font=\footnotesize] {$\dots$};
    \node[font=\footnotesize, black] at (4.4,-2.9) {unrolled across time};
  \end{scope}
\end{tikzpicture}
$$

The forward pass is a left-to-right sweep over the unrolled graph: each step reads the
input, updates the state, and emits an output.

```algorithm
caption: $\textsc{RnnForward}(x_{1:T}, W_{hh}, W_{xh}, W_{hy}, b)$ — one left-to-right sweep
$h_0 \gets 0$ // zero-initialize the memory
for $t \gets 1$ to $T$ do
  $a_t \gets W_{hh}\,h_{t-1} + W_{xh}\,x_t + b$ // pre-activation
  $h_t \gets g(a_t)$ // new hidden state
  $y_t \gets W_{hy}\,h_t$ // output at step $t$
return $h_{1:T},\ y_{1:T}$
```

Unrolling reframes the model exactly: a recurrent network is a very deep feed-forward
network whose depth equals the sequence length and whose every layer is tied to the
same weights. Everything about training follows from that one observation.

## Backpropagation through time

Because the unrolled graph is feed-forward, it is trained by ordinary
[backpropagation](/deep-learning/neural-networks/backpropagation); applied to the
unrolled graph the method is called **backpropagation through time** (BPTT).[^gf-bptt] The total
loss sums per-step losses, $L = \sum_{t=1}^{T} L_t$, and we want $\partial L / \partial
W_{hh}$. The subtlety is that $W_{hh}$ is the _same_ matrix at every step, so it
influences the loss through every hidden state; its gradient is a **sum of
contributions across all time steps**:

$$
\frac{\partial L}{\partial W_{hh}}
= \sum_{t=1}^{T} \frac{\partial L_t}{\partial W_{hh}}
= \sum_{t=1}^{T} \sum_{s=1}^{t}
  \frac{\partial L_t}{\partial h_t}\,
  \frac{\partial h_t}{\partial h_s}\,
  \frac{\partial h_s}{\partial W_{hh}}.
$$

The middle factor is where time enters. To get from step $t$ back to an earlier step
$s$, the gradient must pass through every intermediate state, so $\partial h_t /
\partial h_s$ is itself a **product of one-step Jacobians**:

$$
\frac{\partial h_t}{\partial h_s}
= \prod_{r=s+1}^{t} \frac{\partial h_r}{\partial h_{r-1}},
\qquad
\frac{\partial h_r}{\partial h_{r-1}}
= \diag\!\parens{g'(a_r)}\,W_{hh}^{T}.
$$

Each link is the recurrent weight $W_{hh}^{T}$ scaled by the activation derivative
$g'(a_r)$. The gradient flows _backward_ through the unrolled chain, accumulating one
such Jacobian per step.

$$
% caption: BPTT. The loss propagates backward (red) through the chain; each hop is one one-step Jacobian, and their product spans distant steps.
\begin{tikzpicture}[>=stealth, font=\small,
  cell/.style={draw, thick, minimum width=11mm, minimum height=11mm, align=center},
  io/.style={draw, black, minimum width=8mm, minimum height=7mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \foreach \i in {1,2,3,4} {
    \node[cell, draw=acc, text=acc] (h\i) at (\i*2.6,0) {$h_{\i}$};
    \node[io] (x\i) at (\i*2.6,-2.0) {$x_{\i}$};
    \node[io] (L\i) at (\i*2.6,2.4) {$L_{\i}$};
    \draw[->, black, thick] (x\i) -- (h\i);
    \draw[->, black, thick] (h\i) -- (L\i);
  }
  % between each pair of cells: forward state (blue, top track) and the backward
  % gradient (red, bottom track) run as a clean two-lane double-track.
  \foreach \i [evaluate=\i as \j using int(\i+1)] in {1,2,3} {
    \draw[->, acc, thick] ([yshift=2.4mm]h\i.east) -- ([yshift=2.4mm]h\j.west);
    \draw[->, red, very thick] ([yshift=-2.4mm]h\j.west) -- ([yshift=-2.4mm]h\i.east)
      node[midway, below=1pt, font=\footnotesize, text=red] {$J_{\j}$};
  }
  \node[red, font=\footnotesize] at (6.5,3.5) {gradient f\/lows backward in time};
\end{tikzpicture}
$$

The same backward sweep accumulates $\partial L/\partial W_{hh}$, $\partial L/\partial
W_{xh}$, $\partial L/\partial W_{hy}$, and $\partial L/\partial b$ by adding each step's
contribution into a shared accumulator.

```algorithm
caption: $\textsc{Bptt}(x_{1:T}, y_{1:T}^{\star})$ — accumulate shared-weight gradients
run $\textsc{RnnForward}$, caching $a_t, h_t, y_t$
initialize $\,\mathrm{d}W_{hh}, \mathrm{d}W_{xh}, \mathrm{d}W_{hy}, \mathrm{d}b \gets 0$
$\delta_{T+1} \gets 0$ // incoming state gradient
for $t \gets T$ down to $1$ do
  $\mathrm{d}W_{hy} \mathrel{+}= (\partial L_t / \partial y_t)\,h_t^{T}$ // output weights
  $\delta_t \gets W_{hy}^{T}(\partial L_t / \partial y_t) + W_{hh}^{T}\delta_{t+1}$ // sum output + future
  $\delta_t \gets \delta_t \odot g'(a_t)$ // through the activation
  $\mathrm{d}W_{hh} \mathrel{+}= \delta_t\,h_{t-1}^{T}$ // accumulate, shared weights
  $\mathrm{d}W_{xh} \mathrel{+}= \delta_t\,x_t^{T}$
  $\mathrm{d}b \mathrel{+}= \delta_t$
return $\mathrm{d}W_{hh}, \mathrm{d}W_{xh}, \mathrm{d}W_{hy}, \mathrm{d}b$
```

The cost is memory: BPTT must cache every $h_t$ for the backward pass, so memory
grows with $T$. For a sequence of $10{,}000$ tokens the graph is $10{,}000$ layers deep,
and the whole stack of activations has to live in memory until the backward sweep
reaches step $1$. **Truncated BPTT** caps this by chopping the sequence into windows of
$k$ steps: the forward pass still runs the full length, carrying $h_t$ across window
boundaries, but the backward pass only flows within each window and is cut at the seam.
Gradients travel at most $k$ steps back, which bounds memory and compute at the price of
any dependency longer than $k$.

$$
% caption: Truncated BPTT with window $k=3$. The forward state (blue) runs the full length, but each backward pass (red) is confined to its window and stopped at the boundary, so no gradient crosses a seam.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw=acc, text=acc, thick, minimum width=9mm, minimum height=9mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \foreach \i in {1,2,3,4,5,6} {
    \node[cell] (h\i) at (\i*1.9,0) {$h_{\i}$};
  }
  % forward state, full length (blue)
  \foreach \i [evaluate=\i as \j using int(\i+1)] in {1,2,3,4,5}
    \draw[->, acc, thick] (h\i) -- (h\j);
  % backward gradients within windows only (red), stopped at the seam h3|h4
  \foreach \i [evaluate=\i as \j using int(\i+1)] in {1,2}
    \draw[->, red, very thick] ([yshift=-3mm]h\j.south) -- ([yshift=-3mm]h\i.south);
  \foreach \i [evaluate=\i as \j using int(\i+1)] in {4,5}
    \draw[->, red, very thick] ([yshift=-3mm]h\j.south) -- ([yshift=-3mm]h\i.south);
  % seam marker between window 1 (h1..h3) and window 2 (h4..h6)
  \draw[red, thick, dashed] (6.65,-0.9) -- (6.65,0.9);
  \node[red, font=\footnotesize, anchor=south] at (6.65,0.95) {\texttt{truncation seam}};
  \node[acc, font=\footnotesize, anchor=north] at (3.8,-1.35) {\texttt{window 1}};
  \node[acc, font=\footnotesize, anchor=north] at (9.5,-1.35) {\texttt{window 2}};
\end{tikzpicture}
$$

## Vanishing and exploding gradients

The product of Jacobians is also the network's deepest weakness.[^gf-longterm] Hold the
activation roughly linear ($g' \approx 1$) so the one-step Jacobian is essentially
$W_{hh}^{T}$; then the state-to-state gradient across $t - s$ steps is a **matrix
power**:

$$
\frac{\partial h_t}{\partial h_s} \approx \parens{W_{hh}^{T}}^{\,t-s}.
$$

Diagonalize $W_{hh} = Q\,\Lambda\,Q^{-1}$ with eigenvalues $\lambda_i$. A matrix power
raises each eigenvalue to the same power, so the gradient's growth is governed entirely
by the **spectral radius** $\rho(W_{hh}) = \max_i |\lambda_i|$:

$$
\parens{W_{hh}}^{\,t-s} = Q\,\Lambda^{\,t-s}\,Q^{-1},
\qquad
\Lambda^{\,t-s} = \diag\!\parens{\lambda_1^{\,t-s}, \dots, \lambda_m^{\,t-s}}.
$$

The number $\lambda_i^{\,t-s}$ either collapses to $0$ or blows up to $\infty$ as the
gap $t - s$ grows, depending on a single threshold:

> **Theorem (Vanishing / exploding gradients).** For a vanilla RNN the gradient passed
> back $t - s$ steps scales like $\rho(W_{hh})^{\,t-s}$. If $\rho(W_{hh}) < 1$ the
> gradient **vanishes** geometrically — distant steps stop contributing. If
> $\rho(W_{hh}) > 1$ it **explodes**. Only the knife-edge $\rho(W_{hh}) = 1$ preserves
> the signal, and that case is not stable under training.

> **Proof (sketch).** With $g' \approx 1$, $\partial h_t / \partial h_s \approx
> (W_{hh}^{T})^{t-s}$. Spectral norms are submultiplicative and bounded by powers of the
> largest eigenvalue, so $\lVert (W_{hh})^{t-s} \rVert$ grows like $\rho(W_{hh})^{t-s}$.
> For $\rho < 1$ this $\to 0$; for $\rho > 1$ this $\to \infty$, both exponentially in
> $t - s$. The bounded activation $\tanh$ has $g' \le 1$, which only tightens the
> vanishing case. $\qed$

$$
% caption: Vanishing gradient through time. With $\rho(W_{hh})<1$ the gradient decays geometrically as the gap grows, so distant steps receive almost no learning signal.
\begin{tikzpicture}[>=stealth, font=\small]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (0,0) -- (8.6,0) node[right, font=\footnotesize] {steps back in time};
  \draw[->, thick] (0,0) -- (0,3.6) node[above, font=\footnotesize] {gradient magnitude};
  % decaying bars: rho^k with rho=0.6, scaled
  \foreach \i/\h in {1/3.2, 2/2.05, 3/1.31, 4/0.84, 5/0.54, 6/0.34, 7/0.22, 8/0.14} {
    \draw[acc, thick, fill=acc!15] (\i*0.95-0.28,0) rectangle (\i*0.95+0.28,\h);
  }
  % decay envelope
  \draw[red, thick, dashed] (0.67,3.2) .. controls (3,1.0) and (5,0.35) .. (7.9,0.14);
  \node[red, font=\footnotesize, anchor=west] at (4.4,1.45) {decay $= r^{k},\ r < 1$};
  \node[black, font=\footnotesize] at (1.0,-0.45) {recent};
  \node[black, font=\footnotesize] at (7.6,-0.45) {distant};
\end{tikzpicture}
$$

For example, suppose the largest eigenvalue is
$\rho = 0.9$. Over a $50$-step gap the gradient is scaled by $0.9^{50} \approx 0.005$,
and over $100$ steps by $0.9^{100} \approx 2.7 \times 10^{-5}$: the signal from a hundred
steps back arrives four to five orders of magnitude weaker than a signal from one step
back, drowned out by nearer terms and by numerical noise. Flip the eigenvalue to
$\rho = 1.1$ and the same $100$ steps multiply the gradient by $1.1^{100} \approx
1.4 \times 10^{4}$, and the loss overflows to NaN in a few updates. The stable band
around $\rho = 1$ is vanishingly thin, and nothing in ordinary gradient descent pins
$W_{hh}$ to it.

$$
% caption: The spectral radius sets the fate of the gradient. $\rho<1$ decays geometrically (blue), $\rho>1$ blows up (red), and only the knife-edge $\rho=1$ (dashed) is flat — and unstable under training.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, thick] (0,0) -- (7.6,0) node[right, font=\footnotesize] {\texttt{gap}};
  \draw[->, thick] (0,0) -- (0,4.0) node[above, font=\footnotesize] {\texttt{gradient scale (log)}};
  % knife-edge rho=1: flat line at mid height
  \draw[black, thick, dashed] (0,2.0) -- (6.6,2.0)
    node[right, font=\footnotesize, black] {\texttt{rho=1}};
  % exploding rho>1: rising curve
  \draw[red, very thick] (0,2.0) .. controls (2.5,2.4) and (4.0,3.0) .. (6.2,3.8)
    node[right, font=\footnotesize, text=red] {\texttt{rho}$>$\texttt{1}};
  % vanishing rho<1: falling curve
  \draw[acc, very thick] (0,2.0) .. controls (2.5,1.6) and (4.0,1.0) .. (6.2,0.2)
    node[right, font=\footnotesize, text=acc] {\texttt{rho}$<$\texttt{1}};
  \node[font=\footnotesize, red, anchor=west] at (2.9,3.15) {\texttt{explodes}};
  \node[font=\footnotesize, acc, anchor=west] at (2.9,0.85) {\texttt{vanishes}};
\end{tikzpicture}
$$

The two failures need different fixes, and only one is easy.

| pathology | condition | symptom | fix |
| --- | --- | --- | --- |
| exploding | $\rho(W_{hh}) > 1$ | gradient norm spikes, loss diverges (NaN) | **gradient clipping** |
| vanishing | $\rho(W_{hh}) < 1$ | distant steps get no signal, no long-range memory | **gating** (LSTM / GRU) |

**Gradient clipping** rescales the gradient whenever its norm exceeds a threshold
$\tau$, capping the step size without changing its direction:

$$
g \;\gets\; g \cdot \min\!\parens{1,\ \frac{\tau}{\lVert g\rVert}}.
$$

Clipping controls _explosion_ but does nothing for _vanishing_ — a
signal that has already decayed to numerical zero cannot be amplified. The real fix is
to change the
recurrence so the gradient has a near-identity path to travel along, which is precisely
what gated cells provide.

> **Remark (Why gating).** [LSTM and GRU](/deep-learning/architectures/lstm-and-gru)
> cells replace the multiplicative recurrence $h_t = g(W_{hh} h_{t-1} + \cdots)$ with an
> **additive** cell-state update guarded by learned gates. The additive path keeps the
> state-to-state Jacobian near the identity, so the product over many steps neither
> vanishes nor explodes — long-range dependencies become learnable.

## Sequence-task shapes

The same recurrent cell serves very different tasks depending on _which_ inputs and
outputs are present. Counting elements on each side gives a small taxonomy.

| shape | inputs $\to$ outputs | reads | emits | example |
| --- | --- | --- | --- | --- |
| one-to-one | $1 \to 1$ | single | single | plain classification (degenerate RNN) |
| one-to-many | $1 \to T$ | single | sequence | image captioning |
| many-to-one | $T \to 1$ | sequence | single | sentiment classification |
| many-to-many (aligned) | $T \to T$ | sequence | sequence, step-aligned | part-of-speech tagging |
| many-to-many (seq2seq) | $T \to T'$ | sequence | sequence, different length | machine translation |

$$
% caption: Four sequence-task shapes (black inputs, blue outputs), spanning aligned
% many-to-many and the encoder-decoder seq2seq case that reads the input fully first.
\begin{tikzpicture}[>=stealth, font=\footnotesize, node distance=0pt,
  cell/.style={draw, black, thick, minimum width=8mm, minimum height=8mm},
  src/.style={draw, black, thick, minimum width=8mm, minimum height=8mm},
  dst/.style={draw=acc, text=acc, thick, minimum width=8mm, minimum height=8mm},
  ar/.style={->, black, thick}]
  \definecolor{acc}{HTML}{2348F2}
  \def\g{1.5}   % cell gap
  \def\vy{1.9}  % vertical gap
  % one-to-many
  \begin{scope}[xshift=0cm]
    \node[dst] (a1) at (0,\vy) {}; \node[dst] (a2) at (\g,\vy) {}; \node[dst] (a3) at (2*\g,\vy) {};
    \node[cell] (ac1) at (0,0) {}; \node[cell] (ac2) at (\g,0) {}; \node[cell] (ac3) at (2*\g,0) {};
    \node[src] (ai) at (0,-\vy) {};
    \draw[ar] (ai) -- (ac1); \draw[ar] (ac1) -- (ac2); \draw[ar] (ac2) -- (ac3);
    \draw[ar] (ac1) -- (a1); \draw[ar] (ac2) -- (a2); \draw[ar] (ac3) -- (a3);
    \node[font=\footnotesize] at (\g,-\vy-0.9) {\texttt{one-to-many}};
  \end{scope}
  % many-to-one
  \begin{scope}[xshift=5.2cm]
    \node[dst] (bo) at (2*\g,\vy) {};
    \node[cell] (bc1) at (0,0) {}; \node[cell] (bc2) at (\g,0) {}; \node[cell] (bc3) at (2*\g,0) {};
    \node[src] (bi1) at (0,-\vy) {}; \node[src] (bi2) at (\g,-\vy) {}; \node[src] (bi3) at (2*\g,-\vy) {};
    \draw[ar] (bi1) -- (bc1); \draw[ar] (bi2) -- (bc2); \draw[ar] (bi3) -- (bc3);
    \draw[ar] (bc1) -- (bc2); \draw[ar] (bc2) -- (bc3);
    \draw[ar] (bc3) -- (bo);
    \node[font=\footnotesize] at (\g,-\vy-0.9) {\texttt{many-to-one}};
  \end{scope}
  % many-to-many aligned
  \begin{scope}[xshift=10.4cm]
    \node[dst] (co1) at (0,\vy) {}; \node[dst] (co2) at (\g,\vy) {}; \node[dst] (co3) at (2*\g,\vy) {};
    \node[cell] (cc1) at (0,0) {}; \node[cell] (cc2) at (\g,0) {}; \node[cell] (cc3) at (2*\g,0) {};
    \node[src] (ci1) at (0,-\vy) {}; \node[src] (ci2) at (\g,-\vy) {}; \node[src] (ci3) at (2*\g,-\vy) {};
    \draw[ar] (ci1) -- (cc1); \draw[ar] (ci2) -- (cc2); \draw[ar] (ci3) -- (cc3);
    \draw[ar] (cc1) -- (cc2); \draw[ar] (cc2) -- (cc3);
    \draw[ar] (cc1) -- (co1); \draw[ar] (cc2) -- (co2); \draw[ar] (cc3) -- (co3);
    \node[font=\footnotesize, align=center] at (\g,-\vy-1.1) {\texttt{many-to-many}\\(aligned)};
  \end{scope}
  % seq2seq
  \begin{scope}[xshift=15.6cm]
    \node[dst] (do1) at (2*\g,\vy) {}; \node[dst] (do2) at (3*\g,\vy) {};
    \node[cell] (dc1) at (0,0) {}; \node[cell] (dc2) at (\g,0) {};
    \node[dst] (dc3) at (2*\g,0) {}; \node[dst] (dc4) at (3*\g,0) {};
    \node[src] (di1) at (0,-\vy) {}; \node[src] (di2) at (\g,-\vy) {};
    \draw[ar] (di1) -- (dc1); \draw[ar] (di2) -- (dc2);
    \draw[ar] (dc1) -- (dc2); \draw[ar] (dc2) -- (dc3); \draw[->, acc, thick] (dc3) -- (dc4);
    \draw[->, acc, thick] (dc3) -- (do1); \draw[->, acc, thick] (dc4) -- (do2);
    \node[font=\footnotesize, align=center] at (1.5*\g,-\vy-1.1) {\texttt{many-to-many}\\(seq2seq)};
  \end{scope}
\end{tikzpicture}
$$

The distinction is purely in the wiring, not the cell. **One-to-many** feeds a single
input at the first step and then generates by looping its own output back in as the next
input — an image-captioning model conditions on the image once, then emits words until a
stop token. **Many-to-one** ignores every intermediate output and reads a single
prediction off the final state $h_T$, the only vector that has seen the whole sequence;
a sentiment classifier does exactly this. **Aligned many-to-many** emits one output per
input in lockstep, the natural shape for per-token labeling like part-of-speech tagging.

The **seq2seq** case is special: an _encoder_ RNN reads the whole input into a final
state, which seeds a _decoder_ RNN that generates the output. The input and output
lengths need not match — a five-word English sentence can become a seven-word French one
— because the decoder runs its own clock, generating until it emits a stop token. The
bottleneck of forcing all meaning through one fixed vector $h_T^{\text{enc}}$ is the strain
[attention](/deep-learning/architectures/attention-and-transformers) was invented to
relieve.

## Bidirectional RNNs

A plain RNN at step $t$ has seen only $x_1, \dots, x_t$. For aligned tasks where the
whole sequence is available up front — tagging a word given its full sentence — the
_future_ is just as informative as the past. A **bidirectional RNN** runs two
independent recurrences, one forward and one backward, and concatenates their states.[^gf-bidir]

$$
% caption: A bidirectional RNN. A forward chain (blue) and backward chain (red) are
% concatenated at each step, so the output $y_t$ depends on the entire sequence.
\begin{tikzpicture}[>=stealth, font=\small,
  cell/.style={draw, thick, minimum width=10mm, minimum height=9mm, align=center, font=\footnotesize},
  io/.style={draw, black, thick, minimum width=8mm, minimum height=7mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \foreach \i in {1,2,3,4} {
    \node[io]   (x\i) at (\i*2.8,-3.0) {$x_{\i}$};
    \node[cell, draw=acc, text=acc] (f\i) at (\i*2.8,-1.0) {$f_{\i}$};
    \node[cell, draw=red, text=red] (b\i) at (\i*2.8, 1.0) {$b_{\i}$};
    \node[io]   (y\i) at (\i*2.8, 3.0) {$y_{\i}$};
    % x feeds the forward cell straight up; the path to the backward cell bows
    % clearly to the right of f so it never crosses the box.
    \draw[->, black, thick] (x\i) -- (f\i);
    \draw[->, black, thick] (x\i) to[out=40, in=-70, looseness=1.15] (b\i.south east);
    % forward state bows left of b into the output; backward state goes straight up.
    \draw[->, acc, thick] (f\i) to[out=140, in=-110, looseness=1.15] (y\i.south west);
    \draw[->, red, thick] (b\i) -- (y\i);
  }
  % forward chain (blue, left to right)
  \foreach \i [evaluate=\i as \j using int(\i+1)] in {1,2,3}
    \draw[->, acc, thick] (f\i) -- (f\j);
  % backward chain (red, right to left)
  \foreach \i [evaluate=\i as \j using int(\i+1)] in {1,2,3}
    \draw[->, red, thick] (b\j) -- (b\i);
  \node[acc, font=\footnotesize, anchor=east] at (2.0,-1.0) {forward};
  \node[red, font=\footnotesize, anchor=east] at (2.0, 1.0) {backward};
\end{tikzpicture}
$$

The output stacks both directions, $y_t = W_{hy}\,[\,h_t^{\rightarrow};\, h_t^{\leftarrow}\,]$,
so every prediction is conditioned on the full context. Bidirectionality fits tagging
and classification, where the entire input is known in advance; it is unavailable for
streaming or generation, where the future has not happened yet.

## Around and after the plain RNN

Three practical mechanisms sit around the plain recurrence in Goodfellow's
Chapter 10 but are easy to miss on a first read, and one later development is the
bridge to everything after RNNs.

- **Truncated BPTT.** The unrolled graph is $T$ layers deep, and full BPTT stores
  every hidden state for the backward pass — $O(T)$ memory, impossible for a book-
  length sequence. **Truncated BPTT** processes the stream in windows of length $k$
  (say $k = 35$ tokens): it carries the hidden state _forward_ across window
  boundaries but only propagates the gradient _back_ $k$ steps before detaching.
  Concretely, a $10{,}000$-token document with $k = 35$ needs the
  memory of a $35$-step graph, not a $10{,}000$-step one — a $\approx 285\times$
  saving — at the price that no dependency longer than $35$ steps gets a gradient.
  This makes the vanishing-gradient limit a design parameter rather than an accident.
- **Teacher forcing and exposure bias.** During training a generative RNN is fed the
  _true_ previous token $x_{t-1}$ rather than its own prediction, which decouples the
  steps and lets the whole sequence train in parallel (teacher forcing). At inference
  it must consume its own outputs, so a single early mistake shifts the model into
  states it never saw in training — **exposure bias**. Scheduled sampling (Bengio et
  al., NeurIPS 2015) anneals from true tokens toward sampled ones to close the gap.
- **Echo-state / reservoir networks.** One way to sidestep the exploding-gradient
  problem entirely is not to train the recurrence at all: fix $W_{hh}$ with spectral
  radius tuned just below $1$, let the random reservoir project the input into a rich
  dynamical state, and train _only_ the linear readout. Echo-state networks (Jaeger,
  2001) trade capacity for a convex, gradient-free fit.

The development that mattered most postdates all of this: **sequence-to-sequence
with attention** (Bahdanau et al., ICLR 2015; Sutskever et al., NeurIPS 2014) let a
decoder read _every_ encoder state directly instead of squeezing the source through
one fixed-size hidden vector. That attention step removed the recurrence's memory
bottleneck and led, within two years, to the
[Transformer](/deep-learning/architectures/the-transformer-architecture) dropping
recurrence altogether.

## Takeaways

- A recurrent network is the recurrence $h_t = g(W_{hh} h_{t-1} + W_{xh} x_t + b)$,
  $y_t = W_{hy} h_t$, with **one weight set shared across all time steps** and the
  hidden state $h_t$ as a fixed-size, lossy memory of the prefix.
- **Unrolling** turns the cell into a feed-forward graph $T$ layers deep with tied
  weights; the forward pass is one left-to-right sweep.
- **BPTT** sums the gradient of the shared $W_{hh}$ over every step, and the
  state-to-state term is a **product of one-step Jacobians**
  $\prod_r \partial h_r / \partial h_{r-1}$.
- That product scales like $\rho(W_{hh})^{\,t-s}$: $\rho < 1$ **vanishes**, $\rho > 1$ **explodes**.
  Clip the gradient to control explosion; **gate** the recurrence
  ([LSTM/GRU](/deep-learning/architectures/lstm-and-gru)) to address vanishing.
- Task shapes range over one-to-one, one-to-many, many-to-one, and many-to-many
  (aligned and seq2seq); **bidirectional** RNNs add a backward pass when the whole
  sequence is known up front.

[^gf-rnn]: **Goodfellow**, _Deep Learning_, §10.2 — Recurrent Neural Networks: the shared-weight recurrence $h_t = g(W_{hh}h_{t-1} + W_{xh}x_t + b)$ and the hidden state as a lossy summary of the prefix.
[^gf-unfold]: **Goodfellow**, _Deep Learning_, §10.1 — Unfolding Computational Graphs: rewriting the self-loop as a depth-$T$ feed-forward graph with tied weights.
[^gf-bptt]: **Goodfellow**, _Deep Learning_, §10.2.2 — Computing the Gradient in a Recurrent Network: back-propagation through time as ordinary backprop on the unrolled graph, summing the shared-weight gradient across steps.
[^gf-longterm]: **Goodfellow**, _Deep Learning_, §10.7 — The Challenge of Long-Term Dependencies: the product of Jacobians scaling as $\rho(W_{hh})^{t-s}$, hence vanishing or exploding gradients.
[^gf-bidir]: **Goodfellow**, _Deep Learning_, §10.3 — Bidirectional RNNs: running a forward and a backward recurrence so each output conditions on the entire sequence.