Recurrent Networks

A convolutional network bakes in the prior that the same feature detector should slide across space. A recurrent network bakes in the dual prior for time: the same transition rule should apply at every position in a sequence. It processes $x_1,x_2,\dots ,x_T$ one element at a time, carrying a fixed-size hidden state $h_t$ that summarizes everything seen so far, a learned, lossy memory of the past.¹

The recurrence

A vanilla RNN is one nonlinear map applied repeatedly. At step $t$ it folds the new input $x_t$ into the running state $h_{t-1}$ and emits an output $y_t$:

with $x_t\in {\mathbb{R}}^d$, $h_t\in {\mathbb{R}}^m$, the activation $g$ usually $\tanh$, and the state seeded at $h_0=0$. Three weight matrices carry the whole model:

The defining structural fact is in the subscripts: $W_{hh},W_{xh},W_{hy}$ carry no time index. The same three matrices act at $t=1$ and at $t=T$.

To make the sizing concrete, chase the shapes through one step. Take a concrete $d=3$ input, an $m=4$ state, and a $k=2$ output. Then $W_{xh}$ is $4\times 3$, $W_{hh}$ is $4\times 4$, $b$ is $4\times 1$, and $W_{hy}$ is $2\times 4$. The two matrix-vector products both land in ${\mathbb{R}}^4$, add elementwise, pass through $\tanh$ componentwise, and read out through $W_{hy}$:

The parameter count is $md+m^2+m+km=4\cdot 3+16+4+2\cdot 4=40$, and it stays $40$ whether the sequence is ten steps or ten thousand. The figure below tracks those shapes through one step; the same block is what the unrolled graph copies.

This is the temporal analogue of a convolution's shared filter: it lets one network ingest sequences of any length, and it is a strong regularizer: the count of free parameters does not grow with $T$.

Unrolling through time

The self-loop is a compact fiction. To compute anything we unroll the recurrence: copy the cell once per time step and feed each copy's state into the next. The result is an ordinary feed-forward graph, $T$ layers deep, in which every layer shares weights.²

The forward pass is a left-to-right sweep over the unrolled graph: each step reads the input, updates the state, and emits an output.

Unrolling reframes the model exactly: a recurrent network is a very deep feed-forward network whose depth equals the sequence length and whose every layer is tied to the same weights. Everything about training follows from that one observation.

Backpropagation through time

Because the unrolled graph is feed-forward, it is trained by ordinary backpropagation; applied to the unrolled graph the method is called backpropagation through time (BPTT).³ The total loss sums per-step losses, $L={\sum }_{t=1}^T{L}_t$, and we want $\partial L/\partial W_{hh}$. The subtlety is that $W_{hh}$ is the same matrix at every step, so it influences the loss through every hidden state; its gradient is a sum of contributions across all time steps:

The middle factor is where time enters. To get from step $t$ back to an earlier step $s$, the gradient must pass through every intermediate state, so $\partial h_t/\partial h_s$ is itself a product of one-step Jacobians:

Each link is the recurrent weight $W_{hh}^T$ scaled by the activation derivative $g^{\prime }(a_r)$. The gradient flows backward through the unrolled chain, accumulating one such Jacobian per step.

The same backward sweep accumulates $\partial L/\partial W_{hh}$, $\partial L/\partial W_{xh}$, $\partial L/\partial W_{hy}$, and $\partial L/\partial b$ by adding each step's contribution into a shared accumulator.

The cost is memory: BPTT must cache every $h_t$ for the backward pass, so memory grows with $T$. For a sequence of $10,000$ tokens the graph is $10,000$ layers deep, and the whole stack of activations has to live in memory until the backward sweep reaches step $1$. Truncated BPTT caps this by chopping the sequence into windows of $k$ steps: the forward pass still runs the full length, carrying $h_t$ across window boundaries, but the backward pass only flows within each window and is cut at the seam. Gradients travel at most $k$ steps back, which bounds memory and compute at the price of any dependency longer than $k$.

Vanishing and exploding gradients

The product of Jacobians is also the network's deepest weakness.⁴ Hold the activation roughly linear ($g^{\prime }\approx 1$) so the one-step Jacobian is essentially $W_{hh}^T$; then the state-to-state gradient across $t-s$ steps is a matrix power:

Diagonalize $W_{hh}=Q\Lambda Q^{-1}$ with eigenvalues ${\lambda }_i$. A matrix power raises each eigenvalue to the same power, so the gradient's growth is governed entirely by the spectral radius $\rho (W_{hh})={\max }_i|{\lambda }_i|$:

The number ${\lambda }_i^{t-s}$ either collapses to $0$ or blows up to $\infty$ as the gap $t-s$ grows, depending on a single threshold:

For example, suppose the largest eigenvalue is $\rho =0.9$. Over a $50$-step gap the gradient is scaled by $0.9^{50}\approx 0.005$, and over $100$ steps by $0.9^{100}\approx 2.7\times 10^{-5}$: the signal from a hundred steps back arrives four to five orders of magnitude weaker than a signal from one step back, drowned out by nearer terms and by numerical noise. Flip the eigenvalue to $\rho =1.1$ and the same $100$ steps multiply the gradient by $1.1^{100}\approx 1.4\times 10^4$, and the loss overflows to NaN in a few updates. The stable band around $\rho =1$ is vanishingly thin, and nothing in ordinary gradient descent pins $W_{hh}$ to it.

The two failures need different fixes, and only one is easy.

Gradient clipping rescales the gradient whenever its norm exceeds a threshold $\tau$, capping the step size without changing its direction:

Clipping controls explosion but does nothing for vanishing — a signal that has already decayed to numerical zero cannot be amplified. The real fix is to change the recurrence so the gradient has a near-identity path to travel along, which is precisely what gated cells provide.

Sequence-task shapes

The same recurrent cell serves very different tasks depending on which inputs and outputs are present. Counting elements on each side gives a small taxonomy.

The distinction is purely in the wiring, not the cell. One-to-many feeds a single input at the first step and then generates by looping its own output back in as the next input — an image-captioning model conditions on the image once, then emits words until a stop token. Many-to-one ignores every intermediate output and reads a single prediction off the final state $h_T$, the only vector that has seen the whole sequence; a sentiment classifier does exactly this. Aligned many-to-many emits one output per input in lockstep, the natural shape for per-token labeling like part-of-speech tagging.

The seq2seq case is special: an encoder RNN reads the whole input into a final state, which seeds a decoder RNN that generates the output. The input and output lengths need not match — a five-word English sentence can become a seven-word French one — because the decoder runs its own clock, generating until it emits a stop token. The bottleneck of forcing all meaning through one fixed vector $h_T^{\text{enc}}$ is the strain attention was invented to relieve.

Bidirectional RNNs

A plain RNN at step $t$ has seen only $x_1,\dots ,x_t$. For aligned tasks where the whole sequence is available up front — tagging a word given its full sentence — the future is just as informative as the past. A bidirectional RNN runs two independent recurrences, one forward and one backward, and concatenates their states.⁵

The output stacks both directions, $y_t=W_{hy}[h_t^{\to };h_t^{\leftarrow }]$, so every prediction is conditioned on the full context. Bidirectionality fits tagging and classification, where the entire input is known in advance; it is unavailable for streaming or generation, where the future has not happened yet.

Around and after the plain RNN

Three practical mechanisms sit around the plain recurrence in Goodfellow's Chapter 10 but are easy to miss on a first read, and one later development is the bridge to everything after RNNs.

• Truncated BPTT. The unrolled graph is $T$ layers deep, and full BPTT stores every hidden state for the backward pass — $O(T)$ memory, impossible for a book- length sequence. Truncated BPTT processes the stream in windows of length $k$ (say $k=35$ tokens): it carries the hidden state forward across window boundaries but only propagates the gradient back $k$ steps before detaching. Concretely, a $10,000$-token document with $k=35$ needs the memory of a $35$-step graph, not a $10,000$-step one — a $\approx 285\times$ saving — at the price that no dependency longer than $35$ steps gets a gradient. This makes the vanishing-gradient limit a design parameter rather than an accident.
• Teacher forcing and exposure bias. During training a generative RNN is fed the true previous token $x_{t-1}$ rather than its own prediction, which decouples the steps and lets the whole sequence train in parallel (teacher forcing). At inference it must consume its own outputs, so a single early mistake shifts the model into states it never saw in training — exposure bias. Scheduled sampling (Bengio et al., NeurIPS 2015) anneals from true tokens toward sampled ones to close the gap.
• Echo-state / reservoir networks. One way to sidestep the exploding-gradient problem entirely is not to train the recurrence at all: fix $W_{hh}$ with spectral radius tuned just below $1$, let the random reservoir project the input into a rich dynamical state, and train only the linear readout. Echo-state networks (Jaeger, 2001) trade capacity for a convex, gradient-free fit.

The development that mattered most postdates all of this: sequence-to-sequence with attention (Bahdanau et al., ICLR 2015; Sutskever et al., NeurIPS 2014) let a decoder read every encoder state directly instead of squeezing the source through one fixed-size hidden vector. That attention step removed the recurrence's memory bottleneck and led, within two years, to the Transformer dropping recurrence altogether.

Takeaways

• A recurrent network is the recurrence $h_t=g(W_{hh}{h}_{t-1}+W_{xh}{x}_t+b)$, $y_t=W_{hy}{h}_t$, with one weight set shared across all time steps and the hidden state $h_t$ as a fixed-size, lossy memory of the prefix.
• Unrolling turns the cell into a feed-forward graph $T$ layers deep with tied weights; the forward pass is one left-to-right sweep.
• BPTT sums the gradient of the shared $W_{hh}$ over every step, and the state-to-state term is a product of one-step Jacobians${\prod }_r\partial h_r/\partial h_{r-1}$.
• That product scales like $\rho (W_{hh}{)}^{t-s}$: $\rho <1$ vanishes, $\rho >1$ explodes. Clip the gradient to control explosion; gate the recurrence (LSTM/GRU) to address vanishing.
• Task shapes range over one-to-one, one-to-many, many-to-one, and many-to-many (aligned and seq2seq); bidirectional RNNs add a backward pass when the whole sequence is known up front.