---
title: Deep Equilibrium Models
module: Theory & Frontiers
moduleNumber: 6
lessonNumber: 5
order: 605
summary: >
  A deep network need not be a fixed stack of layers; it can be a single
  weight-tied layer iterated to convergence, its output defined implicitly as the
  fixed point $z^\star = f_\theta(z^\star, x)$. The forward pass becomes
  root-finding and the backward pass becomes implicit differentiation, so training
  costs O(1) memory regardless of effective depth. We derive both passes from the
  implicit function theorem and close the course on defining a layer by a
  fixed-point condition rather than an explicit stack.
topics: [Theory & Frontiers]
sources:
  - book: Goodfellow
    ref: "Ch. 6 — Deep Feedforward Networks; §6.5 Back-Propagation"
  - book: Goodfellow
    ref: "Ch. 8 — Optimization; §8.2 Numerical fixed-point / Newton methods"
---

Every architecture so far has been a **recipe**: an ordered list of layers, each
applied once, the output read off the top of the stack. A
[residual block](/deep-learning/architectures/cnn-architectures), an
[LSTM](/deep-learning/architectures/lstm-and-gru) cell, a
[transformer](/deep-learning/architectures/attention-and-transformers) block —
all are functions you _run_. A **deep equilibrium model** (DEQ) instead defines a
layer not by what it computes but by a _condition_ its
output must satisfy, and lets a black-box solver find whatever output meets the
condition.[^postdate-deq]

> **Definition (Implicit layer).** A layer whose output $z^\star$ is defined not
> by an explicit formula $z^\star = g(x)$ but as the solution of an equation
> $h_\theta(z^\star, x) = 0$. The forward pass _solves_ for $z^\star$; no fixed
> sequence of operations is prescribed.

## A layer that calls itself

Take a single weight-tied transformation $f_\theta(\cdot, x)$ (one layer's worth
of parameters, conditioned on the input $x$) and apply it over and over to its
own output. If the iteration settles, it settles at a point that the layer maps to
itself.

> **Definition (Deep equilibrium model).** A DEQ is the **fixed point** of a
> single weight-tied layer,
> $$
> z^\star = f_\theta(z^\star, x),
> $$
> i.e. the output that, fed back in, reproduces itself. The prediction is read
> from $z^\star$ via an output head $\hat y = g_\phi(z^\star)$.

This describes an **infinite-depth weight-tied network** run to convergence.
Stack the same block $L$ times and you compute $z_L = f_\theta(\,\cdots
f_\theta(f_\theta(z_0, x), x)\cdots, x)$; as $L \to \infty$, if the iteration
converges, $z_L \to z^\star$ independent of $z_0$. The infinitely deep stack and
the self-referential layer are two descriptions of one object.

$$
% caption: As $L\to\infty$, a weight-tied stack collapses into one self-referential block whose output feeds back into itself: the fixed point $z^\star = f_\theta(z^\star, x)$.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  blk/.style={draw, minimum width=13mm, minimum height=10mm, align=center},
  acc/.style={draw=acc, text=acc, thick}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{accmid}{HTML}{6A82F6}
  % explicit weight-tied stack
  \node[blk] (b1) at (0,0)   {$f$};
  \node[blk] (b2) at (1.7,0) {$f$};
  \node[blk] (b3) at (3.4,0) {$f$};
  \node[font=\footnotesize] (dots) at (5.0,0) {. . .};
  \node[blk] (b4) at (6.5,0) {$f$};
  \draw[->, thick] (-1.3,0) -- (b1) node[midway, above] {input};
  \draw[->, thick] (b1) -- (b2);
  \draw[->, thick] (b2) -- (b3);
  \draw[->, thick] (b3) -- (dots);
  \draw[->, thick] (dots) -- (b4);
  \draw[->, thick] (b4) -- ++(1.3,0) node[midway, above] {output};
  \node[font=\footnotesize, text=black] at (3.25,-1.0) {same weights, unbounded depth};
  % arrow to collapsed form
  \draw[->, accmid, thick] (3.25,-1.55) -- (3.25,-2.4);
  \node[font=\footnotesize, anchor=west] at (3.5,-1.95) {collapses to};
  % collapsed self-referential block
  \node[blk, acc] (eq) at (3.25,-3.4) {$f$};
  \draw[->, thick] (1.4,-3.4) -- (eq) node[midway, above] {$x$};
  \draw[->, thick] (eq) -- ++(1.85,0) node[midway, above] {output};
  % feedback loop
  \draw[->, acc, thick] (eq.north) .. controls (5.0,-2.6) and (5.0,-4.2) .. (eq.south);
  \node[font=\footnotesize, text=acc, anchor=west] at (5.1,-3.4) {feed output back in};
  \node[font=\footnotesize, text=acc] at (3.25,-4.6) {f\/ixed point};
\end{tikzpicture}
$$

The payoff is structural. A conventional $L$-layer net stores $L$
sets of activations for the backward pass; the DEQ stores one, at any effective
depth — provided we can solve and
differentiate the fixed-point condition, which is the rest of this lesson.

## Forward pass = root-finding

Finding $z^\star$ is solving a nonlinear system. Rearrange the fixed-point
condition into a **residual** whose zero is the equilibrium:

$$
h_\theta(z, x) \;=\; z - f_\theta(z, x) \;=\; 0.
$$

Any root-finder applies. The simplest is **fixed-point iteration** (literally
re-running the layer), which converges when $f_\theta$ is a contraction.[^gf-fixedpoint]

> **Theorem (Contraction convergence).** If $f_\theta(\cdot, x)$ is Lipschitz in
> its first argument with constant $K < 1$, i.e.
> $\norm{f_\theta(z, x) - f_\theta(z', x)} \le K\,\norm{z - z'}$ for
> all $z, z'$, then the iteration $z_{k+1} = f_\theta(z_k, x)$ converges to a
> unique fixed point $z^\star$ from any start $z_0$, and the error contracts
> geometrically, $\norm{z_k - z^\star} \le K^k\,\norm{z_0 - z^\star}$.

> **Proof.** Subtract the fixed-point identity $z^\star = f_\theta(z^\star, x)$
> from the update and apply the Lipschitz bound:
> $$
> \norm{z_{k+1} - z^\star}
> = \norm{f_\theta(z_k, x) - f_\theta(z^\star, x)}
> \le K\,\norm{z_k - z^\star}.
> $$
> Iterating the inequality gives $\norm{z_k - z^\star} \le K^k\,\norm{z_0 -
> z^\star} \to 0$ since $K < 1$. Uniqueness: if $z^\star$ and $w^\star$ were
> both fixed points, $\norm{z^\star - w^\star} = \norm{f_\theta(z^\star, x) -
> f_\theta(w^\star, x)} \le K\norm{z^\star - w^\star}$ forces
> $\norm{z^\star - w^\star} = 0$. $\qed$

The contraction constant is governed by the **Jacobian** of $f_\theta$: the
iteration converges locally when the spectral radius $\rho(J_f) < 1$, where
$J_f = \partial f_\theta / \partial z$ evaluated near $z^\star$. Each step of plain
fixed-point iteration multiplies the error by roughly $\rho(J_f)$, so convergence
is linear and can be slow when $\rho(J_f)$ is close to $1$.

For example, take the one-dimensional layer $f(z) = 0.5\,z + 1$,
a contraction with $K = 0.5 < 1$. Its fixed point solves $z^\star = 0.5\,z^\star +
1$, so $z^\star = 2$. Start at $z_0 = 0$ and iterate:

| step $k$ | $z_k$ | error $\abs{z_k - 2}$ | ratio to previous |
| --- | --- | --- | --- |
| $0$ | $0.000$ | $2.000$ | — |
| $1$ | $1.000$ | $1.000$ | $0.50$ |
| $2$ | $1.500$ | $0.500$ | $0.50$ |
| $3$ | $1.750$ | $0.250$ | $0.50$ |
| $4$ | $1.875$ | $0.125$ | $0.50$ |

The error halves every step, exactly the predicted geometric decay $K^k = 0.5^k$:
reaching a tolerance of $10^{-3}$ takes about $11$ steps ($0.5^{11}\cdot 2 \approx
10^{-3}$). Now sharpen the map to $f(z) = 0.95\,z + 0.1$, still a contraction but
with $K = 0.95$. Its fixed point is $z^\star = 2$ as well, but the error now shrinks
by only $5\%$ per step, so the same tolerance needs about $150$ iterations — two
orders of magnitude slower. This is why the slope of the map at $z^\star$, not just
the fact that it is below $1$, decides whether plain iteration is practical, and why
the Newton and Broyden solvers below justify their extra per-step cost.

$$
% caption: Cobweb plot of $z_{k+1}=f(z_k,x)$: the iterate spirals into the fixed point
% $z^\star$ where $f$ meets $y=x$, converging when the slope $\abs{f'(z^\star)}<1$.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % axes
  \draw[->, thick] (0,0) -- (4.6,0) node[right] {$z_k$};
  \draw[->, thick] (0,0) -- (0,4.6) node[above] {$z_{k+1}$};
  % identity line y = x
  \draw[black, thick] (0,0) -- (4.3,4.3) node[pos=0.9, above left, black] {$y = x$};
  % the map f: a contraction with slope < 1, crossing y=x at z* = 2.6
  % f(z) = 0.45 z + 1.43  => fixed point at 1.43/0.55 = 2.6
  \draw[acc, very thick] (0,1.43) -- (4.3,3.37) node[pos=0.78, below right, text=acc] {$f$};
  % fixed point
  \fill[green] (2.6,2.6) circle (2.4pt);
  \node[green, anchor=north west, font=\scriptsize] at (2.78,2.5) {equilibrium};
  % cobweb steps from z0 = 0.5
  % step 1: vertical to f(0.5)=1.655
  \draw[red!75, thick] (0.5,0) -- (0.5,1.655);
  \draw[red!75, thick] (0.5,1.655) -- (1.655,1.655);
  \draw[red!75, thick] (1.655,1.655) -- (1.655,2.175);
  \draw[red!75, thick] (1.655,2.175) -- (2.175,2.175);
  \draw[red!75, thick] (2.175,2.175) -- (2.175,2.41);
  \draw[red!75, thick] (2.175,2.41) -- (2.41,2.41);
  \draw[red!75, thick] (2.41,2.41) -- (2.41,2.52);
  \draw[red!75, thick] (2.41,2.52) -- (2.52,2.52);
  \node[anchor=north] at (0.5,-0.05) {$z_0$};
  \definecolor{red}{HTML}{C0392B}
\end{tikzpicture}
$$

Plain iteration is rarely the fastest route. Treating $h_\theta(z, x) = 0$ as a
general root-finding problem licenses quasi-Newton solvers (**Newton** and
**Broyden**) that use (an approximation of) the Jacobian to take superlinear
steps.[^gf-newton] The forward solver is genuinely a _black box_: the rest of the
model needs only a point that satisfies the residual to tolerance, however it was
found.

| Solver | Update | Per-step cost | Convergence |
| --- | --- | --- | --- |
| Fixed-point iteration | $z \gets f_\theta(z, x)$ | one layer eval | linear, rate $\rho(J_f)$ |
| Newton | $z \gets z - (I - J_f)^{-1} h$ | solve a linear system | quadratic (local) |
| Broyden | $z \gets z - B^{-1} h$, $B \approx I - J_f$ | low-rank update | superlinear |

$$
% caption: Forward-solve convergence: plain fixed-point iteration decays linearly (error $\times\rho$ per step) while Broyden/Newton bend to superlinear, reaching tolerance in far fewer layer evals.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \draw[->, thick] (0,0) -- (6.2,0) node[right] {iteration $k$};
  \draw[->, thick] (0,0) -- (0,3.8) node[above, align=center] {log residual};
  \draw[black, dashed] (0,0.4) -- (6.0,0.4) node[right, black, font=\scriptsize] {tol};
  % linear (fixed-point): straight line on a log axis
  \draw[acc, very thick] (0.2,3.4) -- (5.6,0.55);
  \node[acc, anchor=west] at (3.5,2.35) {f\/ixed point};
  \node[acc, font=\scriptsize, anchor=west] at (3.5,2.0) {linear};
  % superlinear (Broyden/Newton): concave-down, plunges early
  \draw[green, very thick] plot[domain=0.2:2.6, samples=40] (\x, {3.4 - 0.9*(\x-0.2)*(\x-0.2)});
  \node[green, anchor=west] at (0.35,1.0) {Broyden};
  \node[green, font=\scriptsize, anchor=west] at (0.35,0.65) {superlinear};
\end{tikzpicture}
$$

```algorithm
caption: $\textsc{DEQForward}(f_\theta, x, z_0, \texttt{tol})$ — solve for the fixed point
$z \gets z_0$ // any initial guess, e.g. zeros
repeat
  $z_{\text{next}} \gets f_\theta(z, x)$ // re-run the weight-tied layer (or a Newton/Broyden step)
  $r \gets \norm{z_{\text{next}} - z}$ // residual norm
  $z \gets z_{\text{next}}$
until $r \le \texttt{tol}$ // converged to the equilibrium
return $z$ // this is $z^\star$
```

## Backward pass = implicit differentiation

To train the DEQ we need $\partial z^\star / \partial
\theta$ — how the equilibrium moves when the weights move. The naive route is to
**backprop through every solver iteration**, storing each intermediate $z_k$;
that destroys the memory advantage and ties the gradient to an arbitrary solver
trajectory. The **implicit function theorem** lets us skip the solver entirely and
differentiate the _condition_ instead.[^gf-backprop]

The fixed point satisfies $z^\star = f_\theta(z^\star, x)$ as an identity in
$\theta$. Differentiate both sides with respect to any quantity $(\cdot)$,
whether $\theta$ or the input $x$, and apply the chain rule, remembering that
$z^\star$ itself depends on $(\cdot)$:

$$
\frac{\partial z^\star}{\partial (\cdot)}
=
\underbrace{\frac{\partial f_\theta}{\partial z}\abs{_{z^\star}}_{J_f}
\frac{\partial z^\star}{\partial (\cdot)}
+
\frac{\partial f_\theta}{\partial (\cdot)}}_{z^\star}.
$$

Collect the $\partial z^\star / \partial (\cdot)$ terms on the left and factor:

$$
\parens{I - J_f}\frac{\partial z^\star}{\partial (\cdot)}
=
\frac{\partial f_\theta}{\partial (\cdot)}\abs{_{z^\star}
\;\Longrightarrow\;
\;
\frac{\partial z^\star}{\partial (\cdot)}
=
\parens{I - J_f}^{-1}\frac{\partial f_\theta}{\partial (\cdot)}_{z^\star}
\;}
$$

This is a closed form for the layer's Jacobian that mentions **only the fixed
point $z^\star$ and the local Jacobian $J_f$** — not a single solver iterate. The
matrix $I - J_f$ is invertible exactly when $\rho(J_f) < 1$, the same condition
that made the forward pass converge.

For example, return to $f(z) = 0.5\,z + 1$ from
the forward-pass trace, but read $0.5$ as a parameter: $f_\theta(z) = \theta z + 1$
with $\theta = 0.5$, whose fixed point is $z^\star = 1/(1-\theta) = 2$. Here $J_f =
\partial f/\partial z = \theta = 0.5$ and $\partial f/\partial\theta = z^\star = 2$,
so the implicit-gradient formula gives

$$
\frac{\partial z^\star}{\partial\theta}
= (1 - J_f)^{-1}\frac{\partial f}{\partial\theta}
= \frac{1}{1 - 0.5}\cdot 2 = 4.
$$

Check it directly: $z^\star(\theta) = 1/(1-\theta)$, so $\mathrm{d}z^\star/\mathrm{d}
\theta = 1/(1-\theta)^2 = 1/0.25 = 4$. The implicit formula and the explicit
derivative agree exactly, and the $(1 - J_f)^{-1}$ factor is precisely the
$1/(1-\theta)$ that a naive one-step derivative would miss — it accounts for the
fact that changing $\theta$ moves the whole equilibrium, not just one application of
$f$. In many dimensions this scalar reciprocal becomes the matrix inverse
$(I - J_f)^{-1}$, and the same "the equilibrium shifts too" correction is what the
adjoint solve below recovers without ever forming that inverse.

> **Theorem (Implicit gradient).** Let $z^\star$ solve $z = f_\theta(z, x)$ and let
> $\mathcal{L} = \mathcal{L}(g_\phi(z^\star))$ be the loss. If $I - J_f$ is
> invertible at $z^\star$, the gradient of the loss with respect to $\theta$ is
> $$
> \frac{\partial \mathcal{L}}{\partial \theta}
> =
> \frac{\partial \mathcal{L}}{\partial z^\star}
> \parens{I - J_f}^{-1}
> \frac{\partial f_\theta}{\partial \theta}\bigg|_{z^\star},
> $$
> computed without backpropagating through the forward solver.

In practice we never form $(I - J_f)^{-1}$: it is a huge dense matrix. We need
only a **vector–Jacobian product**: the upstream gradient $v^\top = \partial
\mathcal{L} / \partial z^\star$ times the inverse. Define the adjoint $u$ by

$$
u^\top \parens{I - J_f} = v^\top
\quad\Longleftrightarrow\quad
u = v + J_f^\top\, u,
$$

which is _itself a fixed-point equation_ — solvable with the same black-box solver
used in the forward pass, since $J_f^\top$ acts on a vector through one
reverse-mode autodiff call on $f_\theta$.[^gf-vjp] Once $u$ is found, every parameter
gradient is the cheap product $u^\top\,\partial f_\theta / \partial \theta$.

```algorithm
caption: $\textsc{DEQBackward}(f_\theta, z^\star, x, v)$ — implicit gradient via an adjoint solve
$u \gets v$ // initialize the adjoint, $v = \partial \mathcal{L} / \partial z^\star$
repeat
  $u_{\text{next}} \gets v + \textsc{vjp}(f_\theta, z^\star, u)$ // $\textsc{vjp}$ computes $J_f^{T} u$ by one autodiff call
  $r \gets \norm{u_{\text{next}} - u}$
  $u \gets u_{\text{next}}$
until $r \le \texttt{tol}$ // adjoint fixed point reached
return $u^{T}\,\partial f_\theta / \partial \theta$ // parameter gradient
```

> **Remark (Constant memory).** The forward solver may take dozens of iterations,
> but the backward pass references _only_ $z^\star$ and evaluates $f_\theta$ at
> that single point. No intermediate state is stored, so training memory is
> $O(1)$ in the effective depth — independent of how many solver steps either pass
> takes.

$$
% caption: Two-phase training: a forward root-solve returns the equilibrium $z^\star$ and a backward adjoint solve yields the gradient, neither phase storing solver iterates.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=30mm, minimum height=13mm, align=center},
  acc/.style={draw=acc, text=acc, thick}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % forward phase
  \node[box, acc] (fwd) at (0,0) {root-sol\/ve\\residual vanishes};
  \node[box] (zstar) at (4.6,0) {equilibrium\\(only this stored)};
  \node[box] (loss) at (9.0,0) {loss};
  \draw[->, acc, thick] (-2.4,0) -- (fwd) node[midway, above] {$x$};
  \draw[->, acc, thick] (fwd) -- (zstar);
  \draw[->, acc, thick] (zstar) -- (loss);
  \node[font=\footnotesize, text=acc, anchor=south] at (2.3,0.75) {forward};
  % backward phase
  \node[box] (vec) at (9.0,-2.5) {upstream grad};
  \node[box, draw=green, text=green, thick] (adj) at (4.6,-2.5) {adjoin\/t sol\/ve\\$u = v + J^{T} u$};
  \node[box] (grad) at (0,-2.5) {param grad};
  \draw[->, green, thick] (loss) -- (vec);
  \draw[->, green, thick] (vec) -- (adj);
  \draw[->, green, thick] (adj) -- (grad);
  \node[font=\footnotesize, text=green, anchor=north] at (2.3,-3.25) {backward};
  % link showing z* reused
  \draw[black, dashed, thick] (zstar.south) -- (adj.north) node[midway, right, black] {reuse equilibrium};
\end{tikzpicture}
$$

## Explicit deep net vs DEQ

The two designs compute related functions but the cost of depth falls in different
places. The explicit net spends **memory** (one activation buffer per layer)
and **parameters** (one weight set per layer); the DEQ spends **solver
iterations** at run time and stores only a single equilibrium.

| Property | Explicit deep net | DEQ |
| --- | --- | --- |
| Depth | fixed $L$ layers | effectively infinite (run to convergence) |
| Parameters | $L$ distinct weight sets | one weight-tied $\theta$ |
| Forward cost | $L$ layer evals | variable: solver iterations to tolerance |
| Training memory | $O(L)$ — store every activation | $O(1)$ — store only $z^\star$ |
| Backward pass | backprop through $L$ stored layers | implicit diff: one adjoint solve |
| Depth–memory coupling | linear (deeper $\Rightarrow$ more memory) | decoupled (depth is free) |

$$
% caption: Training memory: the explicit net stores every layer's activation for backprop, while the DEQ keeps only $z^\star$, one buffer regardless of effective depth.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  cell/.style={draw, minimum width=11mm, minimum height=6mm}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  % explicit: a tall stack of stored activations
  \node[font=\footnotesize] at (0,3.0) {explicit net};
  \foreach \i/\y in {1/0, 2/0.65, 3/1.3, 4/1.95, 5/2.6} {
    \node[cell, fill=acc!18] (e\i) at (0,\y) {};
  }
  \node[font=\footnotesize, text=black, anchor=north, align=center] at (0,-0.45)
    {stores all $L$\\\texttt{activations}};
  \draw[<->, thick] (1.0,0) -- (1.0,2.6) node[midway, right] {$O(L)$};
  % DEQ: a single stored buffer
  \node[font=\footnotesize] at (5.5,3.0) {DEQ};
  \node[cell, draw=green, fill=green!18, thick] (d1) at (5.5,0) {};
  \node[font=\footnotesize, text=green, anchor=west] at (6.15,0) {equilibrium only};
  \node[font=\footnotesize, text=black, anchor=north, align=center] at (5.5,-0.45)
    {one slot,\\any depth};
  \draw[<->, thick] (4.5,0) -- (4.5,0.35) node[midway, left] {$O(1)$};
\end{tikzpicture}
$$

## Convergence in practice

Because the forward pass is a solver, training surfaces a diagnostic the explicit
net never had: the **residual** $\norm{z_k - f_\theta(z_k, x)}$, which a
healthy DEQ drives toward zero each step. A residual that stalls or grows signals
$\rho(J_f) \ge 1$ — the layer is not contractive, the fixed point is not being
reached, and gradients computed from a non-equilibrium $z_k$ are wrong. Stabilizing
$\rho(J_f) < 1$ (via spectral normalization or a Jacobian regularizer) is the
central engineering concern.

$$
% caption: Residual $\norm{z_k - f_\theta(z_k,x)}$ versus solver iteration: a contractive
% layer decays geometrically to tolerance, while a non-contractive one stalls.
\begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \definecolor{red}{HTML}{C0392B}
  % axes
  \draw[->, thick] (0,0) -- (6.2,0) node[right] {iteration $k$};
  \draw[->, thick] (0,0) -- (0,3.8) node[above, align=center] {residual};
  % tolerance line
  \draw[black, dashed] (0,0.4) -- (6.0,0.4) node[right, black, font=\scriptsize] {tol};
  % converging curve (geometric decay): contractive
  \draw[green, very thick]
    plot[domain=0:5.6, samples=60] (\x, {0.35 + 3.1*exp(-0.95*\x)});
  \node[green, anchor=west] at (2.2,1.35) {converges};
  \node[green, font=\scriptsize, anchor=west] at (2.2,1.0) {rho $< 1$};
  % stalling curve: non-contractive, flattens above tol
  \draw[red, very thick]
    plot[domain=0:5.6, samples=60] (\x, {1.7 + 1.7*exp(-0.9*\x)});
  \node[red, anchor=west] at (4.0,2.35) {stalls};
  \node[red, font=\scriptsize, anchor=west] at (4.0,2.05) {rho at least 1};
\end{tikzpicture}
$$

## Tradeoffs and uses

The DEQ gains **constant memory** and **infinite effective depth** at the cost of
**solver time** and **stability**:

| Gain | Cost |
| --- | --- |
| $O(1)$ training memory regardless of depth | forward/backward each run an iterative solver |
| infinite effective depth from one weight set | wall-clock cost varies with input difficulty |
| decoupling depth from parameter count | requires $\rho(J_f) < 1$; can diverge if not |
| a single uniform module to tune | gradients invalid if the solver hasn't converged |

The natural fit is settings where depth helps but memory is the bottleneck and a
single repeated transformation is a reasonable inductive bias: **sequence models**
(a weight-tied recurrence iterated to equilibrium) and **graph models** (message
passing run to a steady state over the graph). In both, a DEQ matches the accuracy
of a deep explicit stack while training within a fixed memory budget.

## Implicit layers as a family

DEQ is one member of a broader **implicit-layer** family, all of which replace "run
these operations" with "return the output satisfying this condition," and all of
which train through the condition rather than the computation. The public literature
gives three landmarks.

**Deep equilibrium models** (Bai, Kolter & Koltun, 2019) introduced the construction
above and showed a single weight-tied DEQ block matches deep transformers and
weight-tied LSTMs on sequence tasks at constant memory. The follow-up **multiscale
DEQ** (Bai, Koltun & Kolter, 2020) drove a single implicit block to competitive
ImageNet accuracy by solving for equilibria at several resolutions at once, evidence
that the constant-memory trick scales beyond toy settings.

The open problem noted above — that nothing guarantees
$\rho(J_f) < 1$, so the solver can diverge — is what **monotone DEQ** (Winston &
Kolter, 2020) addresses. By parameterizing the layer so the residual operator is
provably monotone, it guarantees a unique fixed point and a convergent solver by
construction, trading a little expressiveness for a stability certificate. This is
the DEQ analogue of the certified defenses from
[adversarial robustness](/deep-learning/theory/adversarial-defenses): prove the
property rather than hope for it.

The closest cousin is the **neural ODE** (Chen et al., 2018), which takes the
opposite limit of the same idea. Where a DEQ is a weight-tied layer iterated to a
_fixed point_, a neural ODE is a residual block whose step size shrinks to zero, so
the forward pass integrates $\mathrm{d}z/\mathrm{d}t = f_\theta(z, t)$ and the
backward pass solves an adjoint ODE — again $O(1)$ memory, again differentiating a
_solver_ rather than storing its trace. Fixed-point equations and differential
equations are two routes to the same infinite effective depth.

| Implicit model | Output defined by | Forward solver | Backward |
| --- | --- | --- | --- |
| DEQ | fixed point $z = f_\theta(z, x)$ | root-find (Broyden/Newton) | implicit-function adjoint |
| Monotone DEQ | fixed point, monotone operator | provably convergent split | implicit adjoint, guaranteed |
| Neural ODE | ODE flow $\dot z = f_\theta(z, t)$ | numerical integrator | adjoint ODE (reverse-time) |
| Optimization layer | $\arg\min_z E_\theta(z, x)$ | inner optimizer | KKT differentiation |

The last row, the **optimization layer** (OptNet, Amos & Kolter, 2017), closes the
circle: its output is the argmin of an inner objective, differentiated through the
optimality (KKT) conditions — the same implicit-function move, applied to a minimizer
instead of a fixed point. All four share one design: define the output by a
condition, make the condition differentiable, and let a solver and the
implicit-function theorem do the rest.

## Closing the loop

DEQ postdates [Goodfellow](/deep-learning/foundations/what-is-deep-learning)
(present it as a frontier method), but it is built entirely from machinery the
course has already laid down: the **autodiff** of
[backpropagation](/deep-learning/neural-networks/backpropagation), the
**fixed-point** and **Newton** ideas from the
[optimization landscape](/deep-learning/optimization/the-optimization-landscape),
and the **weight tying** of [recurrent
networks](/deep-learning/architectures/recurrent-networks). And it returns us to
the very first idea of the course. We opened by saying a deep network is the
composition of simple differentiable pieces learned by following the gradient; the
DEQ shows that "deep" need not mean "many stacked layers" at all — a layer can be a
_condition_ a single transformation satisfies, run to convergence, differentiated
through its own definition. From the [perceptron's](/deep-learning/foundations/linear-models-and-the-perceptron)
weighted sum to an infinite-depth network that fits in constant memory, the same
principle applies: define what the output should satisfy, make it differentiable,
and let the gradient do the learning.

[^postdate-deq]: Deep equilibrium models (Bai, Kolter & Koltun, 2019) postdate **Goodfellow** (2016), which covers the components a DEQ is assembled from — feedforward layers (Ch. 6) and fixed-point/Newton methods (Ch. 8) — but not the implicit-layer construction itself.
[^gf-fixedpoint]: **Goodfellow**, _Deep Learning_, §8.2 — fixed-point iteration and the contraction condition under which $z_{k+1}=f(z_k)$ converges to a unique equilibrium.
[^gf-newton]: **Goodfellow**, _Deep Learning_, §4.3 / §8.6 — Newton's method and curvature-based updates: using the Jacobian/Hessian to take superlinear root-finding steps.
[^gf-backprop]: **Goodfellow**, _Deep Learning_, §6.5 — Back-Propagation: reverse-mode differentiation over the computational graph, here replaced by implicit differentiation of the fixed-point condition.
[^gf-vjp]: **Goodfellow**, _Deep Learning_, §6.5.9 — vector–Jacobian products: backprop computes $J^{T}v$ with one reverse-mode pass, never materializing the full Jacobian.