---
title: Optimization and Lagrange Multipliers
module: Partial Derivatives
moduleNumber: 11
lessonNumber: 5
order: 1105
summary: >
  Extrema of a two-variable function sit at critical points where the gradient
  vanishes; the Second Derivatives Test sorts them into peaks, valleys, and
  saddles by the sign of a discriminant. Absolute extrema on a closed region also
  need the boundary. When the domain is itself a constraint curve, Lagrange
  multipliers set the two gradients parallel and solve the constrained problem.
topics: [Partial Derivatives]
draft: false
sources:
  - book: Stewart
    ref: "Ch. 14 — Partial Derivatives; §14.7 Maximum and Minimum Values"
  - book: Stewart
    ref: "Ch. 14; §14.8 Lagrange Multipliers"
---

The high and low points of a surface are the two-variable version of the maxima
and minima of single-variable calculus. Finding them repeats the familiar
structure (locate where the derivative vanishes, then classify), but with two
new features: the derivative is now a pair of partials, and a critical point can
be neither a max nor a min but a **saddle**, rising in one direction and falling
in another.

## Local extrema and critical points

> **Definition (Local extremum).** $f$ has a local maximum at $(a, b)$ if
> $f(x, y) \le f(a, b)$ for all $(x, y)$ in some disk about $(a, b)$; a local
> minimum if $f(x, y) \ge f(a, b)$ there. If the inequality holds across the
> whole domain, the extremum is absolute.

At a smooth local extremum the surface must be flat, because each axis-trace has
an ordinary extremum there.

> **Theorem (Fermat, two variables).** If $f$ has a local maximum or minimum at
> $(a, b)$ and the first partials exist, then $f_x(a, b) = 0$ and $f_y(a, b) = 0$,
> equivalently $\nabla f(a, b) = \vec 0$.

The proof fixes one variable and applies single-variable Fermat to the trace.
Geometrically, $\nabla f = \vec 0$ makes the tangent plane horizontal.

> **Definition (Critical point).** $(a, b)$ is a critical point of $f$ if
> $\nabla f(a, b) = \vec 0$ or one of the partials fails to exist. Local
> extrema occur only at critical points.

The converse fails: a critical point need not be an extremum. The standard
counterexample is the **saddle** $f(x, y) = y^2 - x^2$ at the origin, where
$\nabla f = \langle -2x, 2y \rangle = \vec 0$, yet $f$ rises along the
$y$-axis and falls along the $x$-axis, so the origin is neither a peak nor a
valley.

$$
% caption: The saddle surface $z = y^2 - x^2$. Slices with $x$ fixed are upward
% parabolas, slices with $y$ fixed are downward parabolas, and the origin sits
% flat between them: a critical point that is no extremum.
\tdplotsetmaincoords{70}{110}
\begin{tikzpicture}[tdplot_main_coords, scale=1.9, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
\definecolor{wrm}{HTML}{C0392B}
% wire mesh: downward parabolas (y fixed), muted
\foreach \Y in {-1,-0.66,-0.33,0.33,0.66,1} {
  \draw[black, thin] plot[domain=-1:1, samples=25, variable=\t] (\t, \Y, {\Y*\Y - \t*\t});
}
% wire mesh: upward parabolas (x fixed), muted
\foreach \X in {-1,-0.66,-0.33,0.33,0.66,1} {
  \draw[black, thin] plot[domain=-1:1, samples=25, variable=\t] (\X, \t, {\t*\t - \X*\X});
}
% emphasized sections through the origin
\draw[wrm, very thick] plot[domain=-1:1, samples=30, variable=\t] (\t, 0, {-\t*\t});
\draw[acc, very thick] plot[domain=-1:1, samples=30, variable=\t] (0, \t, {\t*\t});
% saddle point
\fill[black] (0,0,0) circle (1.6pt);
\node[anchor=south west] at (0.06,0,0.08) {saddle point};
\node[acc, anchor=west] at (0,1.05,1.05) {rises};
\node[wrm, anchor=east] at (-1.05,0,-1.05) {falls};
\end{tikzpicture}
$$

Flattened onto one plane, the two principal cross-sections give the sign
pattern that the Second Derivatives Test below detects algebraically:

$$
% caption: A saddle point: the surface $z=y^2-x^2$ rises along one axis and falls
% along the other, so $\nabla f=\mathbf{0}$ without a local extremum.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\small]
  \definecolor{acc}{HTML}{4A6FA5}
  \definecolor{red}{HTML}{C0392B}
  % up-curve (rising, along y)
  \draw[acc, thick] (-2.4,-1.3) .. controls (-0.9,1.3) and (0.9,1.3) .. (2.4,-1.3);
  \node[acc, font=\footnotesize, anchor=south] at (2.4,-1.25) {rises: $f=y^2$};
  % down-curve (falling, along x)
  \draw[red, thick] (-2.4,1.3) .. controls (-0.9,-1.3) and (0.9,-1.3) .. (2.4,1.3);
  \node[red, font=\footnotesize, anchor=north] at (2.4,1.25) {falls: $f=$ -$x^2$};
  % saddle point
  \fill[black] (0,0) circle (1.8pt);
  \node[anchor=north west, font=\footnotesize] at (0.05,0.0) {saddle};
\end{tikzpicture}
$$

## The Second Derivatives Test

Which kind of critical point is at hand is decided by the second partials,
combined into a discriminant.

> **Theorem (Second Derivatives Test).** Let the second partials of $f$ be
> continuous near $(a, b)$, with $\nabla f(a, b) = \vec 0$. Set
> $$
> D = D(a, b) = f_{xx}(a, b)\,f_{yy}(a, b) - \bigl[f_{xy}(a, b)\bigr]^2
> = \begin{vmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{vmatrix}.
> $$
> - If $D > 0$ and $f_{xx}(a, b) > 0$, then $f(a, b)$ is a **local minimum**.
> - If $D > 0$ and $f_{xx}(a, b) < 0$, then $f(a, b)$ is a **local maximum**.
> - If $D < 0$, then $(a, b)$ is a **saddle point**.
> - If $D = 0$, the test is inconclusive.

The discriminant is the determinant of the matrix of second partials (the
Hessian). Its sign captures whether the surface curves the same way in every
direction ($D > 0$, a bowl or dome) or opposite ways along different axes
($D < 0$, a saddle). When $D > 0$ the sign of $f_{xx}$ then says which.

$$
% caption: The Second Derivatives Test as a decision on $(D, f_{xx})$: $D>0$
% gives an extremum whose type $f_{xx}$ fixes; $D<0$ gives a saddle; $D=0$ is
% undecided.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\small,
  b/.style={draw, minimum width=26mm, minimum height=9mm, font=\footnotesize, align=center}]
  \definecolor{acc}{HTML}{4A6FA5}
  \node[b, draw=acc] (d) at (0,0) {sign of $D$};
  \node[b] (pos) at (4.0,1.7)  {$D>0$};
  \node[b] (neg) at (4.0,0)    {$D<0$: saddle};
  \node[b] (zero) at (4.0,-1.7) {$D=0$: no test};
  \node[b] (min) at (8.2,2.5)  {$f_{xx}>0$: minimum};
  \node[b] (max) at (8.2,0.9)  {$f_{xx}<0$: maximum};
  \draw[->, acc, thick] (d) -- (pos);
  \draw[->, acc, thick] (d) -- (neg);
  \draw[->, acc, thick] (d) -- (zero);
  \draw[->, black] (pos) -- (min);
  \draw[->, black] (pos) -- (max);
\end{tikzpicture}
$$

> **Worked example.** Classify the critical points of
> $f(x, y) = x^4 + y^4 - 4xy + 1$.
>
> From $f_x = 4x^3 - 4y$ and $f_y = 4y^3 - 4x$, setting both to zero gives
> $y = x^3$ and $x = y^3$, so $x = x^9$, whose real roots are $x = 0, 1, -1$. The
> critical points are $(0, 0)$, $(1, 1)$, and $(-1, -1)$. The second partials are
> $f_{xx} = 12x^2$, $f_{yy} = 12y^2$, $f_{xy} = -4$, so the discriminant is
> $$
> D = 144 x^2 y^2 - 16.
> $$
> Evaluating $D$ and $f_{xx}$ at each point classifies them.

| Point | $D$ | $f_{xx}$ | Conclusion |
| --- | --- | --- | --- |
| $(0, 0)$ | $-16 < 0$ | — | saddle point |
| $(1, 1)$ | $128 > 0$ | $12 > 0$ | local minimum, value $-1$ |
| $(-1, -1)$ | $128 > 0$ | $12 > 0$ | local minimum, value $-1$ |

## Absolute extrema on a closed region

The single-variable Extreme Value Theorem guarantees a max and a min on a closed
interval, found among the critical numbers and the two endpoints. The plane
version replaces the interval by a closed, bounded region and the endpoints by
the boundary curve.

> **Theorem (Extreme Value Theorem).** If $f$ is continuous on a closed, bounded
> set $D \subseteq \mathbb{R}^2$, then $f$ attains an absolute maximum and an
> absolute minimum at points of $D$.

An extreme value sits either at an interior critical point or on the boundary, so
the search is a three-step procedure.

```algorithm
caption: Absolute extrema of continuous $f$ on a closed, bounded region $D$
find every critical point of $f$ in the interior of $D$, and list $f$ there
parametrize each boundary piece, reducing $f$ to one variable
  find that reduced function's extrema, including the corner endpoints
compare all listed values
  the largest is the absolute maximum; the smallest is the absolute minimum
```

The boundary step turns each edge into a single-variable optimization: substitute
the edge's parametrization into $f$, differentiate in the one remaining
parameter, and include the corners where edges meet. The absolute extrema are
then the largest and smallest of every candidate collected.

> **Worked example.** Find the absolute maximum and minimum of
> $f(x, y) = x^2 - 2xy + 2y$ on the rectangle $D = [0, 3] \times [0, 2]$.
>
> **Interior.** $f_x = 2x - 2y$ and $f_y = -2x + 2$ vanish only at $(1, 1)$,
> where $f(1, 1) = 1$.
>
> **Boundary**, edge by edge:
> - $L_1$ ($y = 0$): $f = x^2$ on $[0, 3]$, running from $0$ to $9$.
> - $L_2$ ($x = 3$): $f = 9 - 4y$ on $[0, 2]$, from $9$ down to $1$.
> - $L_3$ ($y = 2$): $f = (x - 2)^2$ on $[0, 3]$, from $0$ up to $4$.
> - $L_4$ ($x = 0$): $f = 2y$ on $[0, 2]$, from $0$ to $4$.
>
> Comparing every candidate, the absolute maximum is $9$ at $(3, 0)$ and the
> absolute minimum is $0$, attained at both $(0, 0)$ and $(2, 2)$.

$$
% caption: The rectangle $D=[0,3]\times[0,2]$: candidates are the interior
% critical point $(1,1)$ and the extremes along the four edges $L_1$ to $L_4$.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\small]
  \definecolor{acc}{HTML}{4A6FA5}
  \definecolor{wrm}{HTML}{C0392B}
  \draw[->, black] (-0.3,0) -- (4.4,0) node[right, font=\footnotesize] {$x$};
  \draw[->, black] (0,-0.3) -- (0,3.1) node[above, font=\footnotesize] {$y$};
  \draw[acc, thick, fill=acc!8] (0,0) rectangle (3.6,2.4);
  % edge labels
  \node[acc, font=\footnotesize, anchor=north] at (1.8,-0.02) {$L_1$};
  \node[acc, font=\footnotesize, anchor=west] at (3.62,1.2) {$L_2$};
  \node[acc, font=\footnotesize, anchor=south] at (1.8,2.42) {$L_3$};
  \node[acc, font=\footnotesize, anchor=east] at (-0.02,1.2) {$L_4$};
  % corners
  \foreach \x/\y in {0/0, 3.6/0, 3.6/2.4, 0/2.4} { \fill[black!70] (\x,\y) circle (1.4pt); }
  % interior critical point (1,1) at scale 1.2
  \fill[wrm] (1.2,1.2) circle (1.9pt);
  \node[wrm, font=\footnotesize, anchor=south west] at (1.26,1.24) {(1, 1)};
  % annotated extrema
  \fill[wrm] (2.4,2.4) circle (1.6pt);
  \node[font=\footnotesize, anchor=south west] at (3.7,0.4) {max 9};
  \node[font=\footnotesize, anchor=north west] at (2.46,2.4) {min 0};
  \node[font=\footnotesize, anchor=north east] at (-0.06,-0.04) {min 0};
\end{tikzpicture}
$$

## Constrained optimization

Boundary optimization is a special case of a broader problem: extremize $f(x, y)$
not over a region but along a curve $g(x, y) = k$. Substitution works when the
constraint solves cleanly for one variable; when it does not, Lagrange's method
does the job without solving the constraint at all.

The reasoning is geometric. To maximize $f$ along $g = k$ is to find the largest
value $c$ for which the level curve $f = c$ still meets the constraint curve
$g = k$. At that extreme value the two curves cannot cross — a crossing would let
$c$ increase further — so they are tangent, sharing a tangent line. Tangent curves
have parallel normals, and the normal to a level curve is its
[gradient](/calculus/partial-derivatives/directional-derivatives-and-the-gradient).
Therefore $\nabla f$ and $\nabla g$ are parallel at the optimum.

$$
% caption: At a constrained optimum the level curve $f=c$ is tangent to the
% constraint $g=k$, so the gradients $\nabla f$ and $\nabla g$ point along the
% same line.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\small]
  \definecolor{acc}{HTML}{4A6FA5}
  \definecolor{red}{HTML}{C0392B}
  % nested level curves of f, centred low; the outer (bold) one is f=c
  \draw[acc!45] (0,-0.9) ellipse (1.0 and 0.55);
  \draw[acc!45] (0,-0.9) ellipse (1.75 and 0.95);
  \draw[acc, thick] (0,-0.9) ellipse (2.5 and 1.35);
  \node[acc, font=\footnotesize, anchor=east] at (-2.55,-0.9) {$f=c$};
  % constraint curve g=k: tangent to the outer ellipse at its top (0,0.45)
  \draw[red, thick] (-3.0,-0.7) .. controls (-1.2,0.45) and (1.2,0.45) .. (3.0,-0.7);
  \node[red, font=\footnotesize, anchor=west] at (2.5,-0.55) {$g=k$};
  % tangency point, central and clear
  \fill[black] (0,0.45) circle (1.8pt);
  % parallel gradient arrows, both normal (vertical) at the tangency
  \draw[->, acc, very thick] (0,0.45) -- (0,1.75);
  \node[acc, font=\footnotesize, anchor=east] at (-0.1,1.5) {grad $f$};
  \draw[->, red, very thick] (0.28,0.45) -- (0.28,1.55);
  \node[red, font=\footnotesize, anchor=west] at (0.38,1.35) {grad $g$};
\end{tikzpicture}
$$

> **Method (Lagrange multipliers).** To find the extrema of $f(x, y, z)$ subject
> to $g(x, y, z) = k$ (assuming they exist and $\nabla g \ne \vec 0$ on the
> constraint):
> $$
> \nabla f = \lambda\, \nabla g,
> \qquad
> g(x, y, z) = k.
> $$
> Solve this system for $x, y, z$, and the scalar $\lambda$ (the **Lagrange
> multiplier**), then evaluate $f$ at each solution point. The largest value is
> the maximum, the smallest the minimum.

In components the vector equation is $f_x = \lambda g_x$, $f_y = \lambda g_y$,
$f_z = \lambda g_z$, giving one equation per variable plus the constraint — as
many equations as unknowns.

> **Worked example.** Extremize $f(x, y) = x^2 + 2y^2$ on the circle
> $x^2 + y^2 = 1$.
>
> With $g = x^2 + y^2$, the conditions $\nabla f = \lambda \nabla g$ are
> $$
> 2x = \lambda (2x),
> \qquad
> 4y = \lambda (2y),
> \qquad
> x^2 + y^2 = 1.
> $$
> The first equation forces $x = 0$ or $\lambda = 1$. If $x = 0$, the constraint
> gives $y = \pm 1$ and $f = 2$. If $\lambda = 1$, the second gives $4y = 2y$, so
> $y = 0$, then $x = \pm 1$ and $f = 1$. The maximum value is $2$ at $(0, \pm 1)$
> and the minimum is $1$ at $(\pm 1, 0)$: the ends of the major and minor axes of
> the ellipse $f = c$ that the circle reaches.

## Two constraints

A point can be pinned to the intersection of two surfaces $g = k$ and $h = c$,
which is generally a curve. At an extremum along that curve, $\nabla f$ lies in
the plane spanned by $\nabla g$ and $\nabla h$ (both normal to the curve), so

$$
\nabla f = \lambda\, \nabla g + \mu\, \nabla h,
\qquad
g = k,
\qquad
h = c.
$$

There are now two multipliers, $\lambda$ and $\mu$, and five equations in the
five unknowns $x, y, z, \lambda, \mu$. The same logic extends the method to any
number of constraints, one multiplier apiece.

$$
% caption: With two constraints the feasible set is the curve $C$ where $g=k$ and
% $h=c$ meet; $\nabla g$ and $\nabla h$ are both normal to $C$, and at an extremum
% $\nabla f$ lies in the plane they span.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\small]
  \definecolor{acc}{HTML}{4A6FA5}
  \definecolor{wrm}{HTML}{C0392B}
  \draw[black, thick] (-2.6,-0.8) .. controls (-0.8,0.2) and (0.8,-0.2) .. (2.6,0.8);
  \node[black, font=\footnotesize, anchor=north] at (2.4,0.5) {curve $C$};
  \coordinate (P) at (0,-0.05);
  \draw[->, acc, very thick] (P) -- (-0.75,1.7);
  \node[acc, font=\footnotesize, anchor=east] at (-1.05,1.45) {grad g};
  \draw[->, wrm, very thick] (P) -- (0.95,1.85);
  \node[wrm, font=\footnotesize, anchor=west] at (0.98,1.7) {grad h};
  \draw[->, black!75, very thick] (P) -- (0.12,2.05);
  \node[black!80, font=\footnotesize, anchor=south] at (0.12,2.06) {grad f};
  \fill[black] (P) circle (1.7pt);
  \node[anchor=north east, font=\footnotesize] at (-0.05,-0.1) {$P$};
\end{tikzpicture}
$$

> **Worked example.** Maximize $f(x, y, z) = x + 2y + 3z$ on the intersection of
> the plane $x - y + z = 1$ and the cylinder $x^2 + y^2 = 1$.
>
> Take $g = x - y + z$ and $h = x^2 + y^2$. With
> $\nabla f = \langle 1, 2, 3 \rangle$, $\nabla g = \langle 1, -1, 1 \rangle$, and
> $\nabla h = \langle 2x, 2y, 0 \rangle$, the condition
> $\nabla f = \lambda \nabla g + \mu \nabla h$ reads
> $$
> 1 = \lambda + 2\mu x, \qquad 2 = -\lambda + 2\mu y, \qquad 3 = \lambda.
> $$
> So $\lambda = 3$, giving $2\mu x = -2$ and $2\mu y = 5$, i.e. $x = -1/\mu$ and
> $y = 5/(2\mu)$. The cylinder constraint $x^2 + y^2 = 1$ forces
> $$
> \frac{1}{\mu^2} + \frac{25}{4\mu^2} = 1
> \;\Longrightarrow\;
> \mu = \pm\frac{\sqrt{29}}{2}.
> $$
> The plane gives $z = 1 - x + y$. Evaluating $f$ at the two solution points
> yields $3 + \sqrt{29}$ and $3 - \sqrt{29}$, so the maximum value is
> $3 + \sqrt{29}$.

The stationarity condition $\nabla f = \lambda \nabla g$ appears across
constrained optimization, from utility maximization in economics to regularized
training objectives. Extrema over regions rather than curves return with
integration, in
[double integrals](/calculus/multiple-integrals-and-vector-calculus/double-integrals).