Optimization and Lagrange Multipliers

The high and low points of a surface are the two-variable version of the maxima and minima of single-variable calculus. Finding them repeats the familiar structure (locate where the derivative vanishes, then classify), but with two new features: the derivative is now a pair of partials, and a critical point can be neither a max nor a min but a saddle, rising in one direction and falling in another.

Local extrema and critical points

At a smooth local extremum the surface must be flat, because each axis-trace has an ordinary extremum there.

The proof fixes one variable and applies single-variable Fermat to the trace. Geometrically, $\nabla f = 0$ makes the tangent plane horizontal.

The converse fails: a critical point need not be an extremum. The standard counterexample is the saddle $f (x, y) = y^{2} - x^{2}$ at the origin, where $\nabla f = ⟨ - 2 x, 2 y ⟩ = 0$ , yet $f$ rises along the $y$ -axis and falls along the $x$ -axis, so the origin is neither a peak nor a valley.

The saddle surface

z = y^{2} - x^{2}

. Slices with

x

fixed are upward parabolas, slices with

y

fixed are downward parabolas, and the origin sits flat between them: a critical point that is no extremum.

Flattened onto one plane, the two principal cross-sections give the sign pattern that the Second Derivatives Test below detects algebraically:

A saddle point: the surface

z = y^{2} - x^{2}

rises along one axis and falls along the other, so

\nabla f = 0

without a local extremum.

The Second Derivatives Test

Which kind of critical point is at hand is decided by the second partials, combined into a discriminant.

The discriminant is the determinant of the matrix of second partials (the Hessian). Its sign captures whether the surface curves the same way in every direction ( $D > 0$ , a bowl or dome) or opposite ways along different axes ( $D < 0$ , a saddle). When $D > 0$ the sign of $f_{xx}$ then says which.

The Second Derivatives Test as a decision on

(D, f_{xx})

D > 0

gives an extremum whose type

f_{xx}

fixes;

D < 0

gives a saddle;

D = 0

is undecided.

Point	$D$	$f_{xx}$	Conclusion
$(0, 0)$	$- 16 < 0$	—	saddle point
$(1, 1)$	$128 > 0$	$12 > 0$	local minimum, value $- 1$
$(- 1, - 1)$	$128 > 0$	$12 > 0$	local minimum, value $- 1$

Absolute extrema on a closed region

The single-variable Extreme Value Theorem guarantees a max and a min on a closed interval, found among the critical numbers and the two endpoints. The plane version replaces the interval by a closed, bounded region and the endpoints by the boundary curve.

An extreme value sits either at an interior critical point or on the boundary, so the search is a three-step procedure.

Algorithm:Absolute extrema of continuous

f

on a closed, bounded region

D

1
find every critical point of $f$ in the interior of $D$ , and list $f$ there
2
parametrize each boundary piece, reducing $f$ to one variable
3
find that reduced function's extrema, including the corner endpoints
4
compare all listed values
5
the largest is the absolute maximum; the smallest is the absolute minimum

The boundary step turns each edge into a single-variable optimization: substitute the edge's parametrization into $f$ , differentiate in the one remaining parameter, and include the corners where edges meet. The absolute extrema are then the largest and smallest of every candidate collected.

The rectangle

D = [0, 3] \times [0, 2]

: candidates are the interior critical point

(1, 1)

and the extremes along the four edges

L_{1}

L_{4}

Constrained optimization

Boundary optimization is a special case of a broader problem: extremize $f (x, y)$ not over a region but along a curve $g (x, y) = k$ . Substitution works when the constraint solves cleanly for one variable; when it does not, Lagrange's method does the job without solving the constraint at all.

The reasoning is geometric. To maximize $f$ along $g = k$ is to find the largest value $c$ for which the level curve $f = c$ still meets the constraint curve $g = k$ . At that extreme value the two curves cannot cross — a crossing would let $c$ increase further — so they are tangent, sharing a tangent line. Tangent curves have parallel normals, and the normal to a level curve is its gradient. Therefore $\nabla f$ and $\nabla g$ are parallel at the optimum.

At a constrained optimum the level curve

f = c

is tangent to the constraint

g = k

, so the gradients

\nabla f

and

\nabla g

point along the same line.

In components the vector equation is $f_{x} = λ g_{x}$ , $f_{y} = λ g_{y}$ , $f_{z} = λ g_{z}$ , giving one equation per variable plus the constraint — as many equations as unknowns.

Two constraints

A point can be pinned to the intersection of two surfaces $g = k$ and $h = c$ , which is generally a curve. At an extremum along that curve, $\nabla f$ lies in the plane spanned by $\nabla g$ and $\nabla h$ (both normal to the curve), so

\nabla f = λ \nabla g + μ \nabla h, g = k, h = c .

There are now two multipliers, $λ$ and $μ$ , and five equations in the five unknowns $x, y, z, λ, μ$ . The same logic extends the method to any number of constraints, one multiplier apiece.

With two constraints the feasible set is the curve

C

where

g = k

and

h = c

meet;

\nabla g

and

\nabla h

are both normal to

C

, and at an extremum

\nabla f

lies in the plane they span.

The stationarity condition $\nabla f = λ \nabla g$ appears across constrained optimization, from utility maximization in economics to regularized training objectives. Extrema over regions rather than curves return with integration, in double integrals.