Higher Derivatives, Taylor's Theorem, and Extrema

The derivative$Df:A\to L({\mathbb{R}}^n,{\mathbb{R}}^m)$ is itself a map, from points to linear transformations, and it can be differentiated again. Doing so once produces the second derivative that carries the curvature of $f$; doing so repeatedly produces the Taylor expansion. For a scalar field the second derivative is a symmetric matrix, the Hessian, and its definiteness generalizes the sign condition of the one-variable second-derivative test.

The second derivative as a bilinear map

If $f:A\subseteq {\mathbb{R}}^n\to {\mathbb{R}}^m$ is differentiable, then $Df:A\to L({\mathbb{R}}^n,{\mathbb{R}}^m)$, where $L({\mathbb{R}}^n,{\mathbb{R}}^m)$ is the space of linear maps (identified with $m\times n$ matrices). If this map is itself differentiable at $x_0$, its derivative $D^{2}f(x_0)=D(Df)(x_0)$ sends a vector $u$ to a linear map, so that $(D^{2}f(x_0)u)v$ lies in ${\mathbb{R}}^m$ for each pair $u,v$. It is convenient to read $D^{2}f(x_0)$ as a bilinear map

linear in each argument separately. When $f$ is scalar-valued, the bilinear map is represented by a single $n\times n$ matrix.

The proof is immediate: $Df=(\partial f/\partial x_1,\dots ,\partial f/\partial x_n)$, and applying the Jacobian theorem from the first lesson to $Df$ produces the matrix of second partials. Higher derivatives continue the pattern — $D^{3}f(x_0)$ is a trilinear map with components ${\partial }^{3}f/\partial x_i\partial x_j\partial x_k$ — but the matrix picture stops at the second derivative of a scalar field.

Equality of mixed partials

The Hessian written above has, in general, an ordering ambiguity: ${\partial }^{2}f/\partial x_i\partial x_j$ differentiates first in $x_j$ then in $x_i$, and swapping the order might change the answer. Under continuity of the second partials, it does not.

Smoothness classes. A map whose first $k$ derivatives exist and are continuous on $A$ is of class $C^k(A)$; if it is $C^k$ for every $k$ it is smooth, or $C^{\infty }$. Symmetry of all higher derivatives follows by induction. The classes nest, $C^{\infty }\subset \cdots \subset C^2\subset C^1\subset C^0$, and the Hessian test below lives in $C^2$.

Taylor's theorem in several variables

Taylor's theorem expands $f$ around a point as a polynomial in the displacement plus a remainder controlled by the next derivative. The one-variable statement generalizes with the $l$-th derivative acting as an $l$-linear form on $l$ copies of the displacement.

The remainder has a Lagrange integral form,

obtained by repeated integration by parts starting from the fundamental theorem of calculus applied to $t\mapsto f(x+th)$. The case $k=1$ is the mean value theorem; the case $k=2$ is the quadratic expansion the extremum test uses.

Real analytic functions. Letting $k\to \infty$ gives the Taylor series ${\sum }_l\frac{1}{l!}{D}^{l}f(x_0)(x-x_0,\dots )$. When it converges to $f$ near $x_0$, $f$ is real analytic there — equivalent to the remainder tending to $0$. Convergence is not automatic even for smooth functions.

Critical points and extrema

Taylor's second-order expansion turns the search for local maxima and minima into linear algebra. First, the first-order condition.

A critical point need not be extreme. In one variable $f(x)=x^3$ has $f^{\prime }(0)=0$ but no extremum; in two variables $f(x,y)=y^2-x^2$ has $Df=(-2x,2y)$ vanishing at the origin, yet every neighborhood of $(0,0)$ contains points where $f>0$ and points where $f<0$. Such a critical point is a saddle point.

The Hessian test. The second-order term of Taylor's expansion at a critical point is $\frac{1}{2}{D}^{2}f(x_0)(h,h)$, and its sign for all small $h$ decides the classification.

The gap between definite (sufficient) and semidefinite (necessary) is real: $f(x,y)=x^2$ has a minimum along the $y=0$ line where the Hessian is only semidefinite, since it has zero curvature in the $y$-direction.

The $2\times 2$ criterion. For $f:{\mathbb{R}}^2\to \mathbb{R}$ the Hessian is $\left(\begin{array}{cc}a& b\\ b& d\end{array}\right)$ with $a=f_{xx}$, $b=f_{xy}$, $d=f_{yy}$, and definiteness reduces to two scalar checks.

• Positive definite iff $a>0$ and $ad-b^2>0$: local minimum.
• Negative definite iff $a<0$ and $ad-b^2>0$: local maximum.
• $ad-b^2<0$: indefinite, a saddle.

The inconclusive row is where the second-order expansion runs out: the quadratic form has a flat direction, and deciding the extremum needs higher-order terms or a direct argument. Local invertibility of $f$ itself is governed not by definiteness of the Hessian but by nonsingularity of the Jacobian, through the inverse and implicit function theorems.¹