Directional Derivatives and the Gradient

The partials $f_{x}$ and $f_{y}$ report the rate of change of $f$ along the $x$ - and $y$ -axes. Standing on a hillside, those are the slopes due east and due north. The rate of change in an arbitrary compass direction is the directional derivative, and collecting the two partials into one vector (the gradient) gives every directional slope at once, including the steepest.

The directional derivative

Let $\overset{u}{^} = ⟨ a, b ⟩$ be a unit vector. Stepping from $(x_{0}, y_{0})$ a distance $h$ in the direction $\overset{u}{^}$ lands at $(x_{0} + ha, y_{0} + hb)$ . The rate of change of $f$ per unit distance along that direction is the limit of the difference quotient.

The two partials are the special cases along the axes: $\overset{u}{^} = ⟨ 1, 0 ⟩$ gives $D_{\overset{u}{^}} f = f_{x}$ , and $\overset{u}{^} = ⟨ 0, 1 ⟩$ gives $D_{\overset{u}{^}} f = f_{y}$ . Geometrically, $D_{\overset{u}{^}} f$ is the slope of the surface $z = f (x, y)$ in the vertical plane through $P$ carrying the direction $\overset{u}{^}$ , the slope of the tangent to the curve where that plane meets the surface.

The directional derivative is the slope of the surface in the vertical plane through

P

pointing along the unit vector

u

Computing that limit directly is unnecessary. For a differentiable function it reduces to the partials.

The proof applies the chain rule to $g (h) = f (x_{0} + ha, y_{0} + hb)$ : $g^{'} (0) = f_{x} a + f_{y} b$ . Writing $\overset{u}{^}$ by its angle, $\overset{u}{^} = ⟨ cos θ, sin θ ⟩$ , the formula is $D_{\overset{u}{^}} f = f_{x} cos θ + f_{y} sin θ$ .

The gradient vector

The formula $D_{\overset{u}{^}} f = f_{x} a + f_{y} b$ is a dot product of $\overset{u}{^}$ with the vector of partials. That vector has a name.

With this, the directional derivative is a single dot product,

D_{\overset{u}{^}} f = \nabla f \cdot \overset{u}{^},

the scalar projection of the gradient onto $\overset{u}{^}$ . Every slope in every direction is read off the one vector $\nabla f$ .

Steepest ascent

Because $D_{\overset{u}{^}} f = \nabla f \cdot \overset{u}{^} = ∣\nabla f ∣ ∣ \overset{u}{^} ∣ cos ϕ = ∣\nabla f ∣ cos ϕ$ , where $ϕ$ is the angle between $\nabla f$ and $\overset{u}{^}$ , the directional derivative is largest when $cos ϕ = 1$ , that is when $\overset{u}{^}$ points along $\nabla f$ .

Three facts, one for each value of $cos ϕ$ :

Uphill fastest. $f$ increases most rapidly in the direction of $\nabla f$ , at rate $∣\nabla f ∣$ .
Downhill fastest. $f$ decreases most rapidly in the direction of $- \nabla f$ , at rate $∣\nabla f ∣$ .
Level. $f$ is momentarily unchanging in the directions perpendicular to $\nabla f$ , the directions along the level curve.

The directional derivative

D_{u} f = ∣\nabla f ∣ cos ϕ

is greatest along the gradient, zero across it, and least in the opposite direction;

ϕ

is the angle from

u

\nabla f

Gradient perpendicular to level curves

The last bullet has a geometric consequence. Along a level curve $f (x, y) = k$ the value of $f$ is constant, so its rate of change in the tangent direction of the curve is zero. But that rate is $D_{\overset{u}{^}} f = \nabla f \cdot \overset{u}{^}$ , so $\nabla f$ is orthogonal to the curve's tangent: the gradient meets every level curve at a right angle, pointing toward higher values.

At each point the gradient is perpendicular to the level curve through that point and aims toward larger

k

; the closer the curves, the longer the gradient.

Two objects follow from this.

Gradient field. Plotting $\nabla f (a, b)$ as an arrow based at each point $(a, b)$ produces the gradient vector field, laid over the contour map. Every arrow crosses its contour perpendicularly and points uphill; arrows are long where contours crowd (steep ground) and short where they spread (flat ground).

The gradient field over a contour map. Arrows cross contours at right angles, point toward higher ground, and lengthen where the slope is steep.

Steepest-ascent path. A curve that follows the gradient at every point, always turning to head straight uphill, cuts across the contours at right angles everywhere. This is the path of steepest ascent, and the same idea run downhill, following $- \nabla f$ , is the geometry behind gradient-descent optimization.

Level surfaces and tangent planes

For three variables the picture rises one dimension. If a surface $S$ is a level surface $F (x, y, z) = k$ , take any curve $r (t)$ lying on $S$ through $P = (x_{0}, y_{0}, z_{0})$ . Then $F (r (t)) = k$ is constant, and differentiating by the chain rule,

\nabla F (x_{0}, y_{0}, z_{0}) \cdot r^{'} (t_{0}) = 0.

Since $r^{'} (t_{0})$ is a tangent vector to an arbitrary curve on $S$ , the gradient $\nabla F (x_{0}, y_{0}, z_{0})$ is perpendicular to every tangent direction at $P$ : it is a normal vector to the surface.

On a level surface

F = k

, the gradient

\nabla F (P)

is normal to every curve through

P

, so it is normal to the tangent plane; each curve tangent lies in that plane.

This subsumes the earlier tangent plane. A graph $z = f (x, y)$ is the level surface $F (x, y, z) = f (x, y) - z = 0$ , whose gradient is $⟨ f_{x}, f_{y}, - 1 ⟩$ ; the normal equation reduces to $z - z_{0} = f_{x} (x_{0}, y_{0}) (x - x_{0}) + f_{y} (x_{0}, y_{0}) (y - y_{0})$ , the tangent-plane formula for a graph.

The perpendicularity of the gradient to level sets is what makes constrained optimization tractable: at a constrained optimum the gradient of the objective and the gradient of the constraint are parallel, because both are normal to the same tangent direction.