---
title: Inner Product Spaces
module: Orthogonality and Least Squares
moduleNumber: 6
lessonNumber: 6
order: 606
summary: >
  Promoting the four properties of the dot product to axioms defines an inner
  product on any vector space, including spaces of functions. Length, distance,
  orthogonality, Gram-Schmidt, and best approximation all carry over, along with
  the Cauchy-Schwarz and triangle inequalities and the integral inner product
  behind Fourier approximation.
topics: [Orthogonality and Least Squares]
sources:
  - book: Lay
    ref: "§6.7 Inner Product Spaces; §6.8 Applications of Inner Product Spaces"
---

Every geometric fact so far followed from the four properties of the dot product,
never from the coordinate formula $\sum u_i v_i$. Promoting those properties to
axioms defines an inner product on _any_ vector space, and the apparatus — length,
distance, orthogonality, projection, Gram–Schmidt, best approximation — transfers
unchanged. The spaces can then hold polynomials or continuous functions, and
least-squares approximation becomes function approximation.

## The axioms

> **Definition (Inner product).** An **inner product** on a vector space $V$ is a
> rule assigning to each pair $\mathbf{u}, \mathbf{v} \in V$ a real number
> $\langle \mathbf{u}, \mathbf{v} \rangle$ such that, for all $\mathbf{u},
> \mathbf{v}, \mathbf{w} \in V$ and scalars $c$:
> - $\langle \mathbf{u}, \mathbf{v} \rangle
>   = \langle \mathbf{v}, \mathbf{u} \rangle$;
> - $\langle \mathbf{u} + \mathbf{v}, \mathbf{w} \rangle
>   = \langle \mathbf{u}, \mathbf{w} \rangle + \langle \mathbf{v}, \mathbf{w} \rangle$;
> - $\langle c\mathbf{u}, \mathbf{v} \rangle = c\langle \mathbf{u}, \mathbf{v} \rangle$;
> - $\langle \mathbf{u}, \mathbf{u} \rangle \ge 0$, with equality iff
>   $\mathbf{u} = \mathbf{0}$.
>
> A vector space with an inner product is an **inner product space.**

The list restates the conclusions of the inner-product-properties theorem for the dot product, now taken as
requirements. Any rule satisfying them supports the full geometric vocabulary.

**Weighted inner product on $\mathbb{R}^n$.** Fix positive weights $w_1, \dots,
w_n$ and set $\langle \mathbf{u}, \mathbf{v} \rangle = w_1 u_1 v_1 + \cdots + w_n
u_n v_n$. Each axiom checks directly, so this is an inner product. Larger weights
emphasize the more reliable coordinates, the basis of weighted least squares.

**Evaluation inner product on $P_n$.** Fix distinct reals $t_0, \dots, t_n$. For
polynomials $p, q$ of degree at most $n$, set
$$
\langle p, q \rangle = p(t_0)q(t_0) + p(t_1)q(t_1) + \cdots + p(t_n)q(t_n).
$$
The first three axioms are immediate. For the fourth, $\langle p, p \rangle =
\sum p(t_i)^2 \ge 0$, and it vanishes only if $p$ is zero at $n + 1$ points — which
forces the degree-$\le n$ polynomial $p$ to be identically zero.

## Length, distance, orthogonality

The definitions copy over verbatim, with $\langle\,\cdot\,,\,\cdot\,\rangle$ in
place of the dot product.

- **Length:** $\lVert \mathbf{v} \rVert = \sqrt{\langle \mathbf{v}, \mathbf{v}
  \rangle}$.
- **Distance:** $\operatorname{dist}(\mathbf{u}, \mathbf{v}) = \lVert \mathbf{u} -
  \mathbf{v} \rVert$.
- **Orthogonality:** $\mathbf{u} \perp \mathbf{v}$ means $\langle \mathbf{u},
  \mathbf{v} \rangle = 0$.

Because the axioms match, the earlier theorems hold without change. Gram–Schmidt
still produces orthogonal bases of finite-dimensional subspaces, and the
orthogonal projection onto a subspace $W$ with orthogonal basis $\{v_1, \dots,
v_p\}$ is
$$
\operatorname{proj}_W \mathbf{u}
  = \frac{\langle \mathbf{u}, v_1 \rangle}{\langle v_1, v_1 \rangle}v_1 + \cdots
  + \frac{\langle \mathbf{u}, v_p \rangle}{\langle v_p, v_p \rangle}v_p,
$$
which remains the best approximation to $\mathbf{u}$ by elements of $W$.

### Orthogonal polynomials

> **Worked example.** Let $V = P_4$ with the evaluation inner product at
> $-2, -1, 0, 1, 2$, and view $P_2$ as a subspace. Apply Gram–Schmidt to
> $1, t, t^2$. Here $t$ is already orthogonal to $1$ (the sampled values sum to
> zero), so $p_0 = 1$ and $p_1 = t$. For $p_2$, project $t^2$ onto
> $\operatorname{Span}\{p_0, p_1\}$: with $\langle t^2, 1 \rangle = 10$,
> $\langle 1, 1 \rangle = 5$, and $\langle t^2, t \rangle = 0$,
> $$
> p_2 = t^2 - \tfrac{10}{5}\cdot 1 = t^2 - 2.
> $$
> The set $\{1,\ t,\ t^2 - 2\}$ is an orthogonal basis for $P_2$.

These are the **orthogonal polynomials** used in the statistical trend analysis of
evenly spaced data.

$$
% caption: The orthogonal polynomials for evenly spaced data: the constant p0, the
% linear p1 = t, and the quadratic p2 = t-squared minus 2, over the five sample
% points where their pairwise value-products sum to zero.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (-2.7,0) -- (2.7,0) node[right] {$t$};
\draw[->, black] (0,-2.5) -- (0,2.9);
\draw[black, thick] (-2.3,1.0) -- (2.3,1.0) node[right, black] {$p_0$};
\draw[thick] (-2.1,-2.1) -- (2.1,2.1) node[above right] {$p_1$};
\draw[black, thick] (-2.0,2.0) .. controls (-1.0,-3.3) and (1.0,-3.3) .. (2.0,2.0);
\node[black] at (0,-2.15) {$p_2$};
\foreach \x in {-2,-1,0,1,2} \fill[black!70] (\x,0) circle (1.8pt);
\end{tikzpicture}
$$

## Best approximation of functions

A recurring applied problem is to approximate a function $f$ by a simpler function
$g$ drawn from a subspace $W$. When closeness is measured by an inner product, the
best $g$ is the orthogonal projection of $f$ onto $W$.

$$
% caption: The orthogonal decomposition of v into its projection onto W and a
% perpendicular component makes the Pythagorean relation the hypotenuse-longest
% statement behind the Cauchy-Schwarz inequality.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\draw[thick] (-0.5,0) -- (5.2,0);
\node at (4.9,-0.35) {$W$};
\coordinate (o) at (0.4,0);
\coordinate (pw) at (3.2,0);
\coordinate (v) at (3.2,2.1);
\draw[->, black, very thick] (o) -- (v) node[above right] {$\mathbf{v}$};
\draw[->, acc, very thick] (o) -- (pw) node[below] {$\operatorname{proj}_W\mathbf{v}$};
\draw[black, thick] (pw) -- (v) node[midway, right] {$\mathbf{v}-\operatorname{proj}_W\mathbf{v}$};
\draw[black] (3.2,0.22) -- (2.98,0.22) -- (2.98,0);
\fill[black] (o) circle (1.6pt);
\end{tikzpicture}
$$

> **Worked example.** In $P_4$ with the evaluation inner product at
> $-2, -1, 0, 1, 2$, approximate $p(t) = 5 - \tfrac{1}{2}t^4$ by a quadratic: its
> projection onto $P_2 = \operatorname{Span}\{p_0, p_1, p_2\}$. The three
> coefficients $\langle p, p_i\rangle / \langle p_i, p_i\rangle$ are
> $\tfrac{8}{5}$, $0$, and $-\tfrac{31}{14}$, so
> $$
> \hat{p} = \tfrac{8}{5}p_0 - \tfrac{31}{14}p_2
>   = \tfrac{8}{5} - \tfrac{31}{14}(t^2 - 2),
> $$
> the polynomial in $P_2$ closest to $p$ at the five sample points.

## Two inequalities

The Pythagorean decomposition of the figure, $\lVert \mathbf{v} \rVert^2 = \lVert
\operatorname{proj}_W \mathbf{v} \rVert^2 + \lVert \mathbf{v} -
\operatorname{proj}_W \mathbf{v} \rVert^2$, shows a projection is never longer than
the vector itself. That single observation yields two central inequalities.

> **Theorem (Cauchy–Schwarz inequality).** For all $\mathbf{u}, \mathbf{v} \in
> V$,
> $$
> |\langle \mathbf{u}, \mathbf{v} \rangle| \le \lVert \mathbf{u} \rVert\, \lVert
> \mathbf{v} \rVert.
> $$

If $\mathbf{u} = \mathbf{0}$ both sides vanish. Otherwise let $W =
\operatorname{Span}\{\mathbf{u}\}$. The projection of $\mathbf{v}$ onto $W$ has
length
$$
\lVert \operatorname{proj}_W \mathbf{v} \rVert
  = \frac{|\langle \mathbf{v}, \mathbf{u} \rangle|}{\lVert \mathbf{u} \rVert^2}
    \lVert \mathbf{u} \rVert
  = \frac{|\langle \mathbf{u}, \mathbf{v} \rangle|}{\lVert \mathbf{u} \rVert},
$$
and since $\lVert \operatorname{proj}_W \mathbf{v} \rVert \le \lVert \mathbf{v}
\rVert$, rearranging gives the bound.

> **Theorem (Triangle inequality).** For all $\mathbf{u}, \mathbf{v} \in V$,
> $\lVert \mathbf{u} + \mathbf{v} \rVert \le \lVert \mathbf{u} \rVert + \lVert
> \mathbf{v} \rVert$.

Expand and apply Cauchy–Schwarz:
$$
\lVert \mathbf{u} + \mathbf{v} \rVert^2
  = \lVert \mathbf{u} \rVert^2 + 2\langle \mathbf{u}, \mathbf{v} \rangle
    + \lVert \mathbf{v} \rVert^2
  \le \lVert \mathbf{u} \rVert^2 + 2\lVert \mathbf{u} \rVert\lVert \mathbf{v} \rVert
    + \lVert \mathbf{v} \rVert^2
  = (\lVert \mathbf{u} \rVert + \lVert \mathbf{v} \rVert)^2,
$$
and take square roots.

## An inner product from an integral

The most widely used inner product space of analysis is $C[a, b]$, the continuous
functions on an interval, with an inner product built by letting the evaluation
sum become a Riemann integral.

> **Definition (Integral inner product).** On $C[a, b]$, set
> $$
> \langle f, g \rangle = \int_a^b f(t)\, g(t)\,\d t.
> $$

The axioms follow from elementary properties of the definite integral; positivity
uses that a continuous nonnegative integrand with zero integral must be identically
zero. Length is then $\lVert f \rVert = \sqrt{\int_a^b f(t)^2\,\d t}$, and two
functions are orthogonal when the integral of their product is zero.

$$
% caption: The integral inner product of f and g measures the signed overlap of
% the two curves: where the product f times g is positive it adds area, where
% negative it subtracts, and orthogonality means the areas cancel to zero.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (-0.2,0) -- (5.6,0) node[right] {$t$};
\draw[->, black] (0,-1.3) -- (0,1.7) node[above] {};
\draw[very thick] (0,0.2) .. controls (1.6,1.6) and (3.6,1.4) .. (5.2,0.3)
  node[right] {$f$};
\draw[black, very thick] (0,-0.9) .. controls (1.6,0.9) and (3.6,-0.6) .. (5.2,0.9)
  node[right, black] {$g$};
\node[black] at (0,-0.3) {$a$};
\node[black] at (5.2,-0.3) {$b$};
\end{tikzpicture}
$$

## Fourier approximation

The integral inner product on $C[0, 2\pi]$ makes the trigonometric functions
orthogonal: for positive integers $m \ne n$,
$$
\langle \cos mt, \cos nt \rangle = \int_0^{2\pi} \cos mt\, \cos nt\,\d t = 0,
$$
and the same holds for the sine pairs and every sine-cosine pair. So
$$
\{\,1,\ \cos t,\ \cos 2t,\ \dots,\ \cos nt,\ \sin t,\ \sin 2t,\ \dots,\ \sin nt\,\}
$$
is an orthogonal set. Let $W$ be its span. The best approximation to a function
$f$ by elements of $W$ is the projection onto $W$, called the **$n$th-order
Fourier approximation.**

> **Definition (Fourier coefficients).** The projection of $f$ onto $W$ is
> $\tfrac{a_0}{2} + \sum_{k=1}^{n}(a_k \cos kt + b_k \sin kt)$ with
> $$
> a_k = \frac{1}{\pi}\int_0^{2\pi} f(t)\cos kt\,\d t,
> \qquad
> b_k = \frac{1}{\pi}\int_0^{2\pi} f(t)\sin kt\,\d t.
> $$

Each coefficient is the projection weight $\langle f, \cos kt\rangle / \langle
\cos kt, \cos kt\rangle$, using $\lVert \cos kt \rVert^2 = \lVert \sin kt \rVert^2 =
\pi$. The constant term is written $a_0/2$ so that the same formula for $a_k$ works
at $k = 0$.

> **Worked example.** Expand $f(t) = t$ on $[0, 2\pi]$. Integration by parts gives
> $a_k = 0$ for $k \ge 1$ and $b_k = -2/k$, with constant term $a_0/2 = \pi$, so the
> $n$th-order Fourier approximation is
> $$
> \pi - 2\sin t - \sin 2t - \tfrac{2}{3}\sin 3t - \cdots - \tfrac{2}{n}\sin nt.
> $$
> As $n$ grows the mean square error $\lVert f - (\text{approximation}) \rVert$
> tends to zero, and the limit is the Fourier series of $f$.

$$
% caption: The Fourier approximation of f(t) = t on the interval from 0 to 2 pi: a
% partial sum of a few sine terms already tracks the line, oscillating around it and
% tightening as more terms are added.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (-0.2,0) -- (6.3,0) node[right] {$t$};
\draw[->, black] (0,-0.3) -- (0,4.0) node[above] {$y$};
\draw[black, thick] (0,0) -- (5.6,3.5) node[above left, black] {$f(t)=t$};
\draw[acc, very thick] (0.2,0.7) .. controls (0.7,-0.1) and (1.2,1.3) .. (1.7,0.95)
  .. controls (2.3,0.55) and (2.9,2.2) .. (3.5,1.95)
  .. controls (4.1,1.7) and (4.7,3.3) .. (5.4,2.9);
\node[acc] at (1.7,2.4) {partial sum};
\end{tikzpicture}
$$

## Summary

| Setting | Inner product | Orthogonality means |
| --- | --- | --- |
| $\mathbb{R}^n$ weighted | $\sum w_i u_i v_i$ | weighted dot product zero |
| $P_n$ evaluation | $\sum p(t_i)q(t_i)$ | products at the sample points sum to zero |
| $C[a,b]$ integral | $\int_a^b fg\,\d t$ | product integrates to zero |

One set of axioms carries the geometry of $\mathbb{R}^n$ into spaces of
polynomials and functions. Cauchy–Schwarz and the triangle inequality hold in all
of them, and least-squares approximation becomes best approximation by projection:
orthogonal polynomials for trend analysis, Fourier series for signals.[^lay-67][^lay-68]

[^lay-67]: Lay, §6.7 — Inner Product Spaces: the inner-product axioms, weighted and evaluation inner products, length and orthogonality in general spaces, Theorem 16 (Cauchy–Schwarz), Theorem 17 (triangle inequality), and the integral inner product on $C[a,b]$.
[^lay-68]: Lay, §6.8 — Applications of Inner Product Spaces: weighted least squares, trend analysis with orthogonal polynomials, and Fourier approximation with the Fourier coefficients on $C[0, 2\pi]$.