Orthogonal Sets and Orthogonal Projections

A basis lets us write any vector as a unique linear combination, but finding the weights normally means solving a linear system. When the basis vectors are mutually perpendicular, the work collapses: each weight is one dot product. The same orthogonality lets us drop a perpendicular from a point onto a subspace and name the closest point in it: the projection, which underlies the least-squares method.

Orthogonal sets

For ${\mathbf{u}}_1=(3,1,1)$, ${\mathbf{u}}_2=(-1,2,1)$, and ${\mathbf{u}}_3=(-\frac{1}{2},-2,\frac{7}{2})$, all three dot products vanish, so the set is orthogonal; the three segments are mutually perpendicular. Orthogonality is a strong enough condition to guarantee independence.

Suppose $c_1{\mathbf{u}}_1+\cdots +c_p{\mathbf{u}}_p=0$. Dotting both sides with ${\mathbf{u}}_1$ and using orthogonality kills every term except the first,

and ${\mathbf{u}}_1\cdot {\mathbf{u}}_1\ne 0$ forces $c_1=0$. Repeating for each ${\mathbf{u}}_j$ makes all weights zero.

An orthogonal basis gives a direct formula for the coordinates.

Dotting $\mathbf{y}=\sum c_i{\mathbf{u}}_i$ with ${\mathbf{u}}_j$ leaves only the $j$-th term, $\mathbf{y}\cdot {\mathbf{u}}_j=c_j({\mathbf{u}}_j\cdot {\mathbf{u}}_j)$, and solving gives $c_j$.

Projection onto a line

Given a nonzero $\mathbf{u}$, split any $\mathbf{y}$ into a piece along $\mathbf{u}$ and a piece perpendicular to it: $\mathbf{y}=\hat{\mathbf{y}}+\mathbf{z}$, where $\hat{\mathbf{y}}=\alpha \mathbf{u}$ and $\mathbf{z}\perp \mathbf{u}$. The perpendicularity condition $(\mathbf{y}-\alpha \mathbf{u})\cdot \mathbf{u}=0$ solves to $\alpha =(\mathbf{y}\cdot \mathbf{u})/(\mathbf{u}\cdot \mathbf{u})$.

The projection depends only on the line $L$, not on which nonzero vector on it is used: replacing $\mathbf{u}$ by $c\mathbf{u}$ leaves the ratio unchanged.

Orthonormal sets and matrices

The standard basis $\{{\mathbf{e}}_1,\dots ,{\mathbf{e}}_n\}$ is orthonormal. Normalizing every vector of an orthogonal set produces an orthonormal one. Collecting orthonormal vectors as columns gives a matrix with a clean identity.

The $(i,j)$ entry of $U^{\top }U$ is ${\mathbf{u}}_i^{\top }{\mathbf{u}}_j$, which is $1$ when $i=j$ (unit length) and $0$ otherwise (orthogonality) exactly when the columns are orthonormal. Such a matrix defines a length- and angle-preserving map.

A square matrix with orthonormal columns is an orthogonal matrix: it satisfies $U^{-1}=U^{\top }$, and its rows are orthonormal as well. Orthogonal matrices represent rigid motions — rotations and reflections — and reappear throughout the diagonalization of symmetric matrices.

Projection onto a subspace

The orthogonal-basis coordinate theorem already writes a vector in $W$ as a sum of one-dimensional projections onto the axes of an orthogonal basis. The same formula projects a vector that is not in $W$ onto it.

Define $\hat{\mathbf{y}}$ by the formula. It lies in $W$ as a combination of the basis. Its residual $\mathbf{z}=\mathbf{y}-\hat{\mathbf{y}}$ is orthogonal to each ${\mathbf{u}}_j$: dotting with ${\mathbf{u}}_j$ leaves $\mathbf{y}\cdot {\mathbf{u}}_j-\frac{\mathbf{y}\cdot {\mathbf{u}}_j}{{\mathbf{u}}_j\cdot {\mathbf{u}}_j}({\mathbf{u}}_j\cdot {\mathbf{u}}_j)=0$, so $\mathbf{z}\in W^{\perp }$. Uniqueness follows because if $\mathbf{y}={\hat{\mathbf{y}}}_1+{\mathbf{z}}_1$ is another such split, then $\hat{\mathbf{y}}-{\hat{\mathbf{y}}}_1={\mathbf{z}}_1-\mathbf{z}$ lies in both $W$ and $W^{\perp }$, hence is orthogonal to itself and zero.

The projection $\hat{\mathbf{y}}$ is the sum of one-dimensional projections onto mutually orthogonal axes. In ${\mathbb{R}}^3$ with $W=\mathrm{Span}\{{\mathbf{u}}_1,{\mathbf{u}}_2\}$, the two component projections along ${\mathbf{u}}_1$ and ${\mathbf{u}}_2$ add to the perpendicular foot $\hat{\mathbf{y}}$ of $\mathbf{y}$ in the plane.

Best approximation

The projection is the nearest point in $W$ to $\mathbf{y}$.

For any $\mathbf{v}\in W$, write $\mathbf{y}-\mathbf{v}=(\mathbf{y}-\hat{\mathbf{y}})+(\hat{\mathbf{y}}-\mathbf{v})$. The first term is in $W^{\perp }$ and the second is in $W$, so they are orthogonal, and the Pythagorean theorem gives

The extra term is positive unless $\mathbf{v}=\hat{\mathbf{y}}$, so $\hat{\mathbf{y}}$ strictly minimizes the distance. The value $\Vert \mathbf{y}-\hat{\mathbf{y}}\Vert$ is the distance from $\mathbf{y}$ to $W$, and $\hat{\mathbf{y}}$ is the best approximation to $\mathbf{y}$ by elements of $W$. This minimizing property is what the least-squares method relies on.

The matrix form of a projection

When the basis is orthonormal, the projection formula simplifies, and it collapses into a matrix product.

Because the columns are unit vectors, the denominators ${\mathbf{u}}_j\cdot {\mathbf{u}}_j$ are $1$. The weights $\mathbf{y}\cdot {\mathbf{u}}_j={\mathbf{u}}_j^{\top }\mathbf{y}$ are the entries of $U^{\top }\mathbf{y}$, so ${\mathrm{proj}}_W\mathbf{y}$ is $U$ times those weights, namely $U{U}^{\top }\mathbf{y}$. Two extremes bracket the formula:

• If $U$ is $n\times p$ with orthonormal columns, $U^{\top }U=I_p$ (characterization of orthonormal columns), while $U{U}^{\top }$ is the projection onto $\mathrm{Col}U$ — generally not the identity.
• If $U$ is square ($p=n$), the column space is all of ${\mathbb{R}}^n$, the projection is the identity, and $U{U}^{\top }=I$.

For hand computation the orthogonal formula of the orthogonal decomposition theorem is preferred, since the orthonormal columns usually carry square roots; the matrix form is the theoretical statement that projection is a linear map with matrix $U{U}^{\top }$.

Decomposing a vector

Summary

Orthogonal bases turn coordinates and projections into dot products, and the best-approximation theorem certifies the projection as the nearest point in the subspace. Building an orthogonal basis from an arbitrary one is the Gram–Schmidt process, and using projection to solve inconsistent systems is the method of least-squares problems.¹