---
title: Orthogonal Sets and Orthogonal Projections
module: Orthogonality and Least Squares
moduleNumber: 6
lessonNumber: 2
order: 602
summary: >
  An orthogonal basis makes coordinates trivial: each weight is a single dot
  product, no linear system required. Orthogonal and orthonormal bases give a
  direct projection formula onto a line and onto a subspace, the orthogonal
  decomposition and best-approximation theorems, and the matrix form U U-transpose
  of a projection.
topics: [Orthogonality and Least Squares]
sources:
  - book: Lay
    ref: "§6.2 Orthogonal Sets; §6.3 Orthogonal Projections"
---

A basis lets us write any vector as a unique linear combination, but finding the
weights normally means solving a linear system. When the basis vectors are
mutually perpendicular, the work collapses: each weight is one dot product. The
same orthogonality lets us drop a perpendicular from a point onto a subspace and
name the closest point in it: the projection, which underlies the least-squares
method.

## Orthogonal sets

> **Definition (Orthogonal set).** A set $\{\mathbf{u}_1, \dots, \mathbf{u}_p\}$
> in $\mathbb{R}^n$ is an **orthogonal set** if every pair of distinct vectors is
> orthogonal: $\mathbf{u}_i \cdot \mathbf{u}_j = 0$ whenever $i \ne j$.

For $\mathbf{u}_1 = (3, 1, 1)$, $\mathbf{u}_2 = (-1, 2, 1)$, and
$\mathbf{u}_3 = (-\tfrac{1}{2}, -2, \tfrac{7}{2})$, all three dot products
vanish, so the set is orthogonal; the three segments are mutually perpendicular.
Orthogonality is a strong enough condition to guarantee independence.

> **Theorem (Orthogonal sets are linearly independent).** An orthogonal set $S = \{\mathbf{u}_1, \dots, \mathbf{u}_p\}$ of
> nonzero vectors in $\mathbb{R}^n$ is linearly independent, and hence a basis for
> the subspace it spans.

Suppose $c_1 \mathbf{u}_1 + \cdots + c_p \mathbf{u}_p = \mathbf{0}$. Dotting both
sides with $\mathbf{u}_1$ and using orthogonality kills every term except the
first,
$$
0 = c_1 (\mathbf{u}_1 \cdot \mathbf{u}_1),
$$
and $\mathbf{u}_1 \cdot \mathbf{u}_1 \ne 0$ forces $c_1 = 0$. Repeating for each
$\mathbf{u}_j$ makes all weights zero.

> **Definition (Orthogonal basis).** An **orthogonal basis** for a subspace $W$
> is a basis that is also an orthogonal set.

An orthogonal basis gives a direct formula for the coordinates.

> **Theorem (Coordinates in an orthogonal basis).** Let
> $\{\mathbf{u}_1, \dots, \mathbf{u}_p\}$ be an orthogonal basis for $W$. Then any
> $\mathbf{y} \in W$ has $\mathbf{y} = c_1 \mathbf{u}_1 + \cdots + c_p
> \mathbf{u}_p$ with weights
> $$
> c_j = \frac{\mathbf{y} \cdot \mathbf{u}_j}{\mathbf{u}_j \cdot \mathbf{u}_j},
> \qquad j = 1, \dots, p.
> $$

Dotting $\mathbf{y} = \sum c_i \mathbf{u}_i$ with $\mathbf{u}_j$ leaves only the
$j$-th term, $\mathbf{y} \cdot \mathbf{u}_j = c_j (\mathbf{u}_j \cdot
\mathbf{u}_j)$, and solving gives $c_j$.

> **Worked example.** Write $\mathbf{y} = (6, 1, -8)$ in the orthogonal basis
> $\mathbf{u}_1 = (3, 1, 1)$, $\mathbf{u}_2 = (-1, 2, 1)$,
> $\mathbf{u}_3 = (-\tfrac{1}{2}, -2, \tfrac{7}{2})$. Each weight is one quotient
> of dot products:
> $$
> \frac{\mathbf{y}\cdot\mathbf{u}_1}{\mathbf{u}_1\cdot\mathbf{u}_1} = \frac{11}{11} = 1,
> \quad
> \frac{\mathbf{y}\cdot\mathbf{u}_2}{\mathbf{u}_2\cdot\mathbf{u}_2} = \frac{-12}{6} = -2,
> \quad
> \frac{\mathbf{y}\cdot\mathbf{u}_3}{\mathbf{u}_3\cdot\mathbf{u}_3} = \frac{-33}{33/2} = -2,
> $$
> so $\mathbf{y} = \mathbf{u}_1 - 2\mathbf{u}_2 - 2\mathbf{u}_3$. A non-orthogonal
> basis would require solving a $3 \times 3$ system.

## Projection onto a line

Given a nonzero $\mathbf{u}$, split any $\mathbf{y}$ into a piece along
$\mathbf{u}$ and a piece perpendicular to it: $\mathbf{y} = \hat{\mathbf{y}} +
\mathbf{z}$, where $\hat{\mathbf{y}} = \alpha \mathbf{u}$ and
$\mathbf{z} \perp \mathbf{u}$. The perpendicularity condition
$(\mathbf{y} - \alpha\mathbf{u}) \cdot \mathbf{u} = 0$ solves to
$\alpha = (\mathbf{y} \cdot \mathbf{u}) / (\mathbf{u} \cdot \mathbf{u})$.

> **Definition (Projection onto a line).** For nonzero $\mathbf{u}$ spanning the
> line $L$, the **orthogonal projection** of $\mathbf{y}$ onto $L$ is
> $$
> \hat{\mathbf{y}} = \operatorname{proj}_L \mathbf{y}
>   = \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\, \mathbf{u},
> $$
> and $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$ is the **component of
> $\mathbf{y}$ orthogonal to $\mathbf{u}$.**

The projection depends only on the line $L$, not on which nonzero vector on it is
used: replacing $\mathbf{u}$ by $c\mathbf{u}$ leaves the ratio unchanged.

> **Worked example.** Project $\mathbf{y} = (7, 6)$ onto the line through
> $\mathbf{u} = (4, 2)$. With $\mathbf{y}\cdot\mathbf{u} = 40$ and
> $\mathbf{u}\cdot\mathbf{u} = 20$,
> $$
> \hat{\mathbf{y}} = \frac{40}{20}\mathbf{u} = (8, 4),
> \qquad
> \mathbf{y} - \hat{\mathbf{y}} = (-1, 2).
> $$
> The check $\hat{\mathbf{y}} \cdot (\mathbf{y} - \hat{\mathbf{y}}) = -8 + 8 = 0$
> confirms the split is orthogonal, and the distance from $\mathbf{y}$ to $L$ is
> $\lVert \mathbf{y} - \hat{\mathbf{y}} \rVert = \sqrt{1 + 4} = \sqrt{5}$.

$$
% caption: The projection of y onto the line L drops a perpendicular foot; the
% projection y-hat lies on L and the residual y minus y-hat is orthogonal to it,
% giving the shortest segment from y to L.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (-0.4,0) -- (5.4,0) node[right] {$x_1$};
\draw[->, black] (0,-0.4) -- (0,3.4) node[above] {$x_2$};
\draw[thick] (0,0) -- (5.0,2.5) node[right] {$L=\operatorname{Span}\{\mathbf{u}\}$};
\coordinate (y) at (3.5,3.0);
\coordinate (yh) at (4.0,2.0);
\draw[->, black, very thick] (0,0) -- (y) node[above left] {$\mathbf{y}$};
\draw[->, acc, very thick] (0,0) -- (yh) node[below right] {$\hat{\mathbf{y}}$};
\draw[black, thick] (y) -- (yh) node[midway, right] {$\mathbf{y}-\hat{\mathbf{y}}$};
\draw[->, thick] (0,0) -- (2.0,1.0) node[below right] {$\mathbf{u}$};
\draw[black] (3.82,1.9) -- (3.66,2.22) -- (3.5,2.14);
\end{tikzpicture}
$$

## Orthonormal sets and matrices

> **Definition (Orthonormal set).** An **orthonormal set** is an orthogonal set
> of unit vectors. An **orthonormal basis** for $W$ is an orthonormal set that
> spans $W$.

The standard basis $\{\mathbf{e}_1, \dots, \mathbf{e}_n\}$ is orthonormal.
Normalizing every vector of an orthogonal set produces an orthonormal one.
Collecting orthonormal vectors as columns gives a matrix with a clean identity.

> **Theorem (Characterization of orthonormal columns).** An $m \times n$ matrix $U$ has orthonormal columns if and only if
> $U^\top U = I$.

The $(i, j)$ entry of $U^\top U$ is $\mathbf{u}_i^\top \mathbf{u}_j$, which is $1$
when $i = j$ (unit length) and $0$ otherwise (orthogonality) exactly when the
columns are orthonormal. Such a matrix defines a length- and angle-preserving map.

> **Theorem (Orthonormal columns preserve length and inner products).** Let $U$ be $m \times n$ with orthonormal columns, and let
> $\mathbf{x}, \mathbf{y} \in \mathbb{R}^n$. Then
> $\lVert U\mathbf{x} \rVert = \lVert \mathbf{x} \rVert$;
> $(U\mathbf{x}) \cdot (U\mathbf{y}) = \mathbf{x} \cdot \mathbf{y}$; and
> $(U\mathbf{x}) \cdot (U\mathbf{y}) = 0$ if and only if
> $\mathbf{x} \cdot \mathbf{y} = 0$.

A square matrix with orthonormal columns is an **orthogonal matrix**: it satisfies
$U^{-1} = U^\top$, and its rows are orthonormal as well. Orthogonal matrices
represent rigid motions — rotations and reflections — and reappear throughout the
[diagonalization of symmetric matrices](/linear-algebra/symmetric-quadratic-svd/diagonalizing-symmetric-matrices).

$$
% caption: A matrix with orthonormal columns acts as a rigid motion: it sends x to
% U x of the same length, so both endpoints lie on one circle, and the angle
% between any two mapped vectors matches the angle between the originals.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\draw[black] (0,0) circle (2.2);
\draw[->, black] (-2.7,0) -- (2.7,0) node[right] {$x_1$};
\draw[->, black] (0,-2.7) -- (0,2.7) node[above] {$x_2$};
\coordinate (o) at (0,0);
\coordinate (x) at (2.09,0.68);
\coordinate (ux) at (0.9,2.01);
\draw[->, black, very thick] (o) -- (x) node[right] {$\mathbf{x}$};
\draw[->, acc, very thick] (o) -- (ux) node[above right] {$U\mathbf{x}$};
\draw[black] (0,0) ++(18:0.55) arc (18:66:0.55);
\fill[black] (o) circle (1.6pt);
\end{tikzpicture}
$$

## Projection onto a subspace

The orthogonal-basis coordinate theorem already writes a vector in $W$ as a sum of one-dimensional projections
onto the axes of an orthogonal basis. The same formula projects a vector that is
_not_ in $W$ onto it.

> **Theorem (Orthogonal decomposition theorem).** Let $W$ be a subspace of
> $\mathbb{R}^n$. Each $\mathbf{y} \in \mathbb{R}^n$ can be written _uniquely_ as
> $$
> \mathbf{y} = \hat{\mathbf{y}} + \mathbf{z},
> \qquad \hat{\mathbf{y}} \in W, \quad \mathbf{z} \in W^\perp.
> $$
> If $\{\mathbf{u}_1, \dots, \mathbf{u}_p\}$ is any orthogonal basis of $W$, then
> $$
> \hat{\mathbf{y}} = \operatorname{proj}_W \mathbf{y}
>   = \frac{\mathbf{y}\cdot\mathbf{u}_1}{\mathbf{u}_1\cdot\mathbf{u}_1}\mathbf{u}_1
>   + \cdots
>   + \frac{\mathbf{y}\cdot\mathbf{u}_p}{\mathbf{u}_p\cdot\mathbf{u}_p}\mathbf{u}_p,
> \qquad \mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}.
> $$

Define $\hat{\mathbf{y}}$ by the formula. It lies in $W$ as a combination of the
basis. Its residual $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$ is orthogonal to
each $\mathbf{u}_j$: dotting with $\mathbf{u}_j$ leaves
$\mathbf{y} \cdot \mathbf{u}_j - \tfrac{\mathbf{y}\cdot\mathbf{u}_j}{\mathbf{u}_j
\cdot\mathbf{u}_j}(\mathbf{u}_j \cdot \mathbf{u}_j) = 0$, so
$\mathbf{z} \in W^\perp$. Uniqueness follows because if $\mathbf{y} =
\hat{\mathbf{y}}_1 + \mathbf{z}_1$ is another such split, then
$\hat{\mathbf{y}} - \hat{\mathbf{y}}_1 = \mathbf{z}_1 - \mathbf{z}$ lies in both
$W$ and $W^\perp$, hence is orthogonal to itself and zero.

The projection $\hat{\mathbf{y}}$ is the sum of one-dimensional projections onto
mutually orthogonal axes. In $\mathbb{R}^3$ with $W = \operatorname{Span}\{
\mathbf{u}_1, \mathbf{u}_2\}$, the two component projections along $\mathbf{u}_1$
and $\mathbf{u}_2$ add to the perpendicular foot $\hat{\mathbf{y}}$ of
$\mathbf{y}$ in the plane.

$$
% caption: y decomposes into its projection onto the plane W plus a component in
% W-perp; the in-plane part is itself the sum of projections onto the two
% orthogonal basis directions u1 and u2.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
% plane
\draw[thick] (-2.4,-1.0) -- (2.6,-1.0) -- (3.6,0.6) -- (-1.4,0.6) -- cycle;
\node at (3.15,-0.7) {$W$};
\coordinate (o) at (0,-0.2);
\coordinate (yh) at (1.6,0.0);
\coordinate (y) at (1.6,2.0);
% basis axes
\draw[->, thick] (o) -- (2.2,-0.75) node[below] {$\mathbf{u}_1$};
\draw[->, thick] (o) -- (0.4,0.45) node[above left] {$\mathbf{u}_2$};
\draw[->, black, very thick] (o) -- (y) node[above right] {$\mathbf{y}$};
\draw[->, acc, very thick] (o) -- (yh) node[below right] {$\hat{\mathbf{y}}$};
\draw[black, thick] (yh) -- (y) node[midway, right] {$\mathbf{z}$};
\draw[black] (1.6,0.2) -- (1.42,0.2) -- (1.42,0.0);
\fill[black] (o) circle (1.6pt);
\end{tikzpicture}
$$

### Best approximation

The projection is the nearest point in $W$ to $\mathbf{y}$.

> **Theorem (Best approximation theorem).** Let $W$ be a subspace of
> $\mathbb{R}^n$, let $\mathbf{y} \in \mathbb{R}^n$, and let
> $\hat{\mathbf{y}} = \operatorname{proj}_W \mathbf{y}$. Then $\hat{\mathbf{y}}$
> is the closest point in $W$ to $\mathbf{y}$:
> $$
> \lVert \mathbf{y} - \hat{\mathbf{y}} \rVert < \lVert \mathbf{y} - \mathbf{v}
> \rVert \quad \text{for all } \mathbf{v} \in W,\ \mathbf{v} \ne \hat{\mathbf{y}}.
> $$

For any $\mathbf{v} \in W$, write $\mathbf{y} - \mathbf{v} = (\mathbf{y} -
\hat{\mathbf{y}}) + (\hat{\mathbf{y}} - \mathbf{v})$. The first term is in
$W^\perp$ and the second is in $W$, so they are orthogonal, and the Pythagorean
theorem gives
$$
\lVert \mathbf{y} - \mathbf{v} \rVert^2
  = \lVert \mathbf{y} - \hat{\mathbf{y}} \rVert^2
  + \lVert \hat{\mathbf{y}} - \mathbf{v} \rVert^2.
$$
The extra term is positive unless $\mathbf{v} = \hat{\mathbf{y}}$, so
$\hat{\mathbf{y}}$ strictly minimizes the distance. The value $\lVert \mathbf{y} -
\hat{\mathbf{y}} \rVert$ is the distance from $\mathbf{y}$ to $W$, and
$\hat{\mathbf{y}}$ is the **best approximation** to $\mathbf{y}$ by elements of
$W$. This minimizing property is what the least-squares method relies on.

$$
% caption: For any other point v in W the segment y-to-v is the hypotenuse of a
% right triangle with legs y-to-y-hat and y-hat-to-v, so it is strictly longer:
% the projection is the unique nearest point.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\draw[thick] (-0.5,0) -- (5.5,0);
\node at (5.2,-0.35) {$W$};
\coordinate (yh) at (2.0,0);
\coordinate (v) at (4.0,0);
\coordinate (y) at (2.0,2.4);
\draw[->, black, very thick] (0.2,0) -- (y);
\node[black] at (1.75,2.4) {$\mathbf{y}$};
\draw[black, thick] (y) -- (yh) node[midway, left] {$\mathbf{y}-\hat{\mathbf{y}}$};
\draw[black, thick] (y) -- (v) node[midway, above right] {$\mathbf{y}-\mathbf{v}$};
\draw[thick] (yh) -- (v) node[midway, below] {$\hat{\mathbf{y}}-\mathbf{v}$};
\draw[black] (2.0,0.24) -- (2.24,0.24) -- (2.24,0);
\fill[acc] (yh) circle (1.6pt) node[below left] {$\hat{\mathbf{y}}$};
\fill[black] (v) circle (1.6pt) node[below right] {$\mathbf{v}$};
\end{tikzpicture}
$$

## The matrix form of a projection

When the basis is orthonormal, the projection formula simplifies, and it collapses
into a matrix product.

> **Theorem (Projection onto an orthonormal basis).** If $\{\mathbf{u}_1, \dots, \mathbf{u}_p\}$ is an _orthonormal_
> basis for $W$, then
> $$
> \operatorname{proj}_W \mathbf{y}
>   = (\mathbf{y}\cdot\mathbf{u}_1)\mathbf{u}_1 + \cdots
>   + (\mathbf{y}\cdot\mathbf{u}_p)\mathbf{u}_p.
> $$
> With $U = [\,\mathbf{u}_1\ \cdots\ \mathbf{u}_p\,]$,
> $$
> \operatorname{proj}_W \mathbf{y} = U U^\top \mathbf{y}
> \quad \text{for all } \mathbf{y} \in \mathbb{R}^n.
> $$

Because the columns are unit vectors, the denominators $\mathbf{u}_j \cdot
\mathbf{u}_j$ are $1$. The weights $\mathbf{y} \cdot \mathbf{u}_j = \mathbf{u}_j^
\top \mathbf{y}$ are the entries of $U^\top \mathbf{y}$, so
$\operatorname{proj}_W \mathbf{y}$ is $U$ times those weights, namely
$U U^\top \mathbf{y}$. Two extremes bracket the formula:

- If $U$ is $n \times p$ with orthonormal columns, $U^\top U = I_p$ (characterization of orthonormal columns),
  while $U U^\top$ is the projection onto $\operatorname{Col} U$ — generally not
  the identity.
- If $U$ is square ($p = n$), the column space is all of $\mathbb{R}^n$, the
  projection is the identity, and $U U^\top = I$.

For hand computation the orthogonal formula of the orthogonal decomposition theorem is preferred, since the
orthonormal columns usually carry square roots; the matrix form is the theoretical
statement that projection is a _linear_ map with matrix $U U^\top$.

## Decomposing a vector

> **Worked example.** Let $\mathbf{u}_1 = (2, 5, -1)$, $\mathbf{u}_2 = (-2, 1, 1)$,
> and $\mathbf{y} = (1, 2, 3)$, with $W = \operatorname{Span}\{\mathbf{u}_1,
> \mathbf{u}_2\}$. The basis is orthogonal
> ($\mathbf{u}_1 \cdot \mathbf{u}_2 = -4 + 5 - 1 = 0$), so the orthogonal decomposition theorem gives
> $$
> \hat{\mathbf{y}}
>   = \frac{\mathbf{y}\cdot\mathbf{u}_1}{\mathbf{u}_1\cdot\mathbf{u}_1}\mathbf{u}_1
>   + \frac{\mathbf{y}\cdot\mathbf{u}_2}{\mathbf{u}_2\cdot\mathbf{u}_2}\mathbf{u}_2
>   = \frac{9}{30}\mathbf{u}_1 + \frac{3}{6}\mathbf{u}_2
>   = \left(-\tfrac{2}{5}, 2, \tfrac{1}{5}\right),
> $$
> and $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}
> = \left(\tfrac{7}{5}, 0, \tfrac{14}{5}\right)$. The check
> $\mathbf{z} \cdot \mathbf{u}_1 = \mathbf{z} \cdot \mathbf{u}_2 = 0$ puts
> $\mathbf{z} \in W^\perp$, so $\mathbf{y} = \hat{\mathbf{y}} + \mathbf{z}$ is the
> orthogonal decomposition. By the best-approximation theorem, $\hat{\mathbf{y}}$ is the point of $W$
> nearest to $\mathbf{y}$.

## Summary

| Object | Orthogonal basis | Orthonormal basis |
| --- | --- | --- |
| Coordinate weight $c_j$ | $\dfrac{\mathbf{y}\cdot\mathbf{u}_j}{\mathbf{u}_j\cdot\mathbf{u}_j}$ | $\mathbf{y}\cdot\mathbf{u}_j$ |
| Projection $\operatorname{proj}_W\mathbf{y}$ | $\sum_j \dfrac{\mathbf{y}\cdot\mathbf{u}_j}{\mathbf{u}_j\cdot\mathbf{u}_j}\mathbf{u}_j$ | $U U^\top\mathbf{y}$ |
| Matrix identity | — | $U^\top U = I$ |

Orthogonal bases turn coordinates and projections into dot products, and the
best-approximation theorem certifies the projection as the nearest point in the
subspace. Building an orthogonal basis from an arbitrary one is the
[Gram–Schmidt process](/linear-algebra/orthogonality-least-squares/gram-schmidt-and-qr),
and using projection to solve inconsistent systems is the method of
[least-squares problems](/linear-algebra/orthogonality-least-squares/least-squares-problems).[^lay-62]

[^lay-62]: Lay, §6.2 — Orthogonal Sets (Theorems 4–7, orthogonal and orthonormal bases, orthogonal matrices) and §6.3 — Orthogonal Projections (Theorems 8–10, the orthogonal decomposition and best-approximation theorems, and the $UU^\top$ matrix form).