---
title: Inner Product, Length, and Orthogonality
module: Orthogonality and Least Squares
moduleNumber: 6
lessonNumber: 1
order: 601
summary: >
  The dot product turns the algebra of vectors in R^n into geometry: length,
  distance, and perpendicularity. The inner product yields the norm, the
  Pythagorean theorem, and the orthogonal complement, and the null space of a
  matrix is the orthogonal complement of its row space.
topics: [Orthogonality and Least Squares]
sources:
  - book: Lay
    ref: "Ch. 6 — Orthogonality and Least Squares; §6.1 Inner Product, Length, and Orthogonality"
---

Row reduction settles whether a system has a solution, and vector spaces organize
the solutions once they exist. Neither notion measures anything. To ask which
vector is _closest_ to another, or whether two directions are perpendicular, we
need geometry on $\mathbb{R}^n$: length, distance, and angle. All three come from
a single operation on pairs of vectors, the inner product.

Many applied systems $A\mathbf{x} = \mathbf{b}$ have no
exact solution because the data are noisy or over-determined. The best available
$\mathbf{x}$ makes $A\mathbf{x}$ as _close_ as possible to $\mathbf{b}$, and
closeness is a statement about distance — the setting of
[least-squares problems](/linear-algebra/orthogonality-least-squares/least-squares-problems).

## The inner product

For two vectors $\mathbf{u}, \mathbf{v}$ in $\mathbb{R}^n$, regard each as an
$n \times 1$ matrix. The transpose $\mathbf{u}^\top$ is $1 \times n$, so the
product $\mathbf{u}^\top \mathbf{v}$ is a $1 \times 1$ matrix, written as a single
scalar.

> **Definition (Inner product).** For $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$,
> the **inner product** (or **dot product**) is the scalar
> $$
> \mathbf{u} \cdot \mathbf{v} = \mathbf{u}^\top \mathbf{v}
>   = u_1 v_1 + u_2 v_2 + \cdots + u_n v_n.
> $$

For $\mathbf{u} = (2, -5, -1)$ and $\mathbf{v} = (3, 2, -3)$,
$$
\mathbf{u} \cdot \mathbf{v} = (2)(3) + (-5)(2) + (-1)(-3) = 6 - 10 + 3 = -1,
$$
and $\mathbf{v} \cdot \mathbf{u}$ gives the same value. Commutativity holds in
general, alongside the other properties inherited from the transpose.[^lay-61]

> **Theorem (Properties of the inner product).** For $\mathbf{u},
> \mathbf{v}, \mathbf{w} \in \mathbb{R}^n$ and any scalar $c$:
> - $\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}$ (symmetry).
> - $(\mathbf{u} + \mathbf{v}) \cdot \mathbf{w}
>   = \mathbf{u} \cdot \mathbf{w} + \mathbf{v} \cdot \mathbf{w}$ (additivity).
> - $(c\mathbf{u}) \cdot \mathbf{v} = c(\mathbf{u} \cdot \mathbf{v})
>   = \mathbf{u} \cdot (c\mathbf{v})$ (homogeneity).
> - $\mathbf{u} \cdot \mathbf{u} \ge 0$, with equality if and only if
>   $\mathbf{u} = \mathbf{0}$ (positive-definiteness).

Additivity and homogeneity combine, so the inner product distributes over any
linear combination:
$$
(c_1 \mathbf{u}_1 + \cdots + c_p \mathbf{u}_p) \cdot \mathbf{w}
  = c_1 (\mathbf{u}_1 \cdot \mathbf{w}) + \cdots
  + c_p (\mathbf{u}_p \cdot \mathbf{w}).
$$
Every geometric fact below follows from these four properties, not from the
coordinate formula, which is what later lets the vectors become
[functions](/linear-algebra/orthogonality-least-squares/inner-product-spaces) and
the dot product an integral.

## Length and distance

The last property of the inner-product-properties theorem makes $\mathbf{v} \cdot \mathbf{v} \ge 0$, so its
square root is defined.

> **Definition (Length).** The **length** (or **norm**) of
> $\mathbf{v} \in \mathbb{R}^n$ is the nonnegative scalar
> $$
> \lVert \mathbf{v} \rVert = \sqrt{\mathbf{v} \cdot \mathbf{v}}
>   = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2},
> \qquad \lVert \mathbf{v} \rVert^2 = \mathbf{v} \cdot \mathbf{v}.
> $$

For $\mathbf{v} = (a, b) \in \mathbb{R}^2$, this is the length of the segment from
the origin to $\mathbf{v}$ by the Pythagorean theorem; the same calculation on the
diagonal of a box recovers the usual length in $\mathbb{R}^3$. Scaling a vector
scales its length by the absolute value of the scalar,
$\lVert c\mathbf{v} \rVert = |c|\,\lVert \mathbf{v} \rVert$, since
$\lVert c\mathbf{v} \rVert^2 = (c\mathbf{v}) \cdot (c\mathbf{v})
= c^2 \lVert \mathbf{v} \rVert^2$.

A vector of length $1$ is a **unit vector**. Dividing a nonzero $\mathbf{v}$ by its
length produces the unit vector $\mathbf{u} = \mathbf{v} / \lVert \mathbf{v}
\rVert$ in the same direction; the operation is **normalizing** $\mathbf{v}$.

> **Worked example.** Normalize $\mathbf{v} = (1, -2, 2, 0)$. The length is
> $$
> \lVert \mathbf{v} \rVert = \sqrt{1^2 + (-2)^2 + 2^2 + 0^2} = \sqrt{9} = 3,
> $$
> so the unit vector in the direction of $\mathbf{v}$ is
> $$
> \mathbf{u} = \frac{1}{\lVert \mathbf{v} \rVert}\mathbf{v}
>   = \tfrac{1}{3}(1, -2, 2, 0)
>   = \left(\tfrac{1}{3}, -\tfrac{2}{3}, \tfrac{2}{3}, 0\right),
> $$
> and $\lVert \mathbf{u} \rVert^2 = \tfrac{1}{9}(1 + 4 + 4 + 0) = 1$ confirms unit
> length.

> **Definition (Distance).** For $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$, the
> **distance** between them is
> $\operatorname{dist}(\mathbf{u}, \mathbf{v})
> = \lVert \mathbf{u} - \mathbf{v} \rVert$.

On the number line this reduces to $|a - b|$, the familiar distance in
$\mathbb{R}^1$. In $\mathbb{R}^2$ and $\mathbb{R}^3$ it is the Euclidean distance
between the two points.

> **Worked example.** Compute the distance between $\mathbf{u} = (7, 1)$ and
> $\mathbf{v} = (3, 2)$. The difference is $\mathbf{u} - \mathbf{v} = (4, -1)$, so
> $$
> \operatorname{dist}(\mathbf{u}, \mathbf{v}) = \lVert \mathbf{u} - \mathbf{v} \rVert
>   = \sqrt{4^2 + (-1)^2} = \sqrt{17}.
> $$

$$
% caption: The distance between u and v is the length of u - v; translating that
% difference vector to the origin shows the same length as the segment joining
% the two points.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (-0.4,0) -- (5.2,0) node[right] {$x_1$};
\draw[->, black] (0,-0.4) -- (0,3.6) node[above] {$x_2$};
\coordinate (u) at (4.4,0.7);
\coordinate (v) at (1.9,1.9);
\draw[->, very thick] (0,0) -- (u) node[below right] {$\mathbf{u}$};
\draw[->, very thick] (0,0) -- (v) node[above left] {$\mathbf{v}$};
\draw[black, thick] (v) -- (u);
\node[black] at (3.35,1.5) {$\mathbf{u} - \mathbf{v}$};
\fill[black] (u) circle (1.6pt);
\fill[black] (v) circle (1.6pt);
\end{tikzpicture}
$$

## Orthogonal vectors

Perpendicularity of two lines through the origin has a clean algebraic test.
Consider $\mathbf{u}$ and $\mathbf{v}$ and compute
$$
\lVert \mathbf{u} + \mathbf{v} \rVert^2
  = (\mathbf{u} + \mathbf{v}) \cdot (\mathbf{u} + \mathbf{v})
  = \lVert \mathbf{u} \rVert^2 + \lVert \mathbf{v} \rVert^2
  + 2\,\mathbf{u} \cdot \mathbf{v}.
$$
The lines through $\mathbf{u}$ and $\mathbf{v}$ are geometrically perpendicular
exactly when $\operatorname{dist}(\mathbf{u}, \mathbf{v})
= \operatorname{dist}(\mathbf{u}, -\mathbf{v})$, and expanding both squared
distances shows this happens if and only if $\mathbf{u} \cdot \mathbf{v} = 0$.

> **Definition (Orthogonal vectors).** Two vectors
> $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$ are **orthogonal** if
> $\mathbf{u} \cdot \mathbf{v} = 0$.

The zero vector is orthogonal to every vector, since $\mathbf{0}^\top \mathbf{v} =
0$ always. The expansion above, read with $\mathbf{u} \cdot \mathbf{v} = 0$,
collapses the cross term and gives a familiar identity.

> **Theorem (Pythagorean theorem).** Two vectors $\mathbf{u}$ and $\mathbf{v}$
> are orthogonal if and only if
> $\lVert \mathbf{u} + \mathbf{v} \rVert^2
> = \lVert \mathbf{u} \rVert^2 + \lVert \mathbf{v} \rVert^2$.

$$
% caption: When u and v are orthogonal, they are the legs of a right triangle
% whose hypotenuse is u + v, and the Pythagorean relation on the three lengths
% holds exactly.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
\coordinate (o) at (0,0);
\coordinate (u) at (3.4,0);
\coordinate (s) at (3.4,2.2);
\draw[->, very thick] (o) -- (u) node[midway, below] {$\mathbf{u}$};
\draw[->, very thick] (u) -- (s) node[midway, right] {$\mathbf{v}$};
\draw[->, black, thick] (o) -- (s) node[midway, above left] {$\mathbf{u}+\mathbf{v}$};
\draw[black] (3.4,0.32) -- (3.08,0.32) -- (3.08,0);
\end{tikzpicture}
$$

For nonzero $\mathbf{u}, \mathbf{v}$ in $\mathbb{R}^2$ or $\mathbb{R}^3$, the inner
product carries the angle $\vartheta$ between the two segments through the law of
cosines:
$$
\mathbf{u} \cdot \mathbf{v}
  = \lVert \mathbf{u} \rVert\, \lVert \mathbf{v} \rVert \cos \vartheta.
$$
When $n > 3$ this equation _defines_ the angle. Orthogonality is the case
$\cos \vartheta = 0$. In statistics, the same quantity, computed for
mean-centered data vectors, is the correlation coefficient.

## Orthogonal complements

Perpendicularity extends from single vectors to whole subspaces. If a vector
$\mathbf{z}$ is orthogonal to every vector in a subspace $W$ of $\mathbb{R}^n$,
then $\mathbf{z}$ is orthogonal to $W$.

> **Definition (Orthogonal complement).** The **orthogonal complement** of a
> subspace $W \subseteq \mathbb{R}^n$ is the set of all vectors orthogonal to
> $W$,
> $$
> W^\perp = \{\, \mathbf{z} \in \mathbb{R}^n :
>   \mathbf{z} \cdot \mathbf{w} = 0 \text{ for all } \mathbf{w} \in W \,\},
> $$
> read "$W$ perp."

Take $W$ a plane through the origin in $\mathbb{R}^3$ and $L$ the line through the
origin perpendicular to it. Every vector on $L$ is orthogonal to every vector in
$W$, and vice versa, so $L = W^\perp$ and $W = L^\perp$.

$$
% caption: In R^3 a plane W through the origin and the line L normal to it are
% orthogonal complements of each other: every vector on one is perpendicular to
% every vector on the other.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\small]
\definecolor{acc}{HTML}{4A6FA5}
% plane W as a parallelogram
\fill[acc!10] (-2.2,-0.9) -- (2.2,-0.5) -- (3.0,0.7) -- (-1.4,0.3) -- cycle;
\draw[acc, thick] (-2.2,-0.9) -- (2.2,-0.5) -- (3.0,0.7) -- (-1.4,0.3) -- cycle;
\node[acc] at (2.5,-0.75) {$W$};
% normal line L
\draw[black, thick] (0.4,-1.6) -- (0.4,1.9);
\node[black] at (0.72,1.85) {$L$};
% vectors
\draw[->, black, very thick] (0.4,-0.1) -- (0.4,1.3) node[right] {$\mathbf{z}$};
\draw[->, very thick] (0.4,-0.1) -- (2.0,0.15) node[above] {$\mathbf{w}$};
\draw[black] (0.4,0.25) -- (0.62,0.28) -- (0.6,0.06);
\fill[black] (0.4,-0.1) circle (1.6pt);
\end{tikzpicture}
$$

Two facts follow directly from the inner-product properties, and both are used
repeatedly later:

- **Spanning-set test.** A vector $\mathbf{x}$ is in $W^\perp$ if and only if
  $\mathbf{x}$ is orthogonal to every vector in a set that spans $W$. Being
  orthogonal to a spanning set forces orthogonality to all linear combinations.
- **$W^\perp$ is a subspace.** It is closed under addition and scalar
  multiplication, again by additivity and homogeneity of the dot product.

### The row space and the null space

The orthogonal complement connects two subspaces attached to a matrix. Recall that
$\operatorname{Row} A$ is the span of the rows of $A$ and
$\operatorname{Nul} A$ is the solution set of $A\mathbf{x} = \mathbf{0}$
([null and column spaces](/linear-algebra/vector-spaces/null-and-column-spaces)).

> **Theorem (Orthogonal complements of the row and column spaces).** For an $m \times n$ matrix $A$, the orthogonal complement of the
> row space is the null space, and the orthogonal complement of the column space
> is the null space of $A^\top$:
> $$
> (\operatorname{Row} A)^\perp = \operatorname{Nul} A,
> \qquad
> (\operatorname{Col} A)^\perp = \operatorname{Nul} A^\top.
> $$

The proof is the row–column rule read as a statement about dot products. If
$\mathbf{x}$ is in $\operatorname{Nul} A$, then $A\mathbf{x} = \mathbf{0}$ means
each entry of $A\mathbf{x}$ — the dot product of a row of $A$ with $\mathbf{x}$ —
is zero. So $\mathbf{x}$ is orthogonal to every row, hence to their span
$\operatorname{Row} A$. Conversely, if $\mathbf{x}$ is orthogonal to
$\operatorname{Row} A$, it is orthogonal to each row and $A\mathbf{x} =
\mathbf{0}$. This proves the first identity. Applying it to $A^\top$, whose row
space is $\operatorname{Col} A$, gives the second.

This is the orthogonal picture of the fundamental subspaces. The four subspaces
attached to $A$ split $\mathbb{R}^n$ and $\mathbb{R}^m$ into perpendicular pairs.

$$
% caption: The four fundamental subspaces of an m-by-n matrix: in the domain
% R^n, the row space and null space are orthogonal complements; in the codomain
% R^m, the column space and the null space of A-transpose are complements.
\begin{tikzpicture}[>=stealth, scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
% domain box
\draw[black] (-0.3,-2.6) rectangle (3.6,2.6);
\node[black, anchor=south] at (1.65,2.6) {domain $\mathbb{R}^n$};
\draw[thick] (0.2,0) -- (3.1,0);
\node[anchor=west] at (1.5,0.35) {$\operatorname{Row} A$};
\draw[black, thick] (1.65,-2.1) -- (1.65,-0.2);
\node[black, anchor=west] at (1.85,-1.2) {$\operatorname{Nul} A$};
\draw[black] (1.65,-0.2) -- (1.9,-0.2) -- (1.9,0);
% codomain box
\begin{scope}[xshift=6.6cm]
\draw[black] (-0.3,-2.6) rectangle (3.6,2.6);
\node[black, anchor=south] at (1.65,2.6) {codomain $\mathbb{R}^m$};
\draw[thick] (0.2,0) -- (3.1,0);
\node[anchor=west] at (1.5,0.35) {$\operatorname{Col} A$};
\draw[black, thick] (1.65,-2.1) -- (1.65,-0.2);
\node[black, anchor=west] at (1.85,-1.2) {$\operatorname{Nul} A^{T}$};
\draw[black] (1.65,-0.2) -- (1.9,-0.2) -- (1.9,0);
\end{scope}
\draw[->, thick] (3.7,1.3) .. controls (5.1,1.9) and (5.5,1.9) .. (6.3,1.3)
  node[midway, above, black] {multiply by $A$};
\end{tikzpicture}
$$

A dimension count completes the picture. If $W$ is any subspace of
$\mathbb{R}^n$, then
$$
\dim W + \dim W^\perp = n.
$$
Writing $W = \operatorname{Row} A$ for a suitable $A$, the orthogonal-complements theorem gives
$W^\perp = \operatorname{Nul} A$, and the
[Rank Theorem](/linear-algebra/matrix-algebra/subspaces-dimension-rank)
($\operatorname{rank} A + \dim \operatorname{Nul} A = n$) supplies the sum.

## Normalization and orthogonality together

> **Worked example.** Let $\mathbf{c} = (\tfrac{4}{3}, 1, \tfrac{2}{3})$ and
> $\mathbf{d} = (5, -6, -1)$.
>
> _Unit vector along $\mathbf{c}$._ Scale by $3$ to clear fractions, giving
> $\mathbf{y} = (4, 3, 2)$ with $\lVert \mathbf{y} \rVert^2 = 16 + 9 + 4 = 29$. The
> unit vector is
> $$
> \mathbf{u} = \tfrac{1}{\sqrt{29}}(4, 3, 2),
> $$
> and it points along $\mathbf{c}$ because $\mathbf{y} = 3\mathbf{c}$.
>
> _Orthogonality of $\mathbf{d}$ to $\mathbf{c}$._ Compute
> $$
> \mathbf{d} \cdot \mathbf{c}
>   = (5)\tfrac{4}{3} + (-6)(1) + (-1)\tfrac{2}{3}
>   = \tfrac{20}{3} - 6 - \tfrac{2}{3} = 0,
> $$
> so $\mathbf{d} \perp \mathbf{c}$. Since $\mathbf{u} = k\mathbf{c}$ for a scalar
> $k$, homogeneity gives $\mathbf{d} \cdot \mathbf{u} = k(\mathbf{d} \cdot
> \mathbf{c}) = 0$: orthogonality to $\mathbf{c}$ carries over to every scalar
> multiple, including the normalized $\mathbf{u}$.

## Summary

| Concept | Definition | Geometric reading |
| --- | --- | --- |
| Inner product | $\mathbf{u} \cdot \mathbf{v} = \sum u_i v_i$ | $\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert\cos\vartheta$ |
| Length | $\lVert\mathbf{v}\rVert = \sqrt{\mathbf{v}\cdot\mathbf{v}}$ | distance from origin |
| Distance | $\lVert\mathbf{u}-\mathbf{v}\rVert$ | separation of two points |
| Orthogonal | $\mathbf{u}\cdot\mathbf{v}=0$ | perpendicular directions |
| $W^\perp$ | vectors orthogonal to all of $W$ | complementary subspace, $\dim W + \dim W^\perp = n$ |

Length, distance, angle, and perpendicularity all reduce to the inner product,
and the orthogonal-complements theorem identifies $(\operatorname{Row} A)^\perp$ with $\operatorname{Nul}
A$. An [orthogonal basis](/linear-algebra/orthogonality-least-squares/orthogonal-sets-and-projections)
turns each projection weight into a single dot product.

[^lay-61]: Lay, §6.1 — Inner Product, Length, and Orthogonality; Theorem 1 and the definitions of length, distance, orthogonality, and the orthogonal complement, with Theorem 3 relating $(\operatorname{Row} A)^\perp$ to $\operatorname{Nul} A$.