Inner Product, Length, and Orthogonality

Row reduction settles whether a system has a solution, and vector spaces organize the solutions once they exist. Neither notion measures anything. To ask which vector is closest to another, or whether two directions are perpendicular, we need geometry on ${\mathbb{R}}^n$: length, distance, and angle. All three come from a single operation on pairs of vectors, the inner product.

Many applied systems $A\mathbf{x}=\mathbf{b}$ have no exact solution because the data are noisy or over-determined. The best available $\mathbf{x}$ makes $A\mathbf{x}$ as close as possible to $\mathbf{b}$, and closeness is a statement about distance — the setting of least-squares problems.

The inner product

For two vectors $\mathbf{u},\mathbf{v}$ in ${\mathbb{R}}^n$, regard each as an $n\times 1$ matrix. The transpose ${\mathbf{u}}^{\top }$ is $1\times n$, so the product ${\mathbf{u}}^{\top }\mathbf{v}$ is a $1\times 1$ matrix, written as a single scalar.

For $\mathbf{u}=(2,-5,-1)$ and $\mathbf{v}=(3,2,-3)$,

and $\mathbf{v}\cdot \mathbf{u}$ gives the same value. Commutativity holds in general, alongside the other properties inherited from the transpose.¹

Additivity and homogeneity combine, so the inner product distributes over any linear combination:

Every geometric fact below follows from these four properties, not from the coordinate formula, which is what later lets the vectors become functions and the dot product an integral.

Length and distance

The last property of the inner-product-properties theorem makes $\mathbf{v}\cdot \mathbf{v}\ge 0$, so its square root is defined.

For $\mathbf{v}=(a,b)\in {\mathbb{R}}^2$, this is the length of the segment from the origin to $\mathbf{v}$ by the Pythagorean theorem; the same calculation on the diagonal of a box recovers the usual length in ${\mathbb{R}}^3$. Scaling a vector scales its length by the absolute value of the scalar, $\Vert c\mathbf{v}\Vert =|c|\Vert \mathbf{v}\Vert$, since $\Vert c\mathbf{v}{\Vert }^2=(c\mathbf{v})\cdot (c\mathbf{v})=c^2\Vert \mathbf{v}{\Vert }^2$.

A vector of length $1$ is a unit vector. Dividing a nonzero $\mathbf{v}$ by its length produces the unit vector $\mathbf{u}=\mathbf{v}/\Vert \mathbf{v}\Vert$ in the same direction; the operation is normalizing $\mathbf{v}$.

On the number line this reduces to $|a-b|$, the familiar distance in ${\mathbb{R}}^1$. In ${\mathbb{R}}^2$ and ${\mathbb{R}}^3$ it is the Euclidean distance between the two points.

Orthogonal vectors

Perpendicularity of two lines through the origin has a clean algebraic test. Consider $\mathbf{u}$ and $\mathbf{v}$ and compute

The lines through $\mathbf{u}$ and $\mathbf{v}$ are geometrically perpendicular exactly when $\mathrm{dist}(\mathbf{u},\mathbf{v})=\mathrm{dist}(\mathbf{u},-\mathbf{v})$, and expanding both squared distances shows this happens if and only if $\mathbf{u}\cdot \mathbf{v}=0$.

The zero vector is orthogonal to every vector, since $0^{\top }\mathbf{v}=0$ always. The expansion above, read with $\mathbf{u}\cdot \mathbf{v}=0$, collapses the cross term and gives a familiar identity.

For nonzero $\mathbf{u},\mathbf{v}$ in ${\mathbb{R}}^2$ or ${\mathbb{R}}^3$, the inner product carries the angle $\vartheta$ between the two segments through the law of cosines:

When $n>3$ this equation defines the angle. Orthogonality is the case $\cos \vartheta =0$. In statistics, the same quantity, computed for mean-centered data vectors, is the correlation coefficient.

Orthogonal complements

Perpendicularity extends from single vectors to whole subspaces. If a vector $\mathbf{z}$ is orthogonal to every vector in a subspace $W$ of ${\mathbb{R}}^n$, then $\mathbf{z}$ is orthogonal to $W$.

Take $W$ a plane through the origin in ${\mathbb{R}}^3$ and $L$ the line through the origin perpendicular to it. Every vector on $L$ is orthogonal to every vector in $W$, and vice versa, so $L=W^{\perp }$ and $W=L^{\perp }$.

Two facts follow directly from the inner-product properties, and both are used repeatedly later:

• Spanning-set test. A vector $\mathbf{x}$ is in $W^{\perp }$ if and only if $\mathbf{x}$ is orthogonal to every vector in a set that spans $W$. Being orthogonal to a spanning set forces orthogonality to all linear combinations.
• $W^{\perp }$ is a subspace. It is closed under addition and scalar multiplication, again by additivity and homogeneity of the dot product.

The row space and the null space

The orthogonal complement connects two subspaces attached to a matrix. Recall that $\mathrm{Row}A$ is the span of the rows of $A$ and $\mathrm{Nul}A$ is the solution set of $A\mathbf{x}=0$ (null and column spaces).

The proof is the row–column rule read as a statement about dot products. If $\mathbf{x}$ is in $\mathrm{Nul}A$, then $A\mathbf{x}=0$ means each entry of $A\mathbf{x}$ — the dot product of a row of $A$ with $\mathbf{x}$ — is zero. So $\mathbf{x}$ is orthogonal to every row, hence to their span $\mathrm{Row}A$. Conversely, if $\mathbf{x}$ is orthogonal to $\mathrm{Row}A$, it is orthogonal to each row and $A\mathbf{x}=0$. This proves the first identity. Applying it to $A^{\top }$, whose row space is $\mathrm{Col}A$, gives the second.

This is the orthogonal picture of the fundamental subspaces. The four subspaces attached to $A$ split ${\mathbb{R}}^n$ and ${\mathbb{R}}^m$ into perpendicular pairs.

A dimension count completes the picture. If $W$ is any subspace of ${\mathbb{R}}^n$, then

Writing $W=\mathrm{Row}A$ for a suitable $A$, the orthogonal-complements theorem gives $W^{\perp }=\mathrm{Nul}A$, and the Rank Theorem ($\mathrm{rank}A+\dim \mathrm{Nul}A=n$) supplies the sum.

Normalization and orthogonality together

Summary

Length, distance, angle, and perpendicularity all reduce to the inner product, and the orthogonal-complements theorem identifies $(\mathrm{Row}A{)}^{\perp }$ with $\mathrm{Nul}A$. An orthogonal basis turns each projection weight into a single dot product.