---
title: The Singular Value Decomposition
module: Symmetric Matrices, Quadratic Forms, and the SVD
moduleNumber: 7
lessonNumber: 4
order: 704
summary: >
  The singular value decomposition factors any m×n matrix as A = UΣVᵀ, with
  orthogonal U and V and a nonnegative diagonal Σ of singular values. The
  singular values are the square roots of the eigenvalues of AᵀA, and they
  describe the matrix geometrically as a rotation, an axiswise stretch, and
  another rotation, exposing rank, the four fundamental subspaces, and a
  best low-rank approximation.
topics: [Symmetric Matrices, Quadratic Forms, and the SVD]
sources:
  - book: Lay
    ref: "Ch. 7 — Symmetric Matrices and Quadratic Forms; §7.4 The Singular Value Decomposition"
draft: false
---

[Diagonalization](/linear-algebra/eigenvalues/diagonalization) $A = PDP^{-1}$
requires a square matrix with enough independent eigenvectors, and even that can
fail. The
[Spectral Theorem](/linear-algebra/symmetric-quadratic-svd/diagonalizing-symmetric-matrices)
supplies an orthogonal $P$ in the symmetric case, but neither result covers a
rectangular matrix. The singular value decomposition has no such restriction:
**any** $m \times n$ matrix factors as $A = U\Sigma V^\top$ with $U$ and $V$
orthogonal and $\Sigma$ nonnegative diagonal. The three factors describe the
geometry of $x \mapsto Ax$.

## Stretching, measured by AᵀA

For a symmetric matrix, $|\lambda|$ measures how much $A$ stretches its
eigenvector: if $Ax = \lambda x$ with $\|x\| = 1$, then $\|Ax\| = |\lambda|$. A
rectangular $A$ has no eigenvectors, but the quantity $\|Ax\|$ still has a
largest value on the unit sphere, and finding it leads straight to $A^\top A$.
Because

$$
\|Ax\|^2 = (Ax)^\top(Ax) = x^\top (A^\top A)\, x,
$$

maximizing $\|Ax\|$ is a
[constrained optimization](/linear-algebra/symmetric-quadratic-svd/constrained-optimization)
of the quadratic form of $A^\top A$. That matrix is symmetric, since $(A^\top
A)^\top = A^\top A$, and positive semidefinite, since the form equals
$\|Ax\|^2 \geq 0$. Its eigenvalues are therefore real and nonnegative.

For example, take $A = \begin{bmatrix} 4 & 11 & 14 \\ 8 & 7 & -2
\end{bmatrix}$, which maps the unit sphere in $\mathbb{R}^3$ onto an ellipse
in $\mathbb{R}^2$. Compute

$$
A^\top A = \begin{bmatrix} 80 & 100 & 40 \\ 100 & 170 & 140 \\ 40 & 140 & 200 \end{bmatrix},
$$

with eigenvalues $\lambda_1 = 360$, $\lambda_2 = 90$, $\lambda_3 = 0$ and unit
eigenvectors

$$
v_1 = \begin{bmatrix} 1/3 \\ 2/3 \\ 2/3 \end{bmatrix},\quad
v_2 = \begin{bmatrix} -2/3 \\ -1/3 \\ 2/3 \end{bmatrix},\quad
v_3 = \begin{bmatrix} 2/3 \\ -2/3 \\ 1/3 \end{bmatrix}.
$$

The maximum of $\|Ax\|^2$ is $\lambda_1 = 360$, attained at $v_1$, so the
longest output has length $\|Av_1\| = \sqrt{360} = 6\sqrt{10}$, reached in the
direction $Av_1 = (18, 6)$. The second-largest stretch, over directions
orthogonal to $v_1$, is $\sqrt{90} = 3\sqrt{10}$ at $v_2$, with $Av_2 = (3,-9)$.
These are the semi-axes of the image ellipse.

## Singular values

> **Definition (Singular values).** For an $m \times n$ matrix $A$, let
> $\lambda_1 \geq \cdots \geq \lambda_n \geq 0$ be the eigenvalues of $A^\top A$
> with orthonormal eigenvectors $v_1, \dots, v_n$. The **singular values** of
> $A$ are $\sigma_i = \sqrt{\lambda_i}$, arranged in decreasing order. They equal
> the lengths of the image vectors: $\sigma_i = \|Av_i\|$.

The identity $\|Av_i\| = \sigma_i$ follows from $\|Av_i\|^2 = v_i^\top(A^\top A)
v_i = \lambda_i \|v_i\|^2 = \lambda_i$. In the example, the singular values are
$6\sqrt{10}$, $3\sqrt{10}$, and $0$. The number of nonzero singular values
equals the rank.

> **Theorem (Orthogonal image basis).** Suppose $v_1, \dots, v_n$ is an
> orthonormal eigenvector basis of $A^\top A$ ordered by decreasing eigenvalue,
> and $A$ has $r$ nonzero singular values. Then $\{Av_1, \dots, Av_r\}$ is an
> orthogonal basis for $\operatorname{Col} A$, and $\operatorname{rank} A = r$.

> **Proof.** For $i \neq j$, $(Av_i)^\top(Av_j) = v_i^\top(A^\top A)v_j =
> \lambda_j (v_i^\top v_j) = 0$, so the vectors $Av_1, \dots, Av_n$ are mutually
> orthogonal. Exactly $r$ of them are nonzero (those with $\sigma_i > 0$), so
> $Av_1, \dots, Av_r$ are orthogonal, hence independent, and they lie in
> $\operatorname{Col} A$. Any $Ax = \sum_i c_i Av_i$ collapses to a combination of
> just these $r$ vectors, since the rest vanish, so they span $\operatorname{Col}
> A$. Thus $\operatorname{rank} A = r$. $\;\blacksquare$

Counting nonzero singular values is the numerically dependable way to find rank.
Counting pivots after row reduction is fragile: rounding easily turns a true
zero into a tiny nonzero pivot and reports full rank. In practice, singular
values below a small threshold are treated as zero, and the count of the rest is
the **effective rank**.

$$
% caption: A sends the orthonormal right singular vectors v₁, v₂ on the unit
% circle to orthogonal image vectors Av₁, Av₂ along the axes of an ellipse, with
% lengths equal to the singular values s₁, s₂.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{4A6FA5}
  % domain circle
  \draw[acc, thick] (-4.0,0) circle (1.0);
  \draw[->, acc, very thick] (-4.0,0) -- (-3.0,0) node[right, black] {$v_1$};
  \draw[->, acc, very thick] (-4.0,0) -- (-4.0,1.0) node[above, black] {$v_2$};
  \node[black] at (-4.0,-1.4) {unit circle};
  % arrow
  \draw[->, black, thick] (-2.4,0) -- (-1.0,0) node[midway, above] {$A$};
  % codomain ellipse
  \draw[acc, thick] (1.6,0) ellipse (2.0 and 0.85);
  \draw[->, acc, very thick] (1.6,0) -- (3.6,0) node[right, black] {$Av_1$};
  \draw[->, acc, very thick] (1.6,0) -- (1.6,0.85) node[above, black] {$Av_2$};
  \node[black] at (2.85,-0.35) {$s_1$};
  \node[black] at (1.15,0.5) {$s_2$};
  \node[black] at (1.6,-1.4) {image ellipse};
\end{tikzpicture}
$$

## The decomposition

Normalizing the image vectors $Av_i$ produces the left half of the
factorization.

> **Theorem (Singular value decomposition).** Let $A$ be $m \times n$ with
> rank $r$. Then there exist an $m \times m$ orthogonal matrix $U$, an
> $n \times n$ orthogonal matrix $V$, and an $m \times n$ matrix $\Sigma$ with
> $$
> \Sigma = \begin{bmatrix} D & 0 \\ 0 & 0 \end{bmatrix},
> \qquad D = \operatorname{diag}(\sigma_1, \dots, \sigma_r), \; \sigma_1 \geq \cdots \geq \sigma_r > 0,
> $$
> such that $A = U\Sigma V^\top$.

The columns of $V$ are **right singular vectors**, the columns of $U$ are **left
singular vectors**, and the diagonal of $\Sigma$ holds the singular values. The
construction:

- **Right singular vectors.** Take $V = [\,v_1 \; \cdots \; v_n\,]$, the
  orthonormal eigenvectors of $A^\top A$.
- **Singular value matrix.** Place $\sigma_1, \dots, \sigma_r$ on the diagonal
  of $\Sigma$, padded with zeros to size $m \times n$.
- **Left singular vectors.** For $i \leq r$ set $u_i = \tfrac{1}{\sigma_i} Av_i$,
  a unit vector; then $Av_i = \sigma_i u_i$. Extend $\{u_1, \dots, u_r\}$ to an
  orthonormal basis $\{u_1, \dots, u_m\}$ of $\mathbb{R}^m$.

By construction $U$ and $V$ are orthogonal, and $AV = [\,\sigma_1 u_1 \; \cdots
\; \sigma_r u_r \; 0 \; \cdots \; 0\,] = U\Sigma$. Right-multiplying by $V^\top$,
and using $VV^\top = I$, gives $A = U\Sigma V^\top$. The singular values are
determined by $A$, but $U$ and $V$ are not unique.

$$
% caption: The factorization A = UΣVᵀ writes an m×n matrix as an orthogonal
% rotation Vᵀ, an axiswise scaling Σ by the singular values, and an orthogonal
% rotation U; the block S carries the singular values on its diagonal.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\footnotesize,
  blk/.style={draw, minimum width=13mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{4A6FA5}
  \node[blk, draw=acc, text=acc, thick] (A) at (0,0) {$A$};
  \node at (1.1,0) {$=$};
  \node[blk] (U) at (2.2,0) {$U$};
  \node[blk] (S) at (3.9,0) {$S$};
  \node[blk] (V) at (5.6,0) {$V^T$};
  \node[black, align=center] at (0,-1.25) {$m \times n$};
  \node[black, align=center] at (2.2,-1.25) {$m \times m$\\rotation};
  \node[black, align=center] at (3.9,-1.25) {$m \times n$\\scaling};
  \node[black, align=center] at (5.6,-1.25) {$n \times n$\\rotation};
\end{tikzpicture}
$$

### A rectangular example

> **Worked example.** Complete the singular value decomposition of
> $A = \begin{bmatrix} 4 & 11 & 14 \\ 8 & 7 & -2 \end{bmatrix}$, whose eigendata
> for $A^\top A$ are computed above. The right singular vectors and singular
> values give
> $$
> V = \begin{bmatrix} 1/3 & -2/3 & 2/3 \\ 2/3 & -1/3 & -2/3 \\ 2/3 & 2/3 & 1/3 \end{bmatrix},
> \qquad
> \Sigma = \begin{bmatrix} 6\sqrt{10} & 0 & 0 \\ 0 & 3\sqrt{10} & 0 \end{bmatrix}.
> $$
> Since $A$ has rank $2$, both left singular vectors come from normalizing image
> vectors:
> $$
> u_1 = \frac{1}{6\sqrt{10}}\begin{bmatrix} 18 \\ 6 \end{bmatrix}
> = \frac{1}{\sqrt{10}}\begin{bmatrix} 3 \\ 1 \end{bmatrix},
> \qquad
> u_2 = \frac{1}{3\sqrt{10}}\begin{bmatrix} 3 \\ -9 \end{bmatrix}
> = \frac{1}{\sqrt{10}}\begin{bmatrix} 1 \\ -3 \end{bmatrix}.
> $$
> The set $\{u_1, u_2\}$ is already an orthonormal basis for $\mathbb{R}^2$, so no
> extension is needed and $U = [\,u_1 \; u_2\,]$. The completed decomposition is
> $$
> A = \frac{1}{\sqrt{10}}\begin{bmatrix} 3 & 1 \\ 1 & -3 \end{bmatrix}
> \begin{bmatrix} 6\sqrt{10} & 0 & 0 \\ 0 & 3\sqrt{10} & 0 \end{bmatrix} V^\top.
> $$

### Extending U when there are zero singular values

When $r < m$, the left singular vectors from image normalization fall short of a
full basis and must be extended.

> **Worked example.** Decompose $A = \begin{bmatrix} 1 & 1 \\ 2 & 2 \\ 2 & 2
> \end{bmatrix}$, with $A^\top A = \begin{bmatrix} 9 & 9 \\ 9 & 9 \end{bmatrix}$,
> eigenvalues $18$ and $0$, and unit eigenvectors $v_1 = \tfrac{1}{\sqrt2}(1,1)$,
> $v_2 = \tfrac{1}{\sqrt2}(1,-1)$. The only singular value is
> $\sigma_1 = \sqrt{18} = 3\sqrt2$, so $r = 1$ and
> $$
> u_1 = \frac{1}{3\sqrt2}\, Av_1
> = \frac{1}{3\sqrt2}\begin{bmatrix} \sqrt2 \\ 2\sqrt2 \\ 2\sqrt2 \end{bmatrix}
> = \frac13\begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}.
> $$
> To finish $U$, extend $\{u_1\}$ to an orthonormal basis of $\mathbb{R}^3$. Any
> completing vector must satisfy $u_1^\top x = 0$, i.e. $x_1 + 2x_2 + 2x_3 = 0$;
> a basis for that plane is $w_1 = (-2, 1, 0)$, $w_2 = (-2, 0, 1)$. Applying
> [Gram–Schmidt](/linear-algebra/orthogonality-least-squares/gram-schmidt-and-qr)
> and normalizing gives
> $$
> u_2 = \frac{1}{\sqrt5}\begin{bmatrix} -2 \\ 1 \\ 0 \end{bmatrix},
> \qquad
> u_3 = \frac{1}{3\sqrt5}\begin{bmatrix} -2 \\ -4 \\ 5 \end{bmatrix}.
> $$
> Then $U = [\,u_1 \; u_2 \; u_3\,]$, $\Sigma = \begin{bmatrix} 3\sqrt2 & 0 \\
> 0 & 0 \\ 0 & 0 \end{bmatrix}$, and $A = U\Sigma V^\top$. The extra columns
> $u_2, u_3$ do not affect $A$ (they pair with zero rows of $\Sigma$), but they
> make $U$ orthogonal, and they are an orthonormal basis for
> $\operatorname{Nul} A^\top$.

## The rank-one expansion

Expanding $U\Sigma V^\top$ columnwise, as with the
[spectral decomposition](/linear-algebra/symmetric-quadratic-svd/diagonalizing-symmetric-matrices),
writes $A$ as a sum of rank-one pieces, one per nonzero singular value:

$$
A = \sigma_1 u_1 v_1^\top + \sigma_2 u_2 v_2^\top + \cdots + \sigma_r u_r v_r^\top.
$$

Each term $u_i v_i^\top$ is an $m \times n$ rank-one matrix, and the singular
values order the terms by size: the $\sigma_1$ term contributes most, then the
$\sigma_2$ term, and so on. Truncating the sum after $k < r$ terms yields the
best rank-$k$ approximation of $A$, the basis for
[compression and denoising](/linear-algebra/symmetric-quadratic-svd/svd-applications-pca-imaging).

$$
% caption: The SVD builds A additively from rank-one layers σᵢuᵢvᵢᵀ ordered by
% singular value; keeping only the leading layers gives the best low-rank
% approximation.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\footnotesize,
  lyr/.style={draw, minimum width=17mm, minimum height=13mm, align=center}]
  \definecolor{acc}{HTML}{4A6FA5}
  \node[lyr, draw=acc, text=acc, thick] (A) at (0,0) {$A$};
  \node at (1.25,0) {$=$};
  \node[lyr, fill=acc!14] (t1) at (2.6,0) {$s_1\, u_1 v_1^T$};
  \node at (3.9,0) {$+$};
  \node[lyr, fill=acc!9] (t2) at (5.2,0) {$s_2\, u_2 v_2^T$};
  \node at (6.5,0) {$+ \cdots$};
  \node[black, align=center] at (2.6,-1.15) {largest\\layer};
  \node[black, align=center] at (5.2,-1.15) {next\\layer};
\end{tikzpicture}
$$

## The four fundamental subspaces

The SVD produces orthonormal bases for all four subspaces associated with $A$ at
once, sorted by whether their singular value is zero. With rank $r$:

- **Column space.** $\{u_1, \dots, u_r\}$ is an orthonormal basis for
  $\operatorname{Col} A$, by the orthogonal-image-basis theorem.
- **Left null space.** $\{u_{r+1}, \dots, u_m\}$ is an orthonormal basis for
  $\operatorname{Nul} A^\top = (\operatorname{Col} A)^\perp$.
- **Row space.** $\{v_1, \dots, v_r\}$ is an orthonormal basis for
  $\operatorname{Row} A = \operatorname{Col} A^\top$.
- **Null space.** $\{v_{r+1}, \dots, v_n\}$ is an orthonormal basis for
  $\operatorname{Nul} A$, since $Av_i = 0$ exactly when $\sigma_i = 0$.

The action of $A$ is transparent in these bases: it sends each row-space basis
vector $v_i$ to $\sigma_i u_i$ in the column space, and annihilates the null-space
vectors. This ties together the
[four fundamental subspaces](/linear-algebra/vector-spaces/dimension-and-rank)
with the orthogonality that the SVD makes explicit.

$$
% caption: The SVD splits the domain into Row A ⟂ Nul A and the codomain into
% Col A ⟂ Nul Aᵀ; A carries each vᵢ to σᵢuᵢ and sends the null-space vectors to
% zero.
\begin{tikzpicture}[scale=1.0, >=stealth, font=\footnotesize,
  sub/.style={draw, minimum width=24mm, minimum height=11mm, align=center}]
  \definecolor{acc}{HTML}{4A6FA5}
  % domain
  \node[sub, fill=acc!10] (row) at (0,1.4) {$\operatorname{Row} A$\\$r$ vectors $v_i$};
  \node[sub] (nulA) at (0,-1.4) {$\operatorname{Nul} A$\\remaining $v_i$};
  \node[black] at (0,2.5) {domain $\mathbb{R}^n$};
  % codomain
  \node[sub, fill=acc!10] (col) at (6.0,1.4) {$\operatorname{Col} A$\\$r$ vectors $u_i$};
  \node[sub] (nulAT) at (6.0,-1.4) {$\operatorname{Nul} A^T$\\remaining $u_i$};
  \node[black] at (6.0,2.5) {codomain $\mathbb{R}^m$};
  % maps
  \draw[->, acc, very thick] (row) -- (col) node[midway, above, black] {$v_i \,{:}\, s_i u_i$};
  \draw[->, black, thick] (nulA) -- ++(3.0,0) node[right] {$0$};
\end{tikzpicture}
$$

The singular values also complete the
[Invertible Matrix Theorem](/linear-algebra/matrix-algebra/matrix-inverse-and-invertibility):
for a square $A$, invertibility is equivalent to $A$ having $n$ nonzero singular
values, to $\operatorname{Row} A = \mathbb{R}^n$, and to
$(\operatorname{Nul} A)^\perp = \mathbb{R}^n$.

## Condition number and least squares

Two applications follow immediately from the factorization, both explored in
more depth in the numerical module. Because $U$ and $V$ preserve lengths, any
sensitivity in solving $Ax = b$ is carried by $\Sigma$ alone. For an invertible
$A$, the ratio of the largest to smallest singular value is the **condition
number**

$$
\kappa(A) = \frac{\sigma_1}{\sigma_n},
$$

which measures how much a relative error in $b$ can be amplified in $x$; a large
$\kappa$ signals an ill-conditioned system in which roundoff is amplified. The
[conditioning lesson](/linear-algebra/numerical-linear-algebra/conditioning-and-floating-point)
develops this bound.

The **reduced SVD** keeps only the first $r$ columns, $A = U_r D V_r^\top$ with
$U_r$ ($m \times r$), $D$ ($r \times r$ invertible), and $V_r$ ($n \times r$).
It defines the **pseudoinverse** (Moore–Penrose inverse)

$$
A^{+} = V_r D^{-1} U_r^\top,
$$

which gives the minimum-length
[least-squares solution](/linear-algebra/orthogonality-least-squares/least-squares-problems)
of $Ax = b$ as $\hat{x} = A^{+}b$. Substituting the reduced SVD shows $A\hat{x} =
U_r U_r^\top b$, the orthogonal projection of $b$ onto $\operatorname{Col} A$, so
$\hat{x}$ minimizes $\|Ax - b\|$ exactly as the normal equations require.

Computing $A^\top A$ by hand suits small examples, but in practice it is avoided:
forming $A^\top A$ squares the errors in $A$. Stable algorithms compute the SVD
directly from $A$ by
[iterative bidiagonalization](/linear-algebra/numerical-linear-algebra/numerical-eigenvalues-and-svd).