Linear Factor Models

A generative model proposes a process for how the data was made and then fits that process to the data. The simplest such process is linear: a small set of hidden causes $h$, drawn from a fixed prior, with each observation $x$ a linear mix of those causes plus noise. Every model in this lesson (probabilistic PCA, factor analysis, independent component analysis, sparse coding) instantiates this process; they differ only in the prior $p(h)$ and the noise.¹

The shared template

A linear factor model posits a latent vector $h\in {\mathbb{R}}^k$ and an observed vector $x\in {\mathbb{R}}^d$ related by a linear decoder:

where $W\in {\mathbb{R}}^{d\times k}$ is the factor-loading matrix, $b$ the mean, and the noise is independent of $h$. The columns of $W$ are the factors: $x={\sum }_{j=1}^k{h}_j{W}_{:,j}+b+\text{noise}$ reads each observation as a weighted sum of a fixed set of directions, the weights being the latents.

The dependency structure is directed: the latent causes generate the observations, never the reverse. Each $h_j$ fans out through column $W_{:,j}$ to influence every coordinate of $x$.

Fixing the prior to a standard Gaussian and the decoder to be linear makes the whole joint distribution tractable, because affine maps of Gaussians stay Gaussian. That is the regime of the first two models.

Probabilistic PCA and factor analysis

Take the latent prior to be a unit Gaussian and the noise to be Gaussian as well. Factor analysis allows the noise to have a different variance on each coordinate; probabilistic PCA forces one shared variance. Both write

with the noise covariance $\Psi$ diagonal. The two models are the two choices of $\Psi$:

The marginal over the data

Because $x$ is an affine function of the jointly Gaussian pair $(h,\text{noise})$, the marginal $p(x)$ is Gaussian, and we only need its mean and covariance. Using $\mathbb{E}[h]=0$ and $\mathrm{Cov}(h)=I$, independence of $h$ and the noise:

So the model says the data is a Gaussian whose covariance is low-rank plus diagonal:

The $k$ columns of $W$ supply a rank-$k$ chunk of the covariance, the shared, correlated structure, while $\Psi$ mops up the per-coordinate residual. This is the whole content of the models: explain the correlations in $x$ with few factors, and dump everything left over into independent noise.²

Classical PCA is therefore the ${\sigma }^2\to 0$ corner of this family: a deterministic projection onto the leading eigenvectors, with no noise model at all. The probabilistic version adds back a likelihood, which is what lets it generate new samples and handle missing data.³

The geometric difference between PPCA and FA is the shape of the noise. Isotropic noise (PPCA) is a round blur identical on every axis. Diagonal noise (FA) is an axis-aligned blur that can be wider on some coordinates than others.

Both models are fit by maximum likelihood — closed-form for PCA, by expectation-maximization for general FA and PPCA, alternating between inferring the posterior over $h$ and re-estimating $W,\Psi$.

A worked covariance decomposition

For example, take $d=3$ observed coordinates and $k=1$ latent factor, with a factor-loading vector and diagonal noise

The model predicts $\mathrm{Cov}(x)=W{W}^T+\Psi$. The rank-one outer product is

and adding $\Psi$ to the diagonal gives the full model covariance

Read the structure directly off the matrix. The off-diagonal entries come entirely from $W{W}^T$: coordinates $1$ and $2$ co-vary at $+2$ because both load positively on the factor, while coordinate $3$ anti-correlates ($-2,-1$) because it loads negatively. The single hidden cause explains every correlation among the three observations. The diagonal splits into a shared part and a private part: variance of $x_1$ is $4$ (from the factor) plus $0.5$ (its own noise). A fit with $k$ factors would explain the covariance with $3k+3$ numbers here ($W$ plus $\Psi$) instead of the $6$ free entries of a full $3\times 3$ covariance — the compression that makes the model useful grows as $d$ climbs and $k$ stays small.

The posterior over the latent given an observation follows from the same Gaussian algebra. For a point $x=(3,1,-1{)}^T$ with $b=0$, the posterior mean is $\mathbb{E}[h\mid x]=(W^T{\Psi }^{-1}W+1{)}^{-1}{W}^T{\Psi }^{-1}(x-b)$. Here $W^T{\Psi }^{-1}W=4/0.5+1/1+1/0.25=8+1+4=13$, and $W^T{\Psi }^{-1}x=2\cdot 3/0.5+1\cdot 1/1+(-1)(-1)/0.25=12+1+4=17$, so

The point is best explained by a factor firing at about $1.2$; multiplying back through $W$ gives the reconstruction $\hat{x}=W\cdot 1.21\approx (2.43,1.21,-1.21)$, and the gap $x-\hat{x}$ is what the noise model absorbs.

Independent component analysis

Gaussian latents are convenient but they throw away information: a Gaussian is rotationally symmetric, so $W$ is only identifiable up to an orthogonal rotation, and you cannot tell which factor is which. Independent component analysis (ICA) abandons the Gaussian prior precisely to recover the individual sources. It assumes the latents are statistically independent and non-Gaussian, and the decoder is (in the basic form) noise-free and square:

The canonical setting is the cocktail-party problem: $k$ microphones record $k$ speakers talking at once, each microphone picking up a different linear mixture. ICA recovers the original voices (blind source separation) knowing only the mixtures, not the mixing matrix.⁴

Why non-Gaussianity is required

The identifiability argument is sharp. Suppose the sources were Gaussian: $h\sim \mathcal{N}(0,I)$. For any orthogonal $R$ ($R{R}^T=I$), the rotated latent $h^{\prime }=Rh$ is also $\mathcal{N}(0,I)$ — the standard Gaussian is rotation-invariant. Then the mixing $W^{\prime }=W{R}^T$ produces an identical data distribution:

so $W$ and $W{R}^T$ are indistinguishable from the data alone, for every rotation $R$. The sources are recoverable only up to an arbitrary rotation, exactly the non-identifiability that FA inherits. Non-Gaussian sources break this symmetry: a non-Gaussian distribution is not rotation-invariant, so only the true unmixing makes the recovered components independent.

Practically, ICA maximizes a non-Gaussianity contrast (kurtosis or negentropy) of the recovered components $h=W^{-1}x$, since by the central limit theorem a mixture of independent signals is more Gaussian than its parts, so the most non-Gaussian projections are the un-mixed sources.⁵

For example, suppose two independent sources are mixed by

Given the mixtures $x$, ICA searches for the unmixing matrix $W^{-1}$; inverting the $2\times 2$ (determinant $1\cdot 2-1\cdot 0.5=1.5$) gives

For a specific sample $x=(3,3{)}^T$, applying the estimated unmixing recovers $h=W^{-1}x=(1.33\cdot 3-0.67\cdot 3,-0.33\cdot 3+0.67\cdot 3)=(2,1)$ — the original source amplitudes, up to the scale and sign freedoms in the theorem. In practice $W^{-1}$ is not handed to the algorithm; it is found by rotating the whitened data until the two output coordinates are maximally non-Gaussian, at which point their scatter looks like the pinned square on the right above rather than the ambiguous round blob on the left.

Sparse coding

The previous models keep the latent dimension small. Sparse coding goes the other way: it uses an overcomplete dictionary ($k>d$, more factors than data dimensions) but insists that each observation uses only a few of them. The prior on $h$ is sharply peaked at zero (a Laplace, equivalently an $L_1$ penalty), so most components are exactly zero.

Inference is an optimization

There is no closed-form posterior. For a fixed dictionary $W$, the code $h$ for an input $x$ is the MAP estimate: minimize reconstruction error plus the $L_1$ penalty that comes from the Laplace prior:

This is the LASSO: convex in $h$, with a non-smooth $L_1$ term that drives most coordinates to exactly zero. Learning alternates between this sparse inference and a dictionary update on $W$.

Geometrically, the input is reconstructed as a sparse weighted sum: out of a large dictionary, only a handful of atoms switch on, each scaled by its (nonzero) code.

Learned dictionaries are Gabor-like

Trained on natural image patches, the learned atoms reliably become localized, oriented, band-pass edge detectors, the discrete analogue of Gabor filters and a close match to the receptive fields of simple cells in the primary visual cortex. Sparsity is doing the work: forcing each patch to be explained by few atoms pushes the dictionary toward a basis of elementary edges that compose efficiently.⁶

The family, side by side

All four models are one template, $x=Wh+b+\text{noise}$, distinguished by the prior on $h$, the noise, and what structure they recover.

Gaussian priors (PPCA, FA) give tractability at the cost of identifiability; ICA drops the Gaussian to make individual sources identifiable; sparse coding drops the small latent dimension for an overcomplete, interpretable basis. Each is a different answer to the same question: what simple hidden causes, linearly combined, produced this data?

Linear factors in the deep era

The linear decoder looks primitive next to a neural network, yet each model in this lesson has a direct descendant in current practice, and the reasons they were introduced still hold.

Sparse coding survives as dictionary learning and $L_1$ regularization. Olshausen and Field's original result — that a sparsity prior on natural image patches yields Gabor-like edge filters — was the first computational account of why V1 simple cells look the way they do, and it remains the standard reference for sparse dictionary methods.⁷ The same $L_1$ penalty reappears everywhere a model should use few active units: LASSO regression, sparse attention, and the sparse autoencoders now used to pull interpretable features out of large language models are the same objective with a learned nonlinear encoder in place of the LASSO solve.

ICA is still the default for blind source separation. The FastICA algorithm, which maximizes negentropy by a fixed-point iteration, made ICA fast enough for routine use and is the tool of choice for removing eye-blink and heartbeat artifacts from EEG and MEG recordings, where the independent sources are literally distinct physical generators.⁸ The non-Gaussianity argument from this lesson explains why it works: neural and muscular sources are heavy-tailed, so their independent directions are identifiable.

The template itself became the encoder–decoder. Replacing the linear map $Wh+b$ with a deep network, and the closed-form Gaussian posterior with an amortized neural approximation, turns factor analysis into the variational autoencoder of Kingma and Welling — same latent-variable-and-decoder graph, same Gaussian prior, now nonlinear and trained by stochastic gradients.⁹ Read this way, the rest of the chapter is one long answer to a question the linear models raised and could not fully answer: how to keep the latent-variable structure when the decoder is an arbitrary neural network.