Applications to Linear Models

Fitting a formula to data is the most common use of least squares. A relationship $y={\beta }_0+{\beta }_{1}x$ is posited, the data rarely fall exactly on any such line, and the parameters ${\beta }_0,{\beta }_1$ are chosen to make the fit as good as possible. Written in matrix form the problem is least squares with statistical notation: $X\mathbf{\beta }=\mathbf{y}$, where $X$ is the design matrix, $\mathbf{\beta }$ the parameter vector, and $\mathbf{y}$ the observation vector.

The least-squares line

Given data points $(x_1,y_1),\dots ,(x_n,y_n)$, the line $y={\beta }_0+{\beta }_{1}x$ assigns to each $x_j$ a predicted value ${\beta }_0+{\beta }_1{x}_j$. The gap between the observed $y_j$ and the predicted value is the residual.

If the points were exactly on the line, the parameters would satisfy $n$ equations ${\beta }_0+{\beta }_1{x}_j=y_j$, i.e. $X\mathbf{\beta }=\mathbf{y}$ with

The squared distance $\Vert X\mathbf{\beta }-\mathbf{y}{\Vert }^2$ is the sum of squared residuals, so the least-squares $\mathbf{\beta }$ solves the normal equations $X^{\top }X\mathbf{\beta }=X^{\top }\mathbf{y}$.

A common preprocessing step subtracts the mean $\bar{x}$ from every $x_j$, forming $x^{\ast }=x-\bar{x}$. In this mean-deviation form the two columns of $X$ become orthogonal, $X^{\top }X$ is diagonal, and the normal equations decouple into two independent one-line solves.

The general linear model

Statisticians write the fit with an explicit residual vector$\mathbf{ϵ}=\mathbf{y}-X\mathbf{\beta }$, so

Any equation of this form is a linear model, and the least-squares fit minimizes $\Vert \mathbf{ϵ}\Vert$. The word linear refers to the parameters ${\beta }_i$, not the shape of the fitted curve. A model is linear as long as it has the form

for known functions $f_0,\dots ,f_k$. Changing the $f_i$ changes the columns of $X$ but leaves the normal equations unchanged.

Polynomial fits

If the data curve rather than run straight, fit a polynomial. For a parabola $y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2$, each point contributes an equation $y_j={\beta }_0+{\beta }_1{x}_j+{\beta }_2{x}_j^2+{ϵ}_j$, and the design matrix gains a squared column:

A cubic $y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+{\beta }_3{x}^3$ adds a fourth column of cubes. The columns are the sampled basis functions $1,x,x^2,x^3$; the model stays linear in $\mathbf{\beta }$ because those functions are known once the $x_j$ are fixed.

Multiple regression

With two predictors $u$ and $v$ and a response $y$, a plane $y={\beta }_0+{\beta }_{1}u+{\beta }_{2}v$ fits data $(u_1,v_1,y_1),\dots ,(u_n,v_n,y_n)$. Each observation gives $y_j={\beta }_0+{\beta }_1{u}_j+{\beta }_2{v}_j+{ϵ}_j$, and the model is $\mathbf{y}=X\mathbf{\beta }+\mathbf{ϵ}$ with

The least-squares solution is the least-squares plane. Cross terms and higher powers keep the model linear: an equation like $y={\beta }_0+{\beta }_{1}u+{\beta }_{2}v+{\beta }_3{u}^2+{\beta }_{4}uv+{\beta }_5{v}^2$ (a trend surface) is still linear in $\mathbf{\beta }$, with columns given by the sampled functions $1,u,v,u^2,uv,v^2$. Whatever the predictors, the fit obeys the same normal equations, and whenever $X^{\top }X$ is invertible,

Splitting the sum of squares

The projection geometry gives a bookkeeping identity used throughout regression. The residual vector $\mathbf{ϵ}=\mathbf{y}-X\hat{\mathbf{\beta }}$ is orthogonal to $\mathrm{Col}X$, and $X\hat{\mathbf{\beta }}$ lies in it, so the Pythagorean theorem splits the total variation of $\mathbf{y}$:

The total sum of squares (SS(T)) divides into the part explained by the fit (SS(R), the regression sum of squares) and the part left over (SS(E), the error sum of squares). This is the orthogonal decomposition of $\mathbf{y}$ into $\mathrm{Col}X$ and its complement, read as squared lengths, and it underlies the analysis of variance.

The design-matrix pattern

The examples differ only in how the columns of $X$ are formed. A line samples $\{1,x\}$; a parabola samples $\{1,x,x^2\}$; a plane samples $\{1,u,v\}$. Once $X$ is built, the parameter vector solves the same normal equations, and linear algebra handles all of them at once.

Building a design matrix and solving the normal equations covers straight-line regression, polynomial curve fitting, and multiple regression with one procedure. Replacing the dot product by an abstract inner product extends the same least-squares idea to approximating functions, not just data points.¹