Skip to main content

Ordinary Least Squares and Regularization

For the three points (0,1),(1,2),(2,2)(0,1),(1,2),(2,2), the best straight line is y=7/6+x/2y=7/6+x/2. This page shows how that line is computed, when no single best line can be determined, and which additional assumptions an uncertainty estimate for the coefficients needs. For the model, residuals, and step-by-step gradient updates, start with linear regression.

For design matrix XX, targets yy, and coefficient vector ww, ordinary least squares (OLS) solves

w^=arg⁡min⁡w∥Xw−y∥22.\hat{w}=\arg\min_w\|Xw-y\|_2^2.

From Residuals to a Solution​

Here XX has nn rows (observations) and pp columns (coefficients), y∈Rny\in\mathbb{R}^n, and w∈Rpw\in\mathbb{R}^p. Include a column of ones if an intercept is fitted. For a line y^i=b+axi\hat y_i=b+ax_i, the residual is ri=yi−y^ir_i=y_i-\hat y_i: a vertical difference in target units, not the shortest distance to the line. That perpendicular distance is ∣ri∣/1+a2|r_i|/\sqrt{1+a^2} in unscaled Cartesian coordinates; minimizing it would solve a different problem that also permits movement in xx.

Observed points connected vertically to a regression line, with a target y and prediction ŷ labeled.Open full-size image

For any point, keep x fixed and move vertically to the line: the point gives the observed y, and the line gives the predicted ŷ. Each blue segment shows the magnitude of a residual, rather than the perpendicular distance to the line. OLS chooses the line that minimizes the sum of these squared vertical differences. This D2L illustration uses its own data; the three-observation calculation below is a separate example.

Using J(w)=∥Xw−y∥22/(2n)J(w)=\|Xw-y\|_2^2/(2n) leaves the OLS minimizers unchanged and gives

∇J(w)=1nX⊤(Xw−y),∇2J(w)=1nX⊤X.\nabla J(w)=\frac{1}{n}X^\top(Xw-y),\qquad \nabla^2J(w)=\frac{1}{n}X^\top X.

The Hessian is positive semidefinite, so a zero gradient is a global minimum. Setting it to zero gives the normal equations:

X⊤Xw^=X⊤y.X^\top X\hat w=X^\top y.

If rank⁡(X)=p\operatorname{rank}(X)=p (thus n≥pn\geq p), the solution is unique and equals (X⊤X)−1X⊤y(X^\top X)^{-1}X^\top y. Otherwise coefficients are not unique: adding any vector in the null space of XX leaves the fitted values unchanged. The Moore–Penrose pseudoinverse selects the minimum-Euclidean-norm coefficient vector; the training fitted values are still unique. For example, duplicate columns identify only the sum of their coefficients. Predictions outside that column relationship need not agree.

The pseudoinverse gives a least-squares solution, w^=X+y\hat{w}=X^{+}y. When columns of XX are nearly linearly dependent, small changes in the data can produce large coefficient changes. Prediction may remain acceptable while individual coefficients become unstable.

Use a QR- or SVD-based least-squares solver rather than explicitly forming an inverse; forming X⊤XX^\top X squares the spectral condition number for full-column-rank XX. The scikit-learn linear-model guide describes OLS and its SVD solution.

Three Observations by Hand​

Take (x,y)=(0,1),(1,2),(2,2)(x,y)=(0,1),(1,2),(2,2). The intercept column and xx column are independent. With xˉ=1\bar x=1 and yˉ=5/3\bar y=5/3,

a=∑i(xi−xˉ)(yi−yˉ)∑i(xi−xˉ)2=12,b=yˉ−axˉ=76.a=\frac{\sum_i(x_i-\bar x)(y_i-\bar y)}{\sum_i(x_i-\bar x)^2} =\frac{1}{2},\qquad b=\bar y-a\bar x=\frac{7}{6}.

Predictions are (7/6,10/6,13/6)(7/6,10/6,13/6), residuals are (−1/6,1/3,−1/6)(-1/6,1/3,-1/6), and the squared-error sum is 1/61/6. Both ∑iri=0\sum_i r_i=0 and ∑ixiri=0\sum_i x_i r_i=0 check the normal equations. The training-mean baseline predicts 5/35/3 and has squared-error sum 2/32/3. OLS cuts the training squared-error sum from 2/32/3 to 1/61/6, a quarter of the baseline; whether it does as well on new data is checked on a held-out set.

from math import isclose
from statistics import mean

x, y = [0, 1, 2], [1, 2, 2]
xbar, ybar = mean(x), mean(y)
sxx = sum((v - xbar) ** 2 for v in x)
if sxx == 0:
raise ValueError("A constant x cannot identify slope and intercept separately")
a = sum((u - xbar) * (v - ybar) for u, v in zip(x, y)) / sxx
b = ybar - a * xbar
r = [v - (b + a * u) for u, v in zip(x, y)]
assert isclose(a, 0.5) and isclose(b, 7 / 6)
assert isclose(sum(v * v for v in r), 1 / 6)
assert isclose(sum(r), 0, abs_tol=1e-12)
assert isclose(sum(u * v for u, v in zip(x, r)), 0, abs_tol=1e-12)

Fitting Is Not Statistical Inference​

Computing the coefficients that minimize the residual sum of squares needs no assumption about the error distribution. Standard errors, confidence intervals and t/F tests need extra assumptions about the errors' mean, variance and correlation, and exact finite-sample t/F inference also needs Gaussian errors; the statsmodels regression documentation gives the formulas separately for each error covariance model.

Penalized Objectives​

Below, λ≥0\lambda\geq0 and 0≤ρ≤10\leq\rho\leq1. For these penalized formulas, use training-centered X,yX,y and let ww contain slopes only; recover the unpenalized intercept as yˉ−xˉ⊤w\bar y-\bar x^\top w. Scale features within training folds because coefficient penalties depend on units.

Ridge​

w^ridge=arg⁡min⁡w12n∥Xw−y∥22+λ∥w∥22.\hat{w}_{\text{ridge}} = \arg\min_w \frac{1}{2n}\|Xw-y\|_2^2 + \lambda\|w\|_2^2.

Ridge shrinks correlated or weakly identified coefficients and usually keeps all of them nonzero.

Lasso​

w^lasso=arg⁡min⁡w12n∥Xw−y∥22+λ∥w∥1.\hat{w}_{\text{lasso}} = \arg\min_w \frac{1}{2n}\|Xw-y\|_2^2 + \lambda\|w\|_1.

The ℓ1\ell_1 penalty can set coefficients exactly to zero, which performs feature selection. When two columns are strongly correlated, which one Lasso keeps can change with a different sample, so being selected shows neither that a column is uniquely important nor that it is causal.

Elastic Net​

w^EN=arg⁡min⁡w12n∥Xw−y∥22+λ[ρ∥w∥1+1−ρ2∥w∥22].\hat{w}_{\text{EN}} = \arg\min_w \frac{1}{2n}\|Xw-y\|_2^2 + \lambda\left[ \rho\|w\|_1 + \frac{1-\rho}{2}\|w\|_2^2 \right].

Elastic Net combines sparsity with Ridge-style stabilization and is often useful when predictors occur in correlated groups, though it does not guarantee that a whole group is kept or dropped together.

Matching scikit-learn's parameters​

Normalization is part of the definition. scikit-learn Ridge uses ∥Xw−y∥2+α∥w∥2\|Xw-y\|^2+\alpha\|w\|^2, so the Ridge formula above corresponds to α=2nλ\alpha=2n\lambda. scikit-learn's Lasso and Elastic Net conventions match α=λ\alpha=\lambda, with l1_ratio equal to ρ\rho. The Ridge term at ρ=0\rho=0 in the Elastic Net formula has half the strength of the Ridge formula above at the same λ\lambda.

Working Rules​

  • Fit transformations on training data only: learn scaling and basis transformations from the training set, then apply them unchanged to validation and test data; data splits and leakage explains why.
  • Intercept: do not penalize the intercept unless the formulation explicitly intends it.
  • Hyperparameters: select λ\lambda and ρ\rho using validation or cross-validation inside the training process.
  • Loss choice: when outliers are common and squared error does not reflect the error cost, use a robust loss or a quantile-regression objective.

Polynomial and interaction features can make the input relationship nonlinear while the fitted coefficients remain linear. The resulting model still inherits extrapolation and overfitting risks from the chosen basis.

See the scikit-learn linear-model guide for current solvers and estimator APIs.

Explore connectionsOpen network