Ordinary Least Squares and Regularization
For the three points , the best straight line is . This page shows how that line is computed, when no single best line can be determined, and which additional assumptions an uncertainty estimate for the coefficients needs. For the model, residuals, and step-by-step gradient updates, start with linear regression.
For design matrix , targets , and coefficient vector , ordinary least squares (OLS) solves
From Residuals to a Solution
Here has rows (observations) and columns (coefficients), , and . Include a column of ones if an intercept is fitted. For a line , the residual is : a vertical difference in target units, not the shortest distance to the line. That perpendicular distance is in unscaled Cartesian coordinates; minimizing it would solve a different problem that also permits movement in .
For any point, keep x fixed and move vertically to the line: the point gives the observed y, and the line gives the predicted ŷ. Each blue segment shows the magnitude of a residual, rather than the perpendicular distance to the line. OLS chooses the line that minimizes the sum of these squared vertical differences. This D2L illustration uses its own data; the three-observation calculation below is a separate example.
Using leaves the OLS minimizers unchanged and gives
The Hessian is positive semidefinite, so a zero gradient is a global minimum. Setting it to zero gives the normal equations:
If (thus ), the solution is unique and equals . Otherwise coefficients are not unique: adding any vector in the null space of leaves the fitted values unchanged. The Moore–Penrose pseudoinverse selects the minimum-Euclidean-norm coefficient vector; the training fitted values are still unique. For example, duplicate columns identify only the sum of their coefficients. Predictions outside that column relationship need not agree.
The pseudoinverse gives a least-squares solution, . When columns of are nearly linearly dependent, small changes in the data can produce large coefficient changes. Prediction may remain acceptable while individual coefficients become unstable.
Use a QR- or SVD-based least-squares solver rather than explicitly forming an inverse; forming squares the spectral condition number for full-column-rank . The scikit-learn linear-model guide describes OLS and its SVD solution.
Three Observations by Hand
Take . The intercept column and column are independent. With and ,
Predictions are , residuals are , and the squared-error sum is . Both and check the normal equations. The training-mean baseline predicts and has squared-error sum . OLS cuts the training squared-error sum from to , a quarter of the baseline; whether it does as well on new data is checked on a held-out set.
from math import isclose
from statistics import mean
x, y = [0, 1, 2], [1, 2, 2]
xbar, ybar = mean(x), mean(y)
sxx = sum((v - xbar) ** 2 for v in x)
if sxx == 0:
raise ValueError("A constant x cannot identify slope and intercept separately")
a = sum((u - xbar) * (v - ybar) for u, v in zip(x, y)) / sxx
b = ybar - a * xbar
r = [v - (b + a * u) for u, v in zip(x, y)]
assert isclose(a, 0.5) and isclose(b, 7 / 6)
assert isclose(sum(v * v for v in r), 1 / 6)
assert isclose(sum(r), 0, abs_tol=1e-12)
assert isclose(sum(u * v for u, v in zip(x, r)), 0, abs_tol=1e-12)
Fitting Is Not Statistical Inference
Computing the coefficients that minimize the residual sum of squares needs no assumption about the error distribution. Standard errors, confidence intervals and t/F tests need extra assumptions about the errors' mean, variance and correlation, and exact finite-sample t/F inference also needs Gaussian errors; the statsmodels regression documentation gives the formulas separately for each error covariance model.
Penalized Objectives
Below, and . For these penalized formulas, use training-centered and let contain slopes only; recover the unpenalized intercept as . Scale features within training folds because coefficient penalties depend on units.
Ridge
Ridge shrinks correlated or weakly identified coefficients and usually keeps all of them nonzero.
Lasso
The penalty can set coefficients exactly to zero, which performs feature selection. When two columns are strongly correlated, which one Lasso keeps can change with a different sample, so being selected shows neither that a column is uniquely important nor that it is causal.
Elastic Net
Elastic Net combines sparsity with Ridge-style stabilization and is often useful when predictors occur in correlated groups, though it does not guarantee that a whole group is kept or dropped together.
Matching scikit-learn's parameters
Normalization is part of the definition. scikit-learn Ridge uses , so the Ridge formula above corresponds to . scikit-learn's Lasso and Elastic Net conventions match , with l1_ratio equal to . The Ridge term at in the Elastic Net formula has half the strength of the Ridge formula above at the same .
Working Rules
- Fit transformations on training data only: learn scaling and basis transformations from the training set, then apply them unchanged to validation and test data; data splits and leakage explains why.
- Intercept: do not penalize the intercept unless the formulation explicitly intends it.
- Hyperparameters: select and using validation or cross-validation inside the training process.
- Loss choice: when outliers are common and squared error does not reflect the error cost, use a robust loss or a quantile-regression objective.
Polynomial and interaction features can make the input relationship nonlinear while the fitted coefficients remain linear. The resulting model still inherits extrapolation and overfitting risks from the chosen basis.
See the scikit-learn linear-model guide for current solvers and estimator APIs.