Skip to main content

Ordinary Least Squares and Regularization

For design matrix XX, targets yy, and coefficient vector ww, ordinary least squares (OLS) solves

w^=arg⁡min⁡w∥Xw−y∥22.\hat{w}=\arg\min_w\|Xw-y\|_2^2.

The pseudoinverse gives a least-squares solution, w^=X+y\hat{w}=X^{+}y. When columns of XX are nearly linearly dependent, small changes in the data can produce large coefficient changes. Prediction may remain acceptable while individual coefficients become unstable.

Penalized Objectives​

Ridge​

w^ridge=arg⁡min⁡w12n∥Xw−y∥22+λ∥w∥22.\hat{w}_{\text{ridge}} = \arg\min_w \frac{1}{2n}\|Xw-y\|_2^2 + \lambda\|w\|_2^2.

Ridge shrinks correlated or weakly identified coefficients and usually keeps all of them nonzero.

Lasso​

w^lasso=arg⁡min⁡w12n∥Xw−y∥22+λ∥w∥1.\hat{w}_{\text{lasso}} = \arg\min_w \frac{1}{2n}\|Xw-y\|_2^2 + \lambda\|w\|_1.

The ℓ1\ell_1 penalty can set coefficients exactly to zero. That sparsity is a property of the fitted objective, not proof that the selected variables are uniquely important or causal. With strongly correlated features, the selected member may be unstable.

Elastic Net​

w^EN=arg⁡min⁡w12n∥Xw−y∥22+λ[ρ∥w∥1+1−ρ2∥w∥22].\hat{w}_{\text{EN}} = \arg\min_w \frac{1}{2n}\|Xw-y\|_2^2 + \lambda\left[ \rho\|w\|_1 + \frac{1-\rho}{2}\|w\|_2^2 \right].

Elastic Net combines sparsity with Ridge-style stabilization and is often useful when predictors occur in correlated groups.

Working Rules​

  • Fit scaling and basis transformations on training data only; apply the recorded transformation to validation and test data.
  • Do not penalize the intercept unless the formulation explicitly intends it.
  • Select λ\lambda and ρ\rho using validation or cross-validation inside the training process.
  • Compare predictive error, coefficient stability, and operational simplicity—not training loss alone.
  • Use robust or quantile objectives when squared error does not represent the desired target or error cost.
  • Separate predictive modeling from inferential claims; uncertainty estimates require explicit assumptions about the data-generating process.

Polynomial and interaction features can make the input relationship nonlinear while the fitted coefficients remain linear. The resulting model still inherits extrapolation and overfitting risks from the chosen basis.

See the scikit-learn linear-model guide for current solvers and estimator APIs.