Linear Regression

Linear regression predicts a numerical target as an affine function of the features:

$$\hat{y} = \mathbf{x}^{\top}\mathbf{w} + b.$$

For $n$ observations, collect the rows into a design matrix $\mathbf{X}$. If the intercept is represented by an added column of ones, the model becomes

$$\hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\theta}.$$

The word linear refers to the parameters. Features may include transformations such as interactions or polynomial terms while the model remains linear in $\boldsymbol{\theta}$.

Squared-Error Objective

Ordinary least squares minimizes the residual sum of squares, or equivalently mean squared error:

$$\hat{\boldsymbol{\theta}} = \arg\min_{\boldsymbol{\theta}} \frac{1}{n}\left\|\mathbf{X}\boldsymbol{\theta}-\mathbf{y}\right\|_2^2.$$

When a unique inverse-based solution does not exist, the Moore–Penrose pseudoinverse gives a least-squares solution:

$$\hat{\boldsymbol{\theta}} = \mathbf{X}^{+}\mathbf{y}.$$

Gradient-based optimization is useful when the dataset is large, the model is embedded in a wider differentiable system, or regularization changes the objective.

Interpretation Boundaries

• A coefficient is conditional on the included features and their scaling.
• Good prediction does not establish a causal relationship.
• Statistical inference needs assumptions beyond minimizing squared error; prediction and inference are different goals.
• Collinearity can make coefficients unstable even when predictions remain adequate.
• Extrapolation outside the observed feature range relies heavily on the chosen functional form.
• Residual plots and out-of-sample error reveal failures that training loss cannot.

Linear regression is valuable both as a model and as a baseline. A more complex method should justify itself against this simpler alternative.

Continue with Softmax Regression for mutually exclusive classification. The main external reference is Dive into Deep Learning: Linear Neural Networks for Regression.