Linear Regression: Modeling and Gradient Descent
This page develops the prediction model and iterative fitting. The companion ordinary least-squares note explains the closed-form solution, rank conditions, statistical uncertainty, and coefficient penalties.
Start with two dots and a straight line. The slope tilts the line; the intercept moves it up or down. Fitting means adjusting those two numbers to reduce prediction errors. Try it before reading the matrix notation.
Teach a line to fit two dots
The green dots are observations; the blue line is your prediction. Move the sliders to shorten the dashed errors, then let a gradient step do it for you. MSE is the average of the squared errors: smaller means a closer fit to these points.
ObservationsPrediction
MSE 5.0000
ŷ = 0.000x + (0.000)∇L [b, w] = [-4.000, -3.000]
Try this: reset, take one step at 0.1, then reset and try 1. A bigger step can make the error worse. Can you fit both dots yourself?
The controls stay between −3 and 4. A step outside that range is disabled; lower the learning rate or reset. The chart clips lines outside its vertical window. Playback stops after 60 steps, near zero error, or when this panel is hidden. Reduced-motion mode keeps manual steps available.Open the code · NumPy / JAX
The interaction above runs in JavaScript. These Python examples use the same data and equations, starting from the defaults—not your current controls. JAX can compute the loss gradient with jax.grad. JAX autodiff guide
In a new local project, save the example as demo.py. CPU setup: uv init ai-lab → cd ai-lab → uv add numpy jax → uv run demo.py
- NumPy
- JAX
import numpy as np
x = np.array([0., 1.])
y = np.array([1., 3.])
p = np.array([0., 0.]) # [b, w]
def loss(p):
return np.mean((p[1] * x + p[0] - y) ** 2)
def gradient(p):
error = p[1] * x + p[0] - y
return np.array([2 * error.mean(), 2 * (error * x).mean()])
p = p - 0.1 * gradient(p)
print(p, loss(p)) # [0.4, 0.3], 2.825
import jax
import jax.numpy as jnp
x = jnp.array([0., 1.])
y = jnp.array([1., 3.])
p = jnp.array([0., 0.]) # [b, w]
def loss(p):
return jnp.mean((p[1] * x + p[0] - y) ** 2)
gradient = jax.grad(loss)
p = p - 0.1 * gradient(p)
print(p, loss(p)) # approximately [0.4, 0.3], 2.825
Linear regression predicts a numerical target as an affine function of the features (a linear function plus an intercept; strictly, the version with an intercept is called affine):
For observations, collect the rows into a design matrix . If the intercept is represented by an added column of ones, the model becomes
The word linear refers to the parameters. Features may include transformations such as interactions or polynomial terms while the model remains linear in . Once the model is trained, inference only evaluates ; it does not use the target or update weights.
Squared-Error Objective
Ordinary least squares minimizes the residual sum of squares, or equivalently mean squared error:
Gradient descent reaches the same least-squares objective through successive updates. It is especially useful for large datasets and models embedded in a wider differentiable system.
From Residuals to an Update
With features and an intercept, has shape and has entries. Differentiating the MSE above gives
The learning rate sets the update size. For points , start at . Prediction-minus-target errors are , so the gradient in order is . With , the new parameters are and MSE falls from to . The exact fit is .
Scale and step size
Changing a feature from meters to millimeters multiplies its numerical values by 1,000. At a point where the predictions are unchanged, that feature's gradient grows 1,000-fold too; the loss's curvature along that direction is proportional to the square of the feature values, so it grows by , a factor of one million. A learning rate that used to be stable must shrink by roughly a million as well. The other features are unchanged, so a single learning rate can hardly suit them all. Fit centering and scaling on training data, then reuse those statistics at prediction time. Scaling changes only the path optimization takes; whether a linear relationship is appropriate is judged from residuals and held-out data.
The training-process note derives a learning-rate stability bound for a small quadratic example and connects this update to minibatches, momentum, and validation. Reducing training loss is an optimization result; compare held-out error against a training-mean baseline before claiming useful prediction.
Interpretation Boundaries
A coefficient's meaning depends on which other features the model includes and on their scaling. When house price is predicted from floor area alone, the area coefficient also absorbs the fact that larger houses tend to have more rooms; add the number of rooms to the model and the area coefficient keeps only the effect of one more square meter with the room count held fixed, so its value changes. Before reading a coefficient, check which features it was estimated alongside.
Extrapolation outside the observed feature range rests entirely on the chosen functional form. In the two-point example above, lies only between 0 and 1 and the fitted line is ; at it predicts . Whether that is right depends on whether the relationship stays a straight line beyond 0 to 1, which the data themselves cannot say.
The closed-form solution, why collinearity makes coefficients non-unique or unstable, the extra assumptions needed for coefficient uncertainty, and ridge and lasso are covered in Ordinary Least Squares.
Linear regression is valuable both as a model and as a baseline. A more complex method should justify itself against this simpler alternative.
Continue with Softmax Regression for mutually exclusive classification. The main external reference is Dive into Deep Learning: Linear Neural Networks for Regression.