Skip to main content

Machine Learning General Concepts

Model, Parameters, and Hyperparameters​

A model fθf_{\theta} maps an input xx to a prediction using learned parameters θ\theta. A hypothesis class is the set of functions reachable by the chosen representation. If you choose straight lines y=wx+by=wx+b, the hypothesis class is all straight lines; however you train, it can never draw a parabola. Hyperparameters—such as regularization strength, tree depth, or architecture width—control the learning procedure or hypothesis class rather than being fitted in the same inner optimization.

Loss and Empirical Risk​

A per-example loss measures disagreement between a prediction and target:

ℓ ⁣(fθ(xi),yi).\ell\!\left(f_{\theta}(x_i),y_i\right).

Training commonly minimizes empirical risk plus a regularizer:

J(θ)=1n∑i=1nℓ ⁣(fθ(xi),yi)+λ Ω(θ).J(\theta) = \frac{1}{n}\sum_{i=1}^{n} \ell\!\left(f_{\theta}(x_i),y_i\right) + \lambda\,\Omega(\theta).

Here nn is the number of training examples, and (xi,yi)(x_i,y_i) is the input–target pair for example ii. The average loss is the empirical risk: error measured on those observed examples. The regularizer Ω(θ)\Omega(\theta) penalizes properties of the parameters, such as large coefficients, and λ\lambda controls its weight relative to the average loss. JJ is often called the cost or objective. Two objectives both called “cost” can differ in data split, averaging convention, and penalty, so compare them item by item.

Common losses encode different assumptions:

TaskExample lossImportant boundary
regressionsquared errorstrongly weights large residuals
binary probabilistic classificationbinary cross-entropy / logistic lossexpects a probability model or logit interpretation
mutually exclusive multiclasssoftmax cross-entropynot the same as multilabel classification
margin classificationhinge lossoptimizes a margin, not calibrated probability by itself

Loss versus a Reported Metric​

Loss drives fitting; a metric judges a stated use on a stated split. They may be the same function, but need not be. For targets (0,0,1,1)(0,0,1,1), consider predicted positive-class probabilities:

ModelProbabilitiesAccuracy at threshold 0.5Mean log loss
A(0.1,0.4,0.6,0.9)(0.1,0.4,0.6,0.9)100%about 0.308
B(0.01,0.49,0.51,0.99)(0.01,0.49,0.51,0.99)100%about 0.342

Here log loss is −14∑i[yilog⁡pi+(1−yi)log⁡(1−pi)]-\frac14\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)], with natural logarithms. Accuracy hides B's weaker probabilities on the middle cases. Conversely, a lower log loss does not necessarily serve the business better: if reviewers can check only 100 cases a day, what matters is how many of the top 100 are right, and a lower log loss need not make those 100 more accurate. The scikit-learn metric guide distinguishes these evaluation targets.

With no useful features, a training-mean constant minimizes squared error, a training median minimizes absolute error, and a majority-class rule is an accuracy baseline. Estimate these constants on training data, then compare every model on the same held-out cases.

Likelihood​

For a probabilistic model pθ(y∣x)p_{\theta}(y\mid x), maximum likelihood chooses parameters that make the observations probable:

θ^ML=arg⁡max⁡θ∑ilog⁡pθ(yi∣xi).\hat{\theta}_{\mathrm{ML}} = \arg\max_{\theta} \sum_i \log p_{\theta}(y_i\mid x_i).

Minimizing negative log-likelihood is therefore equivalent to maximizing likelihood. This sum assumes observations are conditionally independent given their inputs and parameters. Dependent sequences require a joint or sequential likelihood instead.

Choosing a loss amounts to assuming how the data were generated. Let a regression target be yi=fθ(xi)+εiy_i=f_{\theta}(x_i)+\varepsilon_i, where the errors εi\varepsilon_i are independent Gaussian noise with a common fixed variance σ2\sigma^2. Each example's log-likelihood is then

log⁡pθ(yi∣xi)=−(yi−fθ(xi))22σ2−12log⁡ ⁣(2πσ2).\log p_{\theta}(y_i\mid x_i) =-\frac{\bigl(y_i-f_{\theta}(x_i)\bigr)^2}{2\sigma^2} -\frac12\log\!\left(2\pi\sigma^2\right).

The second term does not depend on the parameters θ\theta; the first is the squared error times a negative factor. Maximizing likelihood is therefore minimizing the sum of squared errors (up to a positive factor and a constant). Likewise, for Bernoulli observations with labels 0 or 1, the negative log-likelihood is exactly binary cross-entropy. The loss should therefore follow the observation model, not be chosen only because a library happens to offer it.

Optimization​

Gradient descent updates parameters in the direction opposite the objective gradient. The learning rate η>0\eta>0 sets the step size; too large a step can increase the objective or cause divergence:

θ←θ−η∇J(θ).\theta \leftarrow \theta-\eta\nabla J(\theta).

Computing each update from the full training set is batch gradient descent; estimating it from a small batch is minibatch gradient descent. In textbooks, stochastic gradient descent (SGD) means one example per update. PyTorch's torch.optim.SGD simply steps along the gradient of whatever loss you computed. In a training loop that loss is usually a minibatch average, so "SGD" there means minibatch updates. For a hand-worked update and why the learning rate has an upper limit, see the training process. However low the training objective goes, it shows only that the model fits the training examples; how it does on new data is a question of generalization.

Generalization Boundary​

Training error, validation error, and test error answer different questions: hyperparameter choices consume validation information, so the test set stays outside that loop. How to split and how to avoid leakage is covered in data splits and leakage.

Degree 1, 4, and 15 polynomial fits to the same noisy samples show underfitting, a suitable fit, and overfitting.Open full-size image

Compare the blue fitted curve with the orange data-generating function. Degree 1 misses the curvature; degree 4 follows it; degree 15 bends sharply around the noisy observations. The degree is a hyperparameter. The displayed MSE is measured by cross-validation, not training error: fitting the observed points more closely need not improve predictions on unseen points.

For a first model, continue with linear regression. For the mechanics shared by neural-network training, read gradients and optimizers; for a different hypothesis class, compare trees and boosting. Berkeley CS 189 provides the external course sequence.

Explore connectionsOpen network