Machine Learning General Concepts
Model, Parameters, and Hyperparameters
A model maps an input to a prediction using learned parameters . A hypothesis class is the set of functions reachable by the chosen representation. If you choose straight lines , the hypothesis class is all straight lines; however you train, it can never draw a parabola. Hyperparameters—such as regularization strength, tree depth, or architecture width—control the learning procedure or hypothesis class rather than being fitted in the same inner optimization.
Loss and Empirical Risk
A per-example loss measures disagreement between a prediction and target:
Training commonly minimizes empirical risk plus a regularizer:
Here is the number of training examples, and is the input–target pair for example . The average loss is the empirical risk: error measured on those observed examples. The regularizer penalizes properties of the parameters, such as large coefficients, and controls its weight relative to the average loss. is often called the cost or objective. Two objectives both called “cost” can differ in data split, averaging convention, and penalty, so compare them item by item.
Common losses encode different assumptions:
Loss versus a Reported Metric
Loss drives fitting; a metric judges a stated use on a stated split. They may be the same function, but need not be. For targets , consider predicted positive-class probabilities:
Here log loss is , with natural logarithms. Accuracy hides B's weaker probabilities on the middle cases. Conversely, a lower log loss does not necessarily serve the business better: if reviewers can check only 100 cases a day, what matters is how many of the top 100 are right, and a lower log loss need not make those 100 more accurate. The scikit-learn metric guide distinguishes these evaluation targets.
With no useful features, a training-mean constant minimizes squared error, a training median minimizes absolute error, and a majority-class rule is an accuracy baseline. Estimate these constants on training data, then compare every model on the same held-out cases.
Likelihood
For a probabilistic model , maximum likelihood chooses parameters that make the observations probable:
Minimizing negative log-likelihood is therefore equivalent to maximizing likelihood. This sum assumes observations are conditionally independent given their inputs and parameters. Dependent sequences require a joint or sequential likelihood instead.
Choosing a loss amounts to assuming how the data were generated. Let a regression target be , where the errors are independent Gaussian noise with a common fixed variance . Each example's log-likelihood is then
The second term does not depend on the parameters ; the first is the squared error times a negative factor. Maximizing likelihood is therefore minimizing the sum of squared errors (up to a positive factor and a constant). Likewise, for Bernoulli observations with labels 0 or 1, the negative log-likelihood is exactly binary cross-entropy. The loss should therefore follow the observation model, not be chosen only because a library happens to offer it.
Optimization
Gradient descent updates parameters in the direction opposite the objective gradient. The learning rate sets the step size; too large a step can increase the objective or cause divergence:
Computing each update from the full training set is batch gradient descent; estimating it from a small batch is minibatch gradient descent. In textbooks, stochastic gradient descent (SGD) means one example per update. PyTorch's torch.optim.SGD simply steps along the gradient of whatever loss you computed. In a training loop that loss is usually a minibatch average, so "SGD" there means minibatch updates. For a hand-worked update and why the learning rate has an upper limit, see the training process. However low the training objective goes, it shows only that the model fits the training examples; how it does on new data is a question of generalization.
Generalization Boundary
Training error, validation error, and test error answer different questions: hyperparameter choices consume validation information, so the test set stays outside that loop. How to split and how to avoid leakage is covered in data splits and leakage.
Open full-size imageCompare the blue fitted curve with the orange data-generating function. Degree 1 misses the curvature; degree 4 follows it; degree 15 bends sharply around the noisy observations. The degree is a hyperparameter. The displayed MSE is measured by cross-validation, not training error: fitting the observed points more closely need not improve predictions on unseen points.
For a first model, continue with linear regression. For the mechanics shared by neural-network training, read gradients and optimizers; for a different hypothesis class, compare trees and boosting. Berkeley CS 189 provides the external course sequence.