Machine Learning General Concepts

Model, Parameters, and Hyperparameters

A model $f_{\theta}$ maps an input $x$ to a prediction using learned parameters $\theta$. A hypothesis class is the set of functions reachable by the chosen representation. Hyperparameters—such as regularization strength, tree depth, or architecture width—control the learning procedure or hypothesis class rather than being fitted in the same inner optimization.

Loss and Empirical Risk

A per-example loss measures disagreement between a prediction and target:

$$\ell\!\left(f_{\theta}(x_i),y_i\right).$$

Training commonly minimizes empirical risk plus a regularizer:

$$J(\theta) = \frac{1}{n}\sum_{i=1}^{n} \ell\!\left(f_{\theta}(x_i),y_i\right) + \lambda\,\Omega(\theta).$$

The terms cost and objective are often used for $J$. The distinction matters less than stating the exact aggregation, normalization, regularization, and data split.

Common losses encode different assumptions:

Task	Example loss	Important boundary
regression	squared error	strongly weights large residuals
binary probabilistic classification	binary cross-entropy / logistic loss	expects a probability model or logit interpretation
mutually exclusive multiclass	softmax cross-entropy	not the same as multilabel classification
margin classification	hinge loss	optimizes a margin, not calibrated probability by itself

Likelihood

For a probabilistic model $p_{\theta}(y\mid x)$, maximum likelihood chooses parameters that make the observations probable:

$$\hat{\theta}_{\mathrm{ML}} = \arg\max_{\theta} \sum_i \log p_{\theta}(y_i\mid x_i).$$

Minimizing negative log-likelihood is therefore equivalent to maximizing likelihood. The chosen likelihood determines the corresponding loss; it should follow the observation model rather than be selected only because a library exposes it.

Optimization

Gradient descent updates parameters in the direction opposite the objective gradient:

$$\theta \leftarrow \theta-\eta\nabla J(\theta).$$

• Batch gradient descent uses the full training set per update.
• Stochastic gradient descent uses one sampled example.
• Minibatch methods estimate the gradient from a small batch and are the usual computational compromise.

Newton's method uses local curvature:

$$\theta \leftarrow \theta-\left[\nabla^2J(\theta)\right]^{-1}\nabla J(\theta).$$

It can converge quickly near a well-behaved optimum, but forming or solving with the Hessian can be expensive or unstable. An optimizer finding a low training objective does not prove good generalization.

Generalization Boundary

Training error, validation error, and test error answer different questions. Hyperparameter choices consume validation information; the test set should remain outside that loop. Evaluation must also consider distribution shift, leakage, subgroup behavior, uncertainty, and the cost of different errors.

Continue to the Linear Models map. Berkeley CS 189 provides the primary external course sequence.