Skip to main content

Softmax Regression

Softmax regression is a linear model for choosing one class from KK mutually exclusive classes. It produces one unnormalized score, or logit, per class:

o=W⊤x+b.\mathbf{o} = \mathbf{W}^{\top}\mathbf{x} + \mathbf{b}.

Softmax converts the logits into nonnegative values that sum to one:

p(y=k∣x)=exp⁡(ok)∑j=1Kexp⁡(oj).p(y=k\mid\mathbf{x}) = \frac{\exp(o_k)}{\sum_{j=1}^{K}\exp(o_j)}.

Implementations subtract the largest logit before exponentiation for numerical stability; this does not change the result because softmax is invariant to adding the same constant to every logit.

Cross-Entropy​

For a one-hot target y\mathbf{y}, the per-example cross-entropy is

ℓ(y,p)=−∑k=1Kyklog⁡pk=−log⁡py.\ell(\mathbf{y},\mathbf{p}) = -\sum_{k=1}^{K} y_k\log p_k = -\log p_{y}.

Minimizing this objective corresponds to maximizing the conditional likelihood of the observed class labels under the model.

Boundaries​

  • Mutually exclusive multiclass: softmax is appropriate when exactly one class is correct.
  • Multilabel: use independent outputs, commonly sigmoid probabilities with a binary loss, when several labels may be true simultaneously.
  • Decision rule: argmax chooses a class, but asymmetric costs may require a different rule.
  • Probability quality: good classification accuracy does not imply calibrated probabilities.
  • Linear boundary: nonlinear feature relationships require transformed features or a more expressive model.

Evaluate against class frequencies and a simple baseline, then inspect a confusion matrix and per-class errors. Aggregate accuracy can hide poor behavior on rare or important classes.

Continue with the Multilayer Perceptron to introduce nonlinear hidden representations. See Dive into Deep Learning: Linear Neural Networks for Classification for derivations and maintained implementations.