Classification with a Neural Network

A multilayer neural network composes affine transformations with nonlinear activation functions. The nonlinearity lets it represent decision boundaries that a single logistic unit cannot.

One-hidden-layer model

For an input vector $x$, hidden weights $W^{(1)}$, hidden bias $b^{(1)}$, output weights $W^{(2)}$, and output bias $b^{(2)}$:

$$\begin{aligned} z^{(1)} &= W^{(1)}x+b^{(1)},\\ a^{(1)} &= \phi\left(z^{(1)}\right),\\ z^{(2)} &= W^{(2)}a^{(1)}+b^{(2)},\\ \hat y &= \sigma\left(z^{(2)}\right). \end{aligned}$$

Here $\phi$ may be ReLU, GELU, tanh, or another hidden activation. For binary classification, the sigmoid output $\hat y$ is interpreted as a predicted probability and can be trained with binary cross-entropy:

$$L=-\left[y\log \hat y+(1-y)\log(1-\hat y)\right].$$

Backpropagation

Backpropagation applies the chain rule from the loss toward earlier layers. With sigmoid plus binary cross-entropy, the output pre-activation gradient simplifies to

$$\delta^{(2)}=\frac{\partial L}{\partial z^{(2)}}=\hat y-y.$$

The output-layer gradients are

$$\frac{\partial L}{\partial W^{(2)}}=\delta^{(2)}(a^{(1)})^T, \qquad \frac{\partial L}{\partial b^{(2)}}=\delta^{(2)}.$$

Propagating through the hidden activation gives

$$\delta^{(1)}=(W^{(2)})^T\delta^{(2)}\odot\phi'\left(z^{(1)}\right),$$

and therefore

$$\frac{\partial L}{\partial W^{(1)}}=\delta^{(1)}x^T, \qquad \frac{\partial L}{\partial b^{(1)}}=\delta^{(1)}.$$

The symbol $\odot$ denotes elementwise multiplication. Shapes matter: writing the computation in matrix form avoids an ambiguous chain of scalar subscripts.

Parameter update

Plain gradient descent with learning rate $\alpha$ updates each parameter $\theta$ by

$$\theta\leftarrow\theta-\alpha\frac{\partial L}{\partial\theta}.$$

Practical training usually operates on mini-batches and may use Adam or another optimizer, but the gradients still come from the same forward graph and chain rule.

In TensorFlow Playground, choose a classification dataset, keep only the two coordinate inputs, and compare a network with no hidden layer against one with a hidden layer. Watch the decision boundary change as training proceeds; the playground illustrates representation and training rather than reproducing this note’s exact calculation.

What this note leaves out

This small network is enough to show the calculation, not enough to choose a production architecture. Initialization, normalization, regularization, optimizer state, multiclass outputs, numerical stability, and data quality all affect training. The maintained multilayer perceptron note covers the broader architecture.

Shapes and a complete backward calculation

For $d$ input features and $h$ hidden units, use column vectors: $x\in\mathbb R^d$, $W^{(1)}\in\mathbb R^{h\times d}$, $b^{(1)},a^{(1)},\delta^{(1)}\in\mathbb R^h$, $W^{(2)}\in\mathbb R^{1\times h}$, and scalar $b^{(2)},z^{(2)},\delta^{(2)}$. Each weight gradient has the shape of its weight matrix. The outer product $\delta^{(1)}x^T$ is therefore $h\times d$.

For a minimal numerical example, let $d=h=1$, $x=2$, $y=1$, and use ReLU, $\phi(t)=\max(0,t)$. Set $W^{(1)}=0.5$, $b^{(1)}=0$, $W^{(2)}=1$, $b^{(2)}=0$. The forward values are $z^{(1)}=a^{(1)}=z^{(2)}=1$, $\hat y\approx0.731059$, and $L\approx0.313262$. Since the hidden input is positive, $\phi'=1$, giving

$$\delta^{(2)}=\delta^{(1)}\approx-0.268941,\qquad \left(\frac{\partial L}{\partial W^{(1)}},\frac{\partial L}{\partial b^{(1)}},\frac{\partial L}{\partial W^{(2)}},\frac{\partial L}{\partial b^{(2)}}\right) \approx(-0.537883,-0.268941,-0.268941,-0.268941).$$

With $\alpha=0.1$, updating all four parameters from this same forward pass yields approximately $(0.553788,0.026894,1.026894,0.026894)$ in that order. The next logit is about $1.191875$, and the loss is about $0.265169$. Updating the output weights before computing $\delta^{(1)}$ would mix two different parameter states and is not this gradient step.

At ReLU input zero, the ordinary derivative does not exist. PyTorch uses the minimum-norm subgradient, zero, as its backward convention; see Autograd mechanics. Away from such kinks, check an analytic gradient against $[L(\theta+\varepsilon e_j)-L(\theta-\varepsilon e_j)]/(2\varepsilon)$ on a tiny example. This tests differentiation, not whether training will find a global optimum.

What the optimization loop can establish

Backpropagation computes derivatives; the optimizer chooses parameter changes. For a batch loss, average the per-example gradients. Mini-batch gradients fluctuate, so every step need not decrease full training loss. Jointly learning hidden and output weights generally makes the objective nonconvex, unlike fixed-feature logistic regression. Different initializations can lead to different stationary regions; a small update may also reflect a tiny learning rate, saturated activations, or inactive ReLUs.

Use finite-value checks, an iteration budget, gradient and loss diagnostics, and held-out validation for early stopping. Validation-based stopping limits overfitting; it is not an optimality certificate. Compute binary cross-entropy from logits using the stable log-loss expression. These distinctions between optimization and generalization are developed in Deep Learning, Chapter 8.

My reading · Opened notes are remembered in this browser.