Multilayer Perceptron
A hidden layer builds intermediate features before producing an answer. In this tiny network, two neurons measure how the inputs differ. Switch their nonlinearity off to see why adding layers alone is not enough.
Why does XOR need a hidden layer?
XOR means “one input is 1, but not both”. Toggle the inputs, then remove ReLU: the same weights stop solving the problem.
[0, 1] → h = [0, 1] → Output = 1
ReLU keeps positive differences and clips negative ones to zero. Adding the two hidden values detects unequal inputs.
These weights are chosen by hand, not trained. This demonstrates a working representation, not a guarantee that training will find it.Compare all four inputs
| x₁ | x₂ | Target | Output |
|---|---|---|---|
| 0 | 0 | 0 | 0 |
| 0 | 1 | 1 | 1 |
| 1 | 0 | 1 | 1 |
| 1 | 1 | 0 | 0 |
Open the code · NumPy / JAX
The interaction above runs in JavaScript. These Python examples use the same data and equations, starting from the defaults—not your current controls. JAX can compute the loss gradient with jax.grad. JAX autodiff guide
In a new local project, save the example as demo.py. CPU setup: uv init ai-lab → cd ai-lab → uv add numpy jax → uv run demo.py
import jax.numpy as jnp
# NumPy: replace the import above with: import numpy as jnp
x = jnp.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
w1 = jnp.array([[1., -1.], [-1., 1.]])
w2 = jnp.array([1., 1.])
h = jnp.maximum(x @ w1, 0.)
print(h @ w2) # [0., 1., 1., 0.]
print((x @ w1) @ w2) # [0., 0., 0., 0.]: no ReLU
Each layer of an MLP does two things: an affine map, which multiplies by a matrix and adds a vector, followed by an elementwise nonlinearity applied to each entry separately. For input , hidden width , and output width , a network with one hidden layer is
with and . The nonlinearity is what makes depth count. Without it, two layers are , which is still a single affine map; however many layers you stack, they represent the same functions as one. That is why the experiment above outputs 0 for all four inputs once ReLU is switched off.
Two ReLUs Compute XOR
XOR must output 1 for and and 0 for and . Plotted in the plane, the two points that should give 1 lie on one diagonal and the two that should give 0 lie on the other, so no single straight line puts them on opposite sides. The network in the experiment computes, for ,
When the inputs are equal ( or ), both differences are 0 and the output is 0. When they differ ( or ), exactly one difference is 1 and the output is 1. The hidden layer first represents whether the inputs differ, and the output layer only adds that feature up. With ReLU switched off, is always 0, and the intermediate feature disappears.
What Backpropagation Computes
Training needs to know which way, and how far, to move each parameter: the gradient of the loss with respect to that parameter. For one loss , let . The chain rule computes the gradients layer by layer, from the output back:
In words: the output error times that layer's input is the gradient for . The error travels back through to the hidden layer and is multiplied by the activation's slope at that point, giving the hidden error . Then times the input is the gradient for .
One Scalar Backward Pass
Take , , , , , target , and . Each step uses one rule:
All derivatives use the same forward-pass weights; updating before computing would differentiate a different computation.
This is the scalar network from the text, o = w₂·ReLU(w₁x + b₁) + b₂ with loss L = ½(o − y)². Step forward to compute every value, then backward: each node multiplies the gradient arriving from the loss by its local derivative and passes it on. Change any input or weight and both passes update.
The last product gives the first weight its gradient. Each parameter's gradient is the product of the local derivatives along its path back from L.
∂L/∂w₁ = ∂L/∂u · x = 6 × 2 = 12; ∂L/∂x = ∂L/∂u · w₁ = 6 × 1 = 6
Parameter gradients
∂L/∂w₁ = 12∂L/∂b₁ = 6∂L/∂w₂ = 2∂L/∂b₂ = 2
Check: nudge w₁ by ±0.00001 and measure how L changes. The slope is 12; backpropagation gives 12.
ReLU′(0) is taken as 0, as in PyTorch. A real network applies the same rule to vectors and matrices.
Backpropagation computes gradients; an optimizer applies the update. For a mean minibatch loss, average per-example contributions, and clear accumulated gradients before the next independent update. Inference needs only the forward pass. If dropout is present, training randomly suppresses activations (with compensating scaling in inverted dropout), while evaluation disables this randomness. A plain affine/ReLU MLP has no such mode-dependent layer.
Autodiff removes hand-coded derivatives, but shapes, numerical stability, and the objective's assumptions still need checking. A zero gradient may indicate an optimum, Sigmoid/Tanh saturation, a dead ReLU, an incorrect mask, or a disconnected graph.
The Task Chooses the Output Head
The hidden activation builds intermediate features; the output layer's form follows from the question being answered. Decide which kind of task it is, then choose the head and loss, rather than selecting a library loss first and retrofitting the question:
Whether the hidden layers should use ReLU, GELU, SiLU, or SwiGLU with its two projected branches, and how to compare activations fairly, is covered in Activations and Gated Feed-Forward Networks.
Capacity Is Not Learnability
Cybenko's theorem states that finite sums of units with a continuous sigmoidal activation can uniformly approximate any continuous function on a compact cube to any positive error tolerance. The required finite width may grow with the function and tolerance; the theorem does not give a fixed width that suffices for every target.
The XOR example and Cybenko's theorem both show that a representation exists, not that training will reach it. Neither guarantees:
- sufficient training data;
- a parameter- or sample-efficient representation;
- that SGD or another optimizer finds the representation from arbitrary initialization;
- generalization under noisy data;
- calibrated output probabilities;
- causal validity or out-of-distribution (OOD) generalization.
“Universal approximator” is therefore a theoretical property, not a model-selection result.
An MLP Assumes No Structure in the Data
An inductive bias is an assumption the architecture makes for you in advance; a CNN, for example, assumes that neighboring pixels are related. An MLP makes almost none: it sees a flat vector, so translation, sequence order, graph structure, and permutation invariance do not appear automatically. CNNs, RNNs, attention, and graph networks encode stronger sharing or computation patterns and can change sample efficiency.
Conversely, on stable tabular features with limited data, a linear model, tree, or small MLP may beat a fashionable architecture. Compare them under the same split and budget.
Minimal Acceptance
- Assert batch, feature, and output shapes.
- Overfit a tiny sample to test that the implementation can learn.
- Compare against constant, linear, or tree baselines on the same split, and report several seeds rather than only the best run.
Extend the XOR example in TensorFlow Playground: choose a nonlinear dataset, change hidden-layer width or activation, and watch the decision boundary during training. Then increase noise and compare training with test loss; a more flexible boundary can fit noise as well as structure.