Skip to main content

Multilayer Perceptron

A hidden layer builds intermediate features before producing an answer. In this tiny network, two neurons measure how the inputs differ. Switch their nonlinearity off to see why adding layers alone is not enough.

Try it yourself

Why does XOR need a hidden layer?

XOR means “one input is 1, but not both”. Toggle the inputs, then remove ReLU: the same weights stop solving the problem.

Two inputs, two hidden units and one output with fixed weights[0,1] → [0,1] → 1InputsReLUOutput+1−1−1+1+1+10x₁1x₂0h₁1h₂1ŷEdge labels are fixed weights · dashed edges have negative weights

[0, 1] → h = [0, 1] → Output = 1

ReLU keeps positive differences and clips negative ones to zero. Adding the two hidden values detects unequal inputs.

These weights are chosen by hand, not trained. This demonstrates a working representation, not a guarantee that training will find it.
Compare all four inputs
x₁x₂TargetOutput
0000
0111
1011
1100
Open the code · NumPy / JAX

The interaction above runs in JavaScript. These Python examples use the same data and equations, starting from the defaults—not your current controls. JAX can compute the loss gradient with jax.grad. JAX autodiff guide

In a new local project, save the example as demo.py. CPU setup: uv init ai-lab → cd ai-lab → uv add numpy jax → uv run demo.py

import jax.numpy as jnp
# NumPy: replace the import above with: import numpy as jnp

x = jnp.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
w1 = jnp.array([[1., -1.], [-1., 1.]])
w2 = jnp.array([1., 1.])
h = jnp.maximum(x @ w1, 0.)
print(h @ w2) # [0., 1., 1., 0.]
print((x @ w1) @ w2) # [0., 0., 0., 0.]: no ReLU

Each layer of an MLP does two things: an affine map, which multiplies by a matrix and adds a vector, followed by an elementwise nonlinearity applied to each entry separately. For input x∈Rd\mathbf{x}\in\mathbb{R}^{d}, hidden width hh, and output width kk, a network with one hidden layer is

z1=W1x+b1,h=ϕ(z1),o=W2h+b2,\mathbf{z}_1=W_1\mathbf{x}+\mathbf{b}_1, \quad \mathbf{h}=\phi(\mathbf{z}_1), \quad \mathbf{o}=W_2\mathbf{h}+\mathbf{b}_2,

with W1∈Rh×dW_1\in\mathbb{R}^{h\times d} and W2∈Rk×hW_2\in\mathbb{R}^{k\times h}. The nonlinearity ϕ\phi is what makes depth count. Without it, two layers are W2(W1x+b1)+b2=(W2W1)x+(W2b1+b2)W_2(W_1\mathbf x+\mathbf b_1)+\mathbf b_2=(W_2W_1)\mathbf x+(W_2\mathbf b_1+\mathbf b_2), which is still a single affine map; however many layers you stack, they represent the same functions as one. That is why the experiment above outputs 0 for all four inputs once ReLU is switched off.

Two ReLUs Compute XOR​

XOR must output 1 for (1,0)(1,0) and (0,1)(0,1) and 0 for (0,0)(0,0) and (1,1)(1,1). Plotted in the plane, the two points that should give 1 lie on one diagonal and the two that should give 0 lie on the other, so no single straight line puts them on opposite sides. The network in the experiment computes, for x1,x2∈{0,1}x_1,x_2\in\{0,1\},

h1=ReLU⁡(x1−x2),h2=ReLU⁡(x2−x1),h_1=\operatorname{ReLU}(x_1-x_2), \qquad h_2=\operatorname{ReLU}(x_2-x_1), f(x1,x2)=h1+h2.f(x_1,x_2)=h_1+h_2.

When the inputs are equal ((0,0)(0,0) or (1,1)(1,1)), both differences are 0 and the output is 0. When they differ ((1,0)(1,0) or (0,1)(0,1)), exactly one difference is 1 and the output is 1. The hidden layer first represents whether the inputs differ, and the output layer only adds that feature up. With ReLU switched off, h1+h2=(x1−x2)+(x2−x1)h_1+h_2=(x_1-x_2)+(x_2-x_1) is always 0, and the intermediate feature disappears.

What Backpropagation Computes​

Training needs to know which way, and how far, to move each parameter: the gradient of the loss with respect to that parameter. For one loss LL, let δ2=∂L/∂o\boldsymbol{\delta}_2=\partial L/\partial\mathbf{o}. The chain rule computes the gradients layer by layer, from the output back:

∂L∂W2=δ2h⊤,δ1=(W2⊤δ2)⊙ϕ′(z1),\frac{\partial L}{\partial W_2}=\boldsymbol{\delta}_2\mathbf{h}^{\top}, \qquad \boldsymbol{\delta}_1=(W_2^{\top}\boldsymbol{\delta}_2)\odot\phi'(\mathbf{z}_1), ∂L∂W1=δ1x⊤.\frac{\partial L}{\partial W_1}=\boldsymbol{\delta}_1\mathbf{x}^{\top}.

In words: the output error δ2\boldsymbol\delta_2 times that layer's input h\mathbf h is the gradient for W2W_2. The error travels back through W2W_2 to the hidden layer and is multiplied by the activation's slope ϕ′\phi' at that point, giving the hidden error δ1\boldsymbol\delta_1. Then δ1\boldsymbol\delta_1 times the input x\mathbf x is the gradient for W1W_1.

One Scalar Backward Pass​

Take x=2x=2, w1=1w_1=1, b1=−1b_1=-1, w2=3w_2=3, b2=0b_2=0, target y=1y=1, and L=(o−y)2/2L=(o-y)^2/2. Each step uses one rule:

PassRuleSubstitutionResult
forwardz1=w1x+b1z_1=w_1x+b_11⋅2−11\cdot2-111
forwardh=ReLU⁡(z1)h=\operatorname{ReLU}(z_1)ReLU⁡(1)\operatorname{ReLU}(1)11
forwardo=w2h+b2o=w_2h+b_23⋅1+03\cdot1+033
forwardL=(o−y)2/2L=(o-y)^2/2(3−1)2/2(3-1)^2/222
backwardδ2=∂L/∂o=o−y\delta_2=\partial L/\partial o=o-y3−13-122
backward∂L/∂w2=δ2h\partial L/\partial w_2=\delta_2h2⋅12\cdot122
backward∂L/∂b2=δ2\partial L/\partial b_2=\delta_222
backwardδ1=w2δ2ReLU⁡′(z1)\delta_1=w_2\delta_2\operatorname{ReLU}'(z_1)3⋅2⋅13\cdot2\cdot166
backward∂L/∂w1=δ1x\partial L/\partial w_1=\delta_1x6⋅26\cdot21212
backward∂L/∂b1=δ1\partial L/\partial b_1=\delta_166

All derivatives use the same forward-pass weights; updating w2w_2 before computing δ1\delta_1 would differentiate a different computation.

Backpropagation on a small computation graph

This is the scalar network from the text, o = w₂·ReLU(w₁x + b₁) + b₂ with loss L = ½(o − y)². Step forward to compute every value, then backward: each node multiplies the gradient arriving from the loss by its local derivative and passes it on. Change any input or weight and both passes update.

The last product gives the first weight its gradient. Each parameter's gradient is the product of the local derivatives along its path back from L.

∂L/∂w₁ = ∂L/∂u · x = 6 × 2 = 12; ∂L/∂x = ∂L/∂u · w₁ = 6 × 1 = 6

×1×2x2∂L/∂x = 6w₁1∂L/∂w₁ = 12u = w₁·x2∂L/∂u = 6b₁−1∂L/∂b₁ = 6z₁ = u + b₁1∂L/∂z₁ = 6h = ReLU(z₁)1∂L/∂h = 6v = w₂·h3∂L/∂v = 2b₂0∂L/∂b₂ = 2o = v + b₂3∂L/∂o = 2y1L = ½(o − y)²2∂L/∂L = 1w₂3∂L/∂w₂ = 2×1×2x2∂L/∂x = 6w₁1∂L/∂w₁ = 12u = w₁·x2∂L/∂u = 6b₁−1∂L/∂b₁ = 6z₁ = u + b₁1∂L/∂z₁ = 6h = ReLU(z₁)1∂L/∂h = 6w₂3∂L/∂w₂ = 2v = w₂·h3∂L/∂v = 2b₂0∂L/∂b₂ = 2o = v + b₂3∂L/∂o = 2y1L = ½(o − y)²2∂L/∂L = 1

Parameter gradients

∂L/∂w₁ = 12∂L/∂b₁ = 6∂L/∂w₂ = 2∂L/∂b₂ = 2

Check: nudge w₁ by ±0.00001 and measure how L changes. The slope is 12; backpropagation gives 12.

ReLU′(0) is taken as 0, as in PyTorch. A real network applies the same rule to vectors and matrices.

Backpropagation computes gradients; an optimizer applies the update. For a mean minibatch loss, average per-example contributions, and clear accumulated gradients before the next independent update. Inference needs only the forward pass. If dropout is present, training randomly suppresses activations (with compensating scaling in inverted dropout), while evaluation disables this randomness. A plain affine/ReLU MLP has no such mode-dependent layer.

Autodiff removes hand-coded derivatives, but shapes, numerical stability, and the objective's assumptions still need checking. A zero gradient may indicate an optimum, Sigmoid/Tanh saturation, a dead ReLU, an incorrect mask, or a disconnected graph.

The Task Chooses the Output Head​

The hidden activation builds intermediate features; the output layer's form follows from the question being answered. Decide which kind of task it is, then choose the head and loss, rather than selecting a library loss first and retrofitting the question:

ProblemOutputCommon lossCheck
real-valued regressionlinear valueMSE, MAE, or likelihood-derived losstails, scale, heteroscedasticity
binary classificationone logitbinary cross-entropythreshold, costs, calibration
exclusive multiclasskk logitssoftmax cross-entropyclasses really are exclusive
multilabelkk independent logitsindependent BCElabel dependence and thresholds

Whether the hidden layers should use ReLU, GELU, SiLU, or SwiGLU with its two projected branches, and how to compare activations fairly, is covered in Activations and Gated Feed-Forward Networks.

Capacity Is Not Learnability​

Cybenko's theorem states that finite sums of units with a continuous sigmoidal activation can uniformly approximate any continuous function on a compact cube to any positive error tolerance. The required finite width may grow with the function and tolerance; the theorem does not give a fixed width that suffices for every target.

The XOR example and Cybenko's theorem both show that a representation exists, not that training will reach it. Neither guarantees:

  • sufficient training data;
  • a parameter- or sample-efficient representation;
  • that SGD or another optimizer finds the representation from arbitrary initialization;
  • generalization under noisy data;
  • calibrated output probabilities;
  • causal validity or out-of-distribution (OOD) generalization.

“Universal approximator” is therefore a theoretical property, not a model-selection result.

An MLP Assumes No Structure in the Data​

An inductive bias is an assumption the architecture makes for you in advance; a CNN, for example, assumes that neighboring pixels are related. An MLP makes almost none: it sees a flat vector, so translation, sequence order, graph structure, and permutation invariance do not appear automatically. CNNs, RNNs, attention, and graph networks encode stronger sharing or computation patterns and can change sample efficiency.

Conversely, on stable tabular features with limited data, a linear model, tree, or small MLP may beat a fashionable architecture. Compare them under the same split and budget.

Minimal Acceptance​

  1. Assert batch, feature, and output shapes.
  2. Overfit a tiny sample to test that the implementation can learn.
  3. Compare against constant, linear, or tree baselines on the same split, and report several seeds rather than only the best run.

Extend the XOR example in TensorFlow Playground: choose a nonlinear dataset, change hidden-layer width or activation, and watch the decision boundary during training. Then increase noise and compare training with test loss; a more flexible boundary can fit noise as well as structure.

Explore connectionsOpen network