Multilayer Perceptron

An MLP alternates affine maps with elementwise nonlinearities. For input $\mathbf{x}\in\mathbb{R}^{d}$, hidden width $h$, and output width $k$:

$$\mathbf{z}_1=W_1\mathbf{x}+\mathbf{b}_1, \quad \mathbf{h}=\phi(\mathbf{z}_1), \quad \mathbf{o}=W_2\mathbf{h}+\mathbf{b}_2,$$

with $W_1\in\mathbb{R}^{h\times d}$ and $W_2\in\mathbb{R}^{k\times h}$. Without $\phi$, consecutive affine maps collapse into one; depth alone would not enlarge the function class.

Heads, Losses, and Assumptions

Problem	Output	Common loss	Check
real-valued regression	linear value	MSE, MAE, or likelihood-derived loss	tails, scale, heteroscedasticity
binary classification	one logit	binary cross-entropy	threshold, costs, calibration
exclusive multiclass	$k$ logits	softmax cross-entropy	classes really are exclusive
multilabel	$k$ independent logits	independent BCE	label dependence and thresholds

The hidden activation and output semantics serve different purposes. Do not select a library loss first and retrofit the question afterward.

What Backpropagation Computes

For one loss $L$, let $\boldsymbol{\delta}_2=\partial L/\partial\mathbf{o}$. The chain rule gives

$$\frac{\partial L}{\partial W_2}=\boldsymbol{\delta}_2\mathbf{h}^{\top}, \qquad \boldsymbol{\delta}_1=(W_2^{\top}\boldsymbol{\delta}_2)\odot\phi'(\mathbf{z}_1),$$ $$\frac{\partial L}{\partial W_1}=\boldsymbol{\delta}_1\mathbf{x}^{\top}.$$

Autodiff removes hand-coded derivatives, not shape, numerical, or objective assumptions. A zero gradient may indicate an optimum, saturation, a dead ReLU, an incorrect mask, or a disconnected graph.

Worked Example: XOR with Two ReLUs

For $x_1,x_2\in\{0,1\}$, define

$$h_1=\operatorname{ReLU}(x_1-x_2), \qquad h_2=\operatorname{ReLU}(x_2-x_1),$$ $$f(x_1,x_2)=h_1+h_2.$$

The equal pairs produce 0 and unequal pairs produce 1. A single linear boundary cannot separate XOR; the hidden layer first represents whether the inputs differ. This proves that one representation exists—not that SGD will find it from arbitrary initialization or generalize under noise.

Modern Nonlinearities

• ReLU is cheap and nonsaturating on its positive side, but units may remain inactive. It is still a useful baseline, not universally obsolete.
• GELU and SiLU/Swish are smooth elementwise functions common in modern Transformers and some ConvNets.
• GEGLU/SwiGLU use two projected branches and change width, parameters, and kernels; they are not simple drop-in activations.
• Sigmoid/Tanh remain useful in gates and bounded states despite saturation at large magnitude.

See Activations and Gated Feed-Forward Networks. Fair comparisons control parameter count, FLOPs, training budget, and implementation kernels.

Capacity Is Not Learnability

Cybenko's classic theorem establishes broad approximation under specific activation, compact-domain, and approximation conditions. It does not guarantee finite width or data suffices, an efficient representation, successful optimization, calibrated probabilities, causal validity, or out-of-distribution generalization. “Universal approximator” is not a model-selection result.

Inductive Bias and Counterexamples

An MLP sees a flat vector. Translation, sequence order, graph structure, and permutation invariance do not appear automatically. CNNs, RNNs, attention, and graph networks encode stronger sharing or computation patterns and can change sample efficiency.

Conversely, on stable tabular features with limited data, a linear model, tree, or small MLP may beat a fashionable architecture. Compare them under the same split and budget.

Minimal Acceptance

1. Assert batch, feature, and output shapes.
2. Overfit a tiny sample to test that the implementation can learn.
3. Compare against constant, linear, or tree baselines.
4. Fit learned preprocessing only inside training folds.
5. Report multiple seeds, failures, and resources—not only the best run.
6. Test deployment assumptions with Evaluation Under Distribution Shift.

This note focuses on conventional supervised gradient training; it does not treat an MLP as a biological model or survey every activation and optimizer.