Activations and Gated Feed-Forward Networks

“ReLU is obsolete” is too broad; “learn only ReLU, sigmoid, and tanh” is outdated. Choices now differ by architecture, scale, precision, kernels, and hardware. Transformer FFNs also often replace a plain two-layer path with a gated one, which changes more than an elementwise function.

Elementwise Functions

Name	Definition or role	Strength	Boundary
ReLU	$\max(0,x)$	cheap, exact zeros, nonsaturating positive side	zero negative gradient; kink at zero
Leaky/PReLU	$\max(ax,x)$	retains negative-side gradient	adds an assumption or parameter
GELU	$x\Phi(x)$	smooth magnitude-dependent scaling	approximation and kernel variants; no exact sparsity
SiLU/Swish	$x\sigma(\beta x)$	smooth and permits small negative outputs	nonmonotonic region; kernel and quantization costs
Sigmoid	$\sigma(x)$	$[0,1]$ gate or probability map	saturates at large magnitude
Tanh	$\tanh(x)$	bounded, zero-centered recurrent state	saturates at both ends

GELU may use erf or tanh approximations. The Swish paper studies learnable $\beta$; framework SiLU commonly fixes $\beta=1$. Papers often use the names nearly interchangeably, so record the implementation.

Smooth Does Not Mean Universally Better

Smooth functions can help some large training regimes, but their effects interact with initialization, normalization, width, data, optimization, and fused kernels. ReLU remains a counterexample to a simple progress story: cheap kernels and exact zeros can matter in convolutional, lightweight, or quantized deployment; small-data error may be dominated by regularization; and a stable ReLU baseline reveals whether a gated model merely added parameters.

GLUs and Gated FFNs

A plain Transformer FFN is

$$\operatorname{FFN}(x)=W_2\phi(W_1x+b_1)+b_2.$$

A simplified GLU multiplies two paths:

$$\operatorname{GLU}(x)=A(x)\odot\sigma(B(x)).$$

Common Transformer variants include

$$\operatorname{GEGLU}(x)=W_o\left(\operatorname{GELU}(W_gx)\odot W_vx\right),$$ $$\operatorname{SwiGLU}(x)=W_o\left(\operatorname{SiLU}(W_gx)\odot W_vx\right).$$

ReGLU substitutes ReLU in the gate branch. A gated layer has three major projections ($W_g,W_v,W_o$) rather than two. Equal-parameter comparisons therefore reduce its intermediate width; there is no universal ratio once biases, embeddings, and alignment constraints are counted.

Dated Architecture Examples

The BERT paper used GELU; PaLM reported SwiGLU; the Llama 3 architecture report combines SwiGLU with RMSNorm, RoPE, and GQA; ConvNeXt showed GELU in a modern ConvNet. These establish specific designs, not universal superiority. When several block changes move together, their aggregate result cannot be attributed to activation alone.

Output Activations Are a Separate Decision

Binary classification commonly trains one logit; exclusive multiclass uses logits with Softmax; multilabel uses independent sigmoids; regression support determines whether output is linear, positive, or bounded. Stable losses generally consume logits directly rather than manually applying sigmoid/Softmax before a logarithm.

Failure Modes

Symptom	Candidate cause	Check
permanently zero units	dead ReLU, large learning rate, shifted inputs	histograms, gradients, Leaky control
near-zero gradients	saturation or bad scaling	pre-activation distribution and initialization
collapsed gates	gate/value scale imbalance	branch norms and an ungated baseline
mixed-precision NaNs	extremes, approximation, loss scaling	fp32 control and finite-value assertions
slower “new” function	missing fused kernel or extra projection	end-to-end throughput, not FLOPs alone
post-quantization loss	range and calibration mismatch	evaluation on target hardware

Controlled Selection

Fix split, initialization family, optimizer, steps, and stopping rule. For plain ReLU/GELU/SiLU FFNs, make widths and parameter counts explicit. For GEGLU/SwiGLU, report intermediate width, total parameters, FLOPs, memory, and target-hardware throughput. Repeat seeds and predeclare quality, latency, memory, and stability thresholds.

This note is biased toward dense GPU-trained Transformers. Periodic activations, implicit neural representations, splines, spiking models, and specialized accelerators may require different criteria. Ask whether a new function changes representation, optimization, budget, or kernel before adding it to an activation zoo.