Activations and Gated Feed-Forward Networks
Stack several linear maps and the result is still one linear map. Without activation functions, a deep network is no more than a single layer (see the multilayer perceptron). An activation sits between layers and decides how each unit bends its input, which is what lets a network fit curves and boundaries.
Modern Transformers make a larger change on top of this: they split the feed-forward network (FFN) into two branches and multiply them. One branch computes content; the other decides how much of that content passes, and the amount changes with every token.
The subject here is nonlinearity and channel mixing within a position; attention exchanges information across positions. An FFN gate multiplies channel values, whereas MoE routing chooses expert networks for tokens; a model can use both. The training process explains how a loss updates their parameters; whether the output layer takes a sigmoid or a Softmax depends on the task, as the output-head section of Multilayer Perceptron shows.
Elementwise Activations Decide How Each Unit Bends Its Input
In the table, . is the standard normal cumulative distribution function (CDF), the probability that a standard normal variable is at most , so GELU multiplies by a factor between 0 and 1 that grows with . Exact GELU uses ; the common tanh expression approximates it, and different kernels can produce small differences in mixed precision.
Two naming and notation conventions also need stating. The Swish paper studies a learnable , while framework SiLU commonly fixes ; papers use the names nearly interchangeably, so record the implementation. The table's form assumes ; unrestricted PReLU is defined piecewise as for nonnegative inputs and otherwise.
For SiLU, . At zero, ReLU, GELU, and SiLU all output zero, but GELU and SiLU have derivative ; ReLU has no unique derivative there (implementations choose a convention). At , SiLU is approximately and its derivative is approximately . Used as a gate, SiLU is therefore not confined to : it can dip slightly below zero or exceed one.
Open full-size imageFocus on negative inputs: ReLU returns zero, GELU dips below zero then approaches it for very negative inputs, and ELU with α = 1 approaches −1. The blue curve is the standard GELU.
A Gated FFN Lets One Branch Decide How Much the Other Passes
A plain Transformer FFN projects a token's vector into a wider intermediate layer with , applies an activation , and projects back with :
A gated linear unit (GLU) multiplies two projections: supplies content, supplies a pass-through fraction between 0 and 1, and is elementwise multiplication. A simplified form is
Common Transformer variants include
Here produces the gate branch and the value branch; their elementwise product is projected back to the model width by . ReGLU substitutes ReLU in the gate branch.
Follow SwiGLU through one token. Take an intermediate layer with only two channels, a gate branch and a value branch . After SiLU the gate is about , because and . The elementwise product is : the first channel is amplified, and the second is almost shut.
Now take another token whose gate branch is , with the same value branch . The gate becomes about and the product about : this time the first channel is shut and the second is open.
Because the gate is computed from the input, the same channel can open for one token and close for another. In a plain FFN, how much a channel passes and what it outputs are the same number: after ReLU it either outputs its own entry of unchanged or outputs zero. A gated FFN gives these two jobs to two projections, one deciding how much passes and one deciding what passes, so the output is a product of two quantities that both vary with the input.
A gated layer has three major projections (, , ), whereas a plain FFN has only input and output projections; at the same intermediate width, both parameters and computation grow. Equal-parameter comparisons therefore shrink the gated layer's intermediate width.
A bias-free plain FFN mapping width back to through width has weights; a gated FFN of intermediate width has ; setting them equal gives . This is the local parameter-count comparison in GLU Variants Improve Transformer.
For , choose . The gate and value projections each map 6 features to 8, their elementwise product is still length 8, and the output projection maps back to 6. The plain FFN has and the gated FFN : both have 144 weights.
When you download a model, this intermediate width is the intermediate_size in its Hugging Face config.json. Qwen3-8B's configuration lists hidden_size: 4096, intermediate_size: 12288, and hidden_act: "silu", and its weight index shows three FFN matrices per layer, gate_proj, up_proj, and down_proj, which are , , and here. Each layer's FFN therefore has million weights. Working back through , that equals a plain FFN with intermediate width 18432.
Examples in Modern Architectures
- The original BERT architecture uses GELU.
- PaLM reports using SwiGLU.
- The Llama 3 architecture report also uses SwiGLU, alongside RMSNorm, RoPE, and GQA as parts of a complete block.
- ConvNeXt shows that modern ConvNets can use GELU too; smooth activations are used beyond Transformers.
Compare Activations at an Equal Budget
No activation ranks first on every task. A higher average score in a paper, and the choice of GELU or SwiGLU by the models above, hold for their own scale, modality, and hardware. The benefit is also entangled with initialization, normalization, width, data, and fused kernels; when several components of an architecture change together, it cannot all be credited to the activation.
So when you swap an activation, fix the data split, optimizer, number of steps, and seeds first. Then report intermediate width, total parameters, and peak memory. is only a bias-free weight count, and two FFNs with equal parameters can still differ widely in measured latency. So also measure end-to-end throughput at the target dtype and hardware: a new function without a fused kernel runs slower, and quantization degrades it when its numeric range changes.
If many units stay at zero or gradients approach zero during training, check for dead ReLUs and sigmoid/tanh saturation first. Keep a stable ReLU baseline; it shows whether a gate's benefit merely comes from added parameters.