Skip to main content

Activations and Gated Feed-Forward Networks

Stack several linear maps and the result is still one linear map. Without activation functions, a deep network is no more than a single layer (see the multilayer perceptron). An activation sits between layers and decides how each unit bends its input, which is what lets a network fit curves and boundaries.

Modern Transformers make a larger change on top of this: they split the feed-forward network (FFN) into two branches and multiply them. One branch computes content; the other decides how much of that content passes, and the amount changes with every token.

The subject here is nonlinearity and channel mixing within a position; attention exchanges information across positions. An FFN gate multiplies channel values, whereas MoE routing chooses expert networks for tokens; a model can use both. The training process explains how a loss updates their parameters; whether the output layer takes a sigmoid or a Softmax depends on the task, as the output-head section of Multilayer Perceptron shows.

Elementwise Activations Decide How Each Unit Bends Its Input​

NameDefinition or roleStrengthBoundary
ReLUmax⁡(0,x)\max(0,x)cheap, exact zeros, nonsaturating positive sidezero negative gradient; kink at zero
Leaky/PReLUmax⁡(ax,x)\max(ax,x)retains negative-side gradientone more slope aa to set or learn
GELUxΦ(x)x\Phi(x)smooth magnitude-dependent scaling; used in BERT, among othersapproximation and kernel variants; no exact sparsity
SiLU/Swishxσ(βx)x\sigma(\beta x)smooth and permits small negative outputs; β=1\beta=1 is commonly called SiLU/Swishnonmonotonic region; kernel and quantization costs
Sigmoidσ(x)\sigma(x)[0,1][0,1] gate or probability mapsaturates at large magnitude; not a general default for deep hidden layers
Tanhtanh⁡(x)\tanh(x)bounded, zero-centered; common in recurrent statessaturates at both ends

In the table, σ(x)=1/(1+e−x)\sigma(x)=1/(1+e^{-x}). Φ\Phi is the standard normal cumulative distribution function (CDF), the probability that a standard normal variable is at most xx, so GELU multiplies xx by a factor between 0 and 1 that grows with xx. Exact GELU uses Φ(x)=(1+erf⁡(x/2))/2\Phi(x)=(1+\operatorname{erf}(x/\sqrt2))/2; the common tanh expression approximates it, and different kernels can produce small differences in mixed precision.

Two naming and notation conventions also need stating. The Swish paper studies a learnable β\beta, while framework SiLU commonly fixes β=1\beta=1; papers use the names nearly interchangeably, so record the implementation. The table's max⁡(ax,x)\max(ax,x) form assumes a≤1a\leq1; unrestricted PReLU is defined piecewise as xx for nonnegative inputs and axax otherwise.

For SiLU, f′(x)=σ(x)+xσ(x)(1−σ(x))f'(x)=\sigma(x)+x\sigma(x)(1-\sigma(x)). At zero, ReLU, GELU, and SiLU all output zero, but GELU and SiLU have derivative 1/21/2; ReLU has no unique derivative there (implementations choose a convention). At x=−1x=-1, SiLU is approximately −0.2689-0.2689 and its derivative is approximately 0.07230.0723. Used as a gate, SiLU is therefore not confined to [0,1][0,1]: it can dip slightly below zero or exceed one.

GELU, ReLU and ELU curves differ around zero and on negative inputs.Open full-size image

Focus on negative inputs: ReLU returns zero, GELU dips below zero then approaches it for very negative inputs, and ELU with α = 1 approaches −1. The blue curve is the standard GELU.

A Gated FFN Lets One Branch Decide How Much the Other Passes​

A plain Transformer FFN projects a token's vector into a wider intermediate layer with W1W_1, applies an activation ϕ\phi, and projects back with W2W_2:

FFN⁡(x)=W2 ϕ(W1x+b1)+b2.\operatorname{FFN}(x)=W_2\,\phi(W_1x+b_1)+b_2.

A gated linear unit (GLU) multiplies two projections: A(x)A(x) supplies content, σ(B(x))\sigma(B(x)) supplies a pass-through fraction between 0 and 1, and ⊙\odot is elementwise multiplication. A simplified form is

GLU⁡(x)=A(x)⊙σ(B(x)).\operatorname{GLU}(x)=A(x)\odot\sigma(B(x)).

Common Transformer variants include

GEGLU⁡(x)=Wo(GELU⁡(Wgx)⊙Wvx),\operatorname{GEGLU}(x)=W_o\left(\operatorname{GELU}(W_gx)\odot W_vx\right), SwiGLU⁡(x)=Wo(SiLU⁡(Wgx)⊙Wvx).\operatorname{SwiGLU}(x)=W_o\left(\operatorname{SiLU}(W_gx)\odot W_vx\right).

Here WgW_g produces the gate branch and WvW_v the value branch; their elementwise product is projected back to the model width by WoW_o. ReGLU substitutes ReLU in the gate branch.

Follow SwiGLU through one token. Take an intermediate layer with only two channels, a gate branch Wgx=[2,−4]W_gx=[2,-4] and a value branch Wvx=[3,4]W_vx=[3,4]. After SiLU the gate is about [1.7616,−0.0719][1.7616,-0.0719], because 2σ(2)≈1.76162\sigma(2)\approx1.7616 and −4σ(−4)≈−0.0719-4\sigma(-4)\approx-0.0719. The elementwise product is [5.2848,−0.2878]≈[5.28,−0.29][5.2848,-0.2878]\approx[5.28,-0.29]: the first channel is amplified, and the second is almost shut.

Now take another token whose gate branch is [−4,2][-4,2], with the same value branch [3,4][3,4]. The gate becomes about [−0.0719,1.7616][-0.0719,1.7616] and the product about [−0.22,7.05][-0.22,7.05]: this time the first channel is shut and the second is open.

Because the gate is computed from the input, the same channel can open for one token and close for another. In a plain FFN, how much a channel passes and what it outputs are the same number: after ReLU it either outputs its own entry of W1xW_1x unchanged or outputs zero. A gated FFN gives these two jobs to two projections, one deciding how much passes and one deciding what passes, so the output is a product of two quantities that both vary with the input.

A gated layer has three major projections (WgW_g, WvW_v, WoW_o), whereas a plain FFN has only input and output projections; at the same intermediate width, both parameters and computation grow. Equal-parameter comparisons therefore shrink the gated layer's intermediate width.

A bias-free plain FFN mapping width dd back to dd through width mm has 2dm2dm weights; a gated FFN of intermediate width gg has 3dg3dg; setting them equal gives g=2m/3g=2m/3. This is the local parameter-count comparison in GLU Variants Improve Transformer.

For d=6,m=12d=6,m=12, choose g=8g=8. The gate and value projections each map 6 features to 8, their elementwise product is still length 8, and the output projection maps back to 6. The plain FFN has 2×6×122\times6\times12 and the gated FFN 3×6×83\times6\times8: both have 144 weights.

When you download a model, this intermediate width gg is the intermediate_size in its Hugging Face config.json. Qwen3-8B's configuration lists hidden_size: 4096, intermediate_size: 12288, and hidden_act: "silu", and its weight index shows three FFN matrices per layer, gate_proj, up_proj, and down_proj, which are WgW_g, WvW_v, and WoW_o here. Each layer's FFN therefore has 3×4096×12288≈1513\times4096\times12288\approx151 million weights. Working back through g=2m/3g=2m/3, that equals a plain FFN with intermediate width 18432.

Examples in Modern Architectures​

  • The original BERT architecture uses GELU.
  • PaLM reports using SwiGLU.
  • The Llama 3 architecture report also uses SwiGLU, alongside RMSNorm, RoPE, and GQA as parts of a complete block.
  • ConvNeXt shows that modern ConvNets can use GELU too; smooth activations are used beyond Transformers.

Compare Activations at an Equal Budget​

No activation ranks first on every task. A higher average score in a paper, and the choice of GELU or SwiGLU by the models above, hold for their own scale, modality, and hardware. The benefit is also entangled with initialization, normalization, width, data, and fused kernels; when several components of an architecture change together, it cannot all be credited to the activation.

So when you swap an activation, fix the data split, optimizer, number of steps, and seeds first. Then report intermediate width, total parameters, and peak memory. g=2m/3g=2m/3 is only a bias-free weight count, and two FFNs with equal parameters can still differ widely in measured latency. So also measure end-to-end throughput at the target dtype and hardware: a new function without a fused kernel runs slower, and quantization degrades it when its numeric range changes.

If many units stay at zero or gradients approach zero during training, check for dead ReLUs and sigmoid/tanh saturation first. Keep a stable ReLU baseline; it shows whether a gate's benefit merely comes from added parameters.

Explore connectionsOpen network