Mixture of Experts: Sparse Computation, Routing, and Deployment Cost

Mixture of Experts (MoE) stores several parameter groups while activating only some for each input. In common sparse Transformers, MoE replaces selected FFN layers: a router chooses a few experts for each token, then combines their outputs. Total parameter capacity can grow without proportionally increasing each token’s FFN computation.

These experts are subnetworks trained inside one model, not conversational agents holding a meeting. Their name does not guarantee separate mathematics, law, or coding specialties. Start with activations and gated FFNs.

Open full-size image

The expanded blue block follows two tokens. Each router selects one FFN, and its gate value scales that expert’s output. This is Switch’s top-1 routing, a specific MoE design. The top-k formulation below can activate several experts for the same token.

Fedus, Zoph and Shazeer, Switch Transformers, Figure 2; cropped from the paper.

Following one token

The classic sparsely gated MoE paper uses learned gating to select a subset of experts. A simplified token-choice top-k formulation is:

$$r=xW_r,\qquad S(x)=\operatorname{TopK}(r,k),\qquad y=\sum_{i\in S(x)}g_i(x)E_i(x).$$

The token state $x$ has width $d$; router matrix $W_r$ has shape [d,E] for E experts. $E_i$ is an expert network and $g_i$ its mixing weight. Implementations differ in router scores, normalization, shared experts, and biases. This formula captures selection, computation, and combination.

With four experts and top-2, suppose routing probabilities are [0.50,0.30,0.15,0.05]. Under renormalization over selected experts, the first two weights become 0.625 and 0.375. If those experts output [2,0] and [0,4], the result is [1.25,1.50]. These are illustrative values, not measured routes in a real model.

Tokens can choose different experts, and the same token can choose different routes across layers. Shared attention, normalization, and residual paths usually remain. The router selects internal computation, not a readable answer.

Many stored parameters, fewer active parameters

Switch Transformer demonstrates one expert per token. Mixtral reports a design selecting two of eight experts per layer. These are architectural choices, not fixed MoE ratios.

Consider only one layer’s FFNs. With eight 10M-parameter experts and top-2 routing, experts store 80M parameters, while one token uses two groups, about 20M. Shared parameters must be added separately. Across a batch, different tokens may collectively use all eight experts.

Quantity	What it answers	What it does not establish
Total parameters	How many weights exist	Actual operations for one token
Active parameters	Which weights a path uses	Whether all weights fit in GPU memory
FLOPs	Arithmetic under a specified workload	Wall-clock latency
Weight and cache memory	How much data must be stored	Transfer or communication speed

Inactive experts still need storage somewhere. Keeping all weights on the GPU uses total weight memory; CPU or disk offload reduces GPU residency but adds transfers. KV cache is principally determined by attention structure, sequence length, and batch size. Expert count cannot simply multiply or divide the cache formula.

Routing requires load management

If most tokens select one expert, others idle while the popular expert becomes a bottleneck. Training can further reinforce existing routing preferences. Load balancing prevents excessive concentration of computation and learning opportunities; it does not require experts to perform identical tasks.

One Switch implementation assigns expert capacities. With 100 tokens, four experts, top-1 routing, and capacity factor 1.2, each expert gets about $100/4\times1.2=30$ slots. Assignment [55,20,15,10] overloads the first expert by 25 tokens. Having 120 total slots does not automatically move the unused slots to that expert. In Switch, overflow tokens skip expert computation for that layer and continue through the residual path; see the capacity and balancing discussion. Other implementations use different schedules or avoid dropping, so this is not a universal MoE behavior.

An auxiliary balancing loss encourages more even assignment, but excessive weighting can interfere with the task objective. Routing stability, precision, and distribution shift also matter. Balanced training traffic does not guarantee balanced production traffic for long documents or a particular language.

Communication can absorb arithmetic savings

Expert parallelism places experts on different devices. Tokens travel to selected experts and their outputs return for combination, often through all-to-all communication. Cost depends on interconnects, token counts, batch shapes, and scheduling. Fewer FFN operations do not eliminate transfers.

A low-batch request may create tiny expert matrix multiplications with poor hardware utilization. Larger batches can improve work per expert while increasing routing, buffering, and communication. Models with similar active parameter counts can have very different latency because total weights, attention, and implementations differ. Connect deployment decisions to quantization and inference performance.

Reading performance results

Record total and active parameters, expert count, top-k, and shared experts. Then record hardware, weight precision, parallelism, input/output lengths, and concurrency. Separate time to first token, subsequent generation speed, and aggregate throughput. Compare quality on the same tasks.

When routing statistics are available, inspect per-layer loads, overflow or rerouting rates, communication time, and changes across input categories. An expert receiving many code tokens shows a routing preference. Calling it a coding specialist needs controlled interventions or ablations, not a name or a few examples.

MoE creates a design space between capacity and computation. Local deployment still needs enough total weight storage, supported expert kernels, cache budget, and acceptable task latency. A small active parameter count alone cannot establish that a machine will run it smoothly.

My reading · Opened notes are remembered in this browser.