Mixture of Experts: Sparse Computation, Routing, and Deployment Cost

Mixture of Experts (MoE) stores several parameter groups while activating only some for each input. In common sparse Transformers, MoE replaces selected FFN layers: a router chooses a few experts for each token, then combines their outputs. Total parameter capacity can grow without proportionally increasing each token’s FFN computation.

So read an MoE model’s size as two numbers: memory follows total parameters, while per-token speed and cost follow active parameters. In Mixtral 8x7B each token has access to 47B parameters but uses only 13B active ones. The paper states that the memory cost of serving it is proportional to its 47B sparse parameter count, and that using only a subset of its parameters per token gives faster inference at low batch sizes. Local single-user generation is exactly the low-batch case: each step reads the weights it uses from memory, and an MoE model reads only the selected experts, so it runs faster than a dense model with the same total parameters. For MoE models you can run locally today, with their total and active parameters, see local coding models.

These experts are subnetworks trained under a shared objective inside one model, not conversational agents holding a meeting. Their name does not guarantee separate mathematics, law, or coding specialties.

Open full-size image

The expanded blue block follows two tokens. Each router selects one FFN, and its gate value scales that expert’s output. This is Switch’s top-1 routing, a specific MoE design. The top-k formulation below can activate several experts for the same token.

Fedus, Zoph and Shazeer, Switch Transformers, Figure 2; cropped from the paper.

Following one token

The classic sparsely gated MoE paper uses learned gating to select a subset of experts. A simplified token-choice top-k formulation is:

$$r=xW_r,\qquad S(x)=\operatorname{TopK}(r,k),\qquad y=\sum_{i\in S(x)}g_i(x)E_i(x).$$

The token state $x$ has width $d$; router matrix $W_r$ has shape [d,E] for E experts. $E_i$ is an expert network and $g_i$ its mixing weight. Implementations differ in router scores, normalization, shared experts, and biases. This formula captures selection, computation, and combination.

With four experts and top-2, suppose routing probabilities are [0.50,0.30,0.15,0.05]. Under renormalization over selected experts, the first two weights become 0.625 and 0.375. If those experts output [2,0] and [0,4], the result is [1.25,1.50]. These are illustrative values, not measured routes in a real model.

Tokens can choose different experts, and the same token can choose different routes across layers. Shared attention, normalization, and residual paths usually remain. The router selects internal computation, not a readable answer.

Many stored parameters, fewer active parameters

Switch Transformer demonstrates one expert per token. Mixtral reports a design selecting two of eight experts per layer. These are architectural choices, not fixed MoE ratios.

Consider only one layer’s FFNs. With eight 10M-parameter experts and top-2 routing, experts store 80M parameters, while one token uses two groups, about 20M. Shared parameters must be added separately. Across a batch, different tokens may collectively use all eight experts.

Quantity	What it answers	What it does not establish
Total parameters	How many weights exist	Actual operations for one token
Active parameters	Which weights a path uses	Whether all weights fit in GPU memory
FLOPs	Arithmetic under a specified workload	Wall-clock latency
Weight and cache memory	How much data must be stored	Transfer or communication speed

Active parameters do not reduce total weights. Inactive experts still need storage somewhere, because the next token may select any of them. Keeping all weights on the GPU uses total weight memory. When they do not fit, the llama.cpp server can keep all MoE expert weights in CPU memory with --cpu-moe, or only those of the first N layers with --n-cpu-moe N. That frees GPU memory, at the price of reading those experts from slower system memory, so each token takes longer. KV cache is principally determined by attention structure, sequence length, and batch size. Expert count cannot simply multiply or divide the cache formula.

Routing requires load management

If most tokens select one expert, others idle while the popular expert becomes a bottleneck. Training can further reinforce existing routing preferences. Load balancing prevents excessive concentration of computation and learning opportunities; it does not require experts to perform identical tasks.

Switch gives each expert a capacity limit; once a popular expert is full, overflow tokens skip expert computation for that layer and continue through the residual path; see the capacity and balancing discussion. Other implementations use different schedules and do not drop tokens.

An auxiliary balancing loss encourages more even assignment, but excessive weighting can interfere with the task objective. Routing stability, precision, and distribution shift also matter. Balanced training traffic does not guarantee balanced production traffic for long documents or a particular language.

Communication can absorb arithmetic savings

When a provider places experts on different GPUs, each token travels to its selected experts and the outputs return for combination; this all-to-all exchange costs time that depends on interconnect, batch, and scheduling, and eats part of the arithmetic savings. A single local machine skips this step; what matters there is where the total weights live; see above, quantization, and inference performance.

Reading performance results

• Record total and per-token active parameters first: the former sets how much memory you need, the latter how much each step computes and reads.
• Compare time to first token, subsequent generation speed, and throughput on the same hardware, weight precision, input/output lengths, and concurrency; compare quality on the same test set.
• An expert’s name does not tell you its role. An expert receiving many code tokens shows a routing preference; calling it a coding expert needs evidence from controlled interventions or ablations.

My reading