Mixture of experts

A model design where each token is processed by only a few of many sub-networks (experts), so only part of the parameters are active for each token.

Why it matters. Explains why a model with a very large total size can be relatively cheap to run per token, although all its weights still need memory.

How it works

In a mixture-of-experts (MoE) model, some layers contain many parallel sub-networks called experts, and a small router picks a few of them for each token. Makers often report two sizes: total parameters (all experts) and active parameters (those used per token).

Compute per token scales roughly with active parameters, so an MoE model can be cheaper to run than a dense model of the same total size. But every expert must be loaded in memory, so hardware needs follow the total size. Expert counts and routing designs vary between models.

When planning self-hosting, size memory by total parameters and expect compute per token closer to the active count. For API users, MoE mostly shows up indirectly, in price and speed.

Worked example

Illustrative: an MoE model with 400B total and 20B active parameters needs memory for all 400B, about 800 GB at 16-bit, but each token uses roughly the compute of a 20B dense model.

Common mistakes

Sizing GPU memory from active parameters instead of total parameters.
Comparing MoE and dense models by total parameter count as a proxy for quality or speed.
Assuming quantization and fine-tuning tools behave the same on MoE models as on dense ones.