Active parameters

In a mixture-of-experts model, the parameters actually used to process each token, as opposed to the total across all experts. In a dense model, active and total are the same.

Why it matters. Active parameters set compute and speed per token; total parameters set the memory you need.

How it works

A mixture-of-experts model sends each token to a few experts in each expert layer. The active parameters are the shared parts (attention, embeddings and any always-on experts) plus the experts selected for that token. Makers state both, as in a model with N total and M active parameters.

Compute per token scales with active parameters, so a mixture-of-experts model can be faster and cheaper per token than a dense model of the same total size. All experts must still be in memory, because different tokens pick different experts, and in a large batch nearly every expert is used at each step.

Compare capability by evaluations, not by either count. For self-hosting, size memory by total parameters and expect speed closer to a dense model of the active size, minus routing and communication overhead.

Worked example

Illustrative: a model with 120B total and 12B active parameters needs about 120 x 2 = 240 GB at BF16, the same as a 120B dense model. Each token costs about 2 x 12B = 24 billion operations, like a 12B dense model, against 240 billion for the dense 120B.

Common mistakes

Buying memory for the active parameter count, then finding the model doesn't load.
Comparing an MoE model's active count with a dense model's count as if they meant the same capability.
Expecting single-request speed to hold at large batch sizes, where nearly all experts are read every step.