aiaimodel.cards
Weights & training54

Distillation

Training a small model to imitate a larger one

Distillation

Training a smaller student model to imitate a larger teacher model, using the teacher's outputs or token probabilities as training targets.

Why it matters. It is how many small, fast, cheap models get much of a larger model's quality, and many providers' terms restrict it.

How it works

In distillation the student learns from the teacher rather than only from raw data: either from the teacher's full probability distribution over next tokens, or from text the teacher generates, including reasoning traces. The teacher's probabilities carry more information per example than a single correct token.

Makers use it to build smaller members of a model family from a flagship and to make compact reasoning models. Many API providers' terms forbid using outputs to train competing models, and open-weights licenses vary, so check terms before distilling.

A distilled model can match its teacher on the tasks covered by the distillation data and fall further behind on others, on long contexts or on rare knowledge. Evaluate it on your own workload instead of inferring from the teacher.

Worked example

Illustrative: self-hosting a large teacher costs $0.90 per 1M output tokens and a distilled student $0.15. If the student handles 95% of requests and the rest go to the teacher, 1 billion output tokens a month cost 0.95 x 1,000 x $0.15 + 0.05 x 1,000 x $0.90 = $142.50 + $45 = $187.50, against $900 with the teacher alone.

Common mistakes

  • Distilling from an API model whose terms prohibit training competing models on its outputs.
  • Assuming a distilled model keeps the teacher's breadth; gains concentrate on the distilled tasks.
  • Expecting a distilled reasoning model to be as reliable as its teacher on unfamiliar problems.

Related terms