Checkpoint

How it works

Training saves weights periodically, so it can resume after failures and so different points can be evaluated. A release picks one or more checkpoints, for example a base checkpoint after pretraining and an instruct checkpoint after post-training. Training checkpoints also hold optimizer state, which inference does not need.

With APIs, dated model IDs usually pin a specific checkpoint, while aliases may move to newer ones. Open-weights repositories often hold several checkpoints and revisions, and every fine-tune is a new checkpoint derived from a parent.

Record the exact checkpoint, as a dated model ID or a repository plus revision, in evaluations and production configuration, so results can be reproduced and an update can't silently change behaviour.

Worked example

Illustrative: a 7B model trained for 50,000 steps with a checkpoint every 1,000 steps produces 50 checkpoints. If each holds BF16 weights plus two FP32 optimizer values per parameter, that is 7 x (2 + 8) = 70 GB each and 3.5 TB for all 50; the released inference checkpoint is just the 14 GB of weights.

Common mistakes

• 1Using a moving alias in production and blaming your own code when behaviour changes.
• 2Comparing your results with a card that used a different checkpoint, such as base instead of instruct or an older revision.
• 3Running a third-party fine-tune and assuming the original maker's safety evaluations still apply.