Benchmark

How it works

A benchmark is a fixed set of tasks with a scoring method, used to measure a model's ability in an area such as coding, math, knowledge or tool use. Results are usually reported as the percentage of tasks solved or as a rating.

Makers run benchmarks with different settings: reasoning effort, number of attempts, tools allowed, prompts and agent scaffolding. Older benchmarks saturate as top models approach the maximum, and test questions can leak into training data. Scores within one report are comparable; scores across reports often are not.

Use benchmarks to shortlist, then decide with an evaluation on your own tasks and data. On this site, scores come from each maker's own card for its own model.

Worked example

Illustrative: model A reports 72% on a coding benchmark at high effort with a custom agent harness; model B reports 68% at default settings. The 4-point gap does not show A is better; you would need both run under the same conditions.

Common mistakes

• 1Comparing scores from different reports without checking settings, attempts and tools.
• 2Choosing a model on a benchmark unrelated to your workload.
• 3Reading small gaps as meaningful when run-to-run variation can be similar in size.