aiaimodel.cards
← All benchmarks

MMMLU

We haven't written a description for this benchmark yet; see the card notes below for how each maker ran it.

Your workloadRAG over docs · 12K in · 600 out · 50K requests/mo · 30% cached
  1. 1Claude Opus 4.7 · Anthropicbest value
    91.5%
  2. 2Claude Opus 4.6 · Anthropicbest value
    91.1%
  3. 3Claude Opus 4.5 · Anthropicbest value
    90.8%
  4. 4Claude Sonnet 4.6 · Anthropicbest value
    89.3%
  5. 5Gemini 3.1 Flash-Lite · Googlebest value
    88.9%
  6. 6Gemma 4 31B · Googlebest value
    88.4%

“Best value”: no other model here reports a higher score at a lower cost for your workload. Scores are self-reported by each maker with different settings, so treat close calls as ties.

How each maker reported it

  • Claude Opus 4.7 Table 8.1.A, p.191–192

    Opus 4.7 uses adaptive thinking at max effort with default sampling, averaged over 5 trials (Terminal-Bench 2.0 with thinking disabled; Finance Agent at high effort). Competitor figures come from the developers' own published cards or leaderboards.

  • Claude Opus 4.6 Table 2.3.A, p.18

    Opus 4.6 results average 5 trials (SWE-bench: 25) with adaptive thinking at max effort and default sampling. MCP-Atlas is the max-effort score (62.7% at high). Gemini 3 Pro ARC-AGI-2 is Deep Thinking.

  • Claude Opus 4.5 Table 2.3.A, p.19

    Average of 5 trials with a 64k thinking budget, 200k context, default (high) effort and default sampling. SWE-bench, τ²-bench and MCP Atlas are without extended thinking; Terminal-bench uses a 128k budget (57.8% at 64k). GPT-5.1 MMMU is from the MMMU leaderboard.

  • Claude Sonnet 4.6 Table 2.1.A, p.15

    Claude Sonnet 4.6 results are averaged over 10 trials (SWE-bench over 25) with adaptive thinking at max effort and default sampling; Terminal-Bench 2.0 at default thinking. ARC-AGI-2 is 60.4% at high effort. Competitor columns as reported in the card.

  • Gemini 3.1 Flash-Lite p.4 (table also on the model card web page)

    Results as of March 2026, as published in Google's model card. Settings as labelled by Google: 3.1 Flash-Lite High, 2.5 Flash and 2.5 Flash-Lite Dynamic, GPT-5 mini High, Claude 4.5 Haiku Extended Thinking, Grok 4.1 Fast Reasoning. GPT-5 mini CharXiv value is with python.

  • Gemma 4 31B Benchmark results

    Instruction-tuned models, as published in Google's model card. OmniDocBench is edit distance (lower is better). Audio benchmarks are omitted because 31B has no audio input.

Only each model's own maker's number is shown; scores a maker reports for rival models are left out.