Humanity's Last Exam (no tools)

How each maker reported it

• Claude Opus 5.5 Table 8.1.A, p.174

  Claude results use adaptive thinking at max effort, averaged over five trials (Terminal-Bench 4.0 at xhigh). Competitor figures come from the developers' own published results.

• Claude Fable 5.1 Table 8.1.A, p.167

  Fable 5.1 and Mythos 5.1 share weights, so column 0 covers both (likewise Fable 5 / Mythos 5). Claude results use adaptive thinking at max effort, averaged over five trials. Terminal-Bench 4.0 values are from §8.6, p.171 (Mythos 5.1: 60.9%; GPT-5.6 Sol from the public leaderboard). Competitor figures come from the developers' own published results or leaderboards.

• Claude Sonnet 5.5 Table 8.1.A, p.109

  Claude results use adaptive thinking at max effort with default sampling, averaged over five trials. Competitor figures come from the developers' own published system cards or leaderboards.

• Claude Opus 5 Table 8.1.A, p.148–149

  Claude Opus 5 results use adaptive thinking at max effort, averaged over five trials (ARC-AGI-3 at high effort). GPT-5.6 Sol's FrontierBench figure uses Codex and its ARC-AGI-1 figure is at xhigh. The Fable 5 HealthBench Professional figure carries a footnote reading "Mythos 5". Competitor figures come from the developers' own published results or leaderboards.

• Claude Opus 4.8 Table 8.1.A, p.194–195

  Opus 4.8 uses adaptive thinking at max effort with default sampling, averaged over 5 trials. Competitor figures come from the developers' own published cards or leaderboards.

• Claude Opus 4.7 Table 8.1.A, p.191–192

  Opus 4.7 uses adaptive thinking at max effort with default sampling, averaged over 5 trials (Terminal-Bench 2.0 with thinking disabled; Finance Agent at high effort). Competitor figures come from the developers' own published cards or leaderboards.

• Claude Haiku 5.5 Table 8.1.A, p.111

  Haiku 5.5 results use adaptive thinking at max effort with default sampling, averaged over five trials. FrontierCode is shown at max effort (Sonnet 5.5 scores 52.1 at xhigh). OSWorld is the offline subset; GPT-6 Luna was run by Anthropic on the same tasks. Other competitor figures come from the developers' own published results.

• Gemini 3.1 Pro p.4

  Results as of February 2026, Gemini at Thinking (High); competitors at their highest reported settings, as listed by Google.

• Claude Sonnet 5 Table 8.1.A, p.115–116; SWE-bench variants §8.2, p.116

  Claude Sonnet 5 results use adaptive thinking at max effort, averaged over five trials. BrowseComp is the single-agent score (86.6% multi-agent) with a 10M-token limit and compaction. GPT-5.5 Terminal-Bench 2.1 figure uses Codex CLI. Competitor figures come from the developers' own published results.

• DeepSeek-V4-Pro model card (0813)

  Code-agent tasks use DeepSeek Harness minimal mode at max effort, temperature 1.0, top_p 0.95. DSBench rows are DeepSeek internal test sets. Competitor figures as published by DeepSeek.

• DeepSeek-V4.1-Flash Table 3, p.33

  All models at max reasoning effort. Code-agent benchmarks use DeepSeek Harness Minimal mode with 1M context; DeepSWE uses mini-SWE; visual agent tasks use the Claude Code harness with 512K context. HLE for some models is the text-only subset. The Hugging Face card lists NL2Repo-Bench as 64.0; the report and change log say 65.4.

• Gemini 3 Flash launch post (blog.google, 17 Dec 2025)

  The model card and evaluation-methodology PDFs show the results table only as an image. These four scores are the ones Google states in text in the launch post (benchmarks as of December 2025, pass@1, default API settings).

• Claude Sonnet 4.6 Table 2.1.A, p.15

  Claude Sonnet 4.6 results are averaged over 10 trials (SWE-bench over 25) with adaptive thinking at max effort and default sampling; Terminal-Bench 2.0 at default thinking. ARC-AGI-2 is 60.4% at high effort. Competitor columns as reported in the card.

• Gemini 2.5 Pro p.5–6

  Results as of June 2025, as published in Google's model card. Single-attempt pass@1. Competitor figures are providers' own numbers as listed by Google; Claude 4 Opus is 32K thinking. Aider Polyglot uses diff-fenced for 2.5 Pro and diff for the others. MRCR v2 for Claude 4 Opus is with no thinking.

• Gemma 4 31B Benchmark results

  Instruction-tuned models, as published in Google's model card. OmniDocBench is edit distance (lower is better). Audio benchmarks are omitted because 31B has no audio input.

• Gemini 3.1 Flash-Lite p.4 (table also on the model card web page)

  Results as of March 2026, as published in Google's model card. Settings as labelled by Google: 3.1 Flash-Lite High, 2.5 Flash and 2.5 Flash-Lite Dynamic, GPT-5 mini High, Claude 4.5 Haiku Extended Thinking, Grok 4.1 Fast Reasoning. GPT-5 mini CharXiv value is with python.

• Gemini 2.5 Flash p.5–6

  Results as of May 2025 for Gemini 2.5 Flash GA with thinking, as published in Google's model card. Competitor figures are as listed by Google; Claude 3.7 Sonnet uses 64K extended thinking where available, Grok 3 Beta extended thinking. MRCR v2 here is the earlier (non 8-needle) version. Aider Polyglot and SWE-bench rows are left out because their columns are ambiguous in the PDF text.

• Gemini 2.5 Flash-Lite p.5

  Results for the 06-17 release (the stable model) as of June 2025, as published in Google's model card. Results for the shut-down 09-2025 preview are left out.