GPQA Diamond

How each maker reported it

• Gemini 3.1 Pro p.4

  Results as of February 2026, Gemini at Thinking (High); competitors at their highest reported settings, as listed by Google.

• Claude Opus 4.7 Table 8.1.A, p.191–192

  Opus 4.7 uses adaptive thinking at max effort with default sampling, averaged over 5 trials (Terminal-Bench 2.0 with thinking disabled; Finance Agent at high effort). Competitor figures come from the developers' own published cards or leaderboards.

• Claude Opus 4.8 Table 8.1.A, p.194–195

  Opus 4.8 uses adaptive thinking at max effort with default sampling, averaged over 5 trials. Competitor figures come from the developers' own published cards or leaderboards.

• Qwen3.8-Max model card (Benchmark Results)

  Qwen's own table. Terminal Bench 2.1 run with Claude Code (avg@10); competitor scores are the best published across harnesses. SWE-bench Pro uses a corrected task set. Rows marked in-house are Qwen benchmarks. Fable 5 results may involve fallbacks. Agents' Last Exam is split into pass rate and score.

• Qwen3.8-Flash model card (Benchmark Results)

  Qwen's own tables. SWE tasks use Claude Code or mini-SWE-agent at temperature 1.0 and 256K context; HLE judged by GPT-4o. OSWorld 2.0 and two-setting rows are split. DeepSeek-V4-Flash is not in the vision table.

• Claude Opus 4.6 Table 2.3.A, p.18

  Opus 4.6 results average 5 trials (SWE-bench: 25) with adaptive thinking at max effort and default sampling. MCP-Atlas is the max-effort score (62.7% at high). Gemini 3 Pro ARC-AGI-2 is Deep Thinking.

• DeepSeek-V4.1-Flash Table 3, p.33

  All models at max reasoning effort. Code-agent benchmarks use DeepSeek Harness Minimal mode with 1M context; DeepSWE uses mini-SWE; visual agent tasks use the Claude Code harness with 512K context. HLE for some models is the text-only subset. The Hugging Face card lists NL2Repo-Bench as 64.0; the report and change log say 65.4.

• Gemini 3 Flash launch post (blog.google, 17 Dec 2025)

  The model card and evaluation-methodology PDFs show the results table only as an image. These four scores are the ones Google states in text in the launch post (benchmarks as of December 2025, pass@1, default API settings).

• Qwen3.7-Plus Qwen3.8-Flash-Next model card (Benchmark Results)

  Qwen published Qwen3.7-Plus scores as a comparison column in its Qwen3.8 cards. The Qwen3.8-27B card reports slightly different Qwen3.7-Plus numbers for some rows (e.g. SWE-bench Pro 57.6, DeepSWE 14.2).

• Claude Sonnet 4.6 Table 2.1.A, p.15

  Claude Sonnet 4.6 results are averaged over 10 trials (SWE-bench over 25) with adaptive thinking at max effort and default sampling; Terminal-Bench 2.0 at default thinking. ARC-AGI-2 is 60.4% at high effort. Competitor columns as reported in the card.

• Qwen3.8-27B model card (Text Performance, VL Performance)

  Qwen's own tables. SWE-bench Pro uses Claude Code on a corrected task set (Opus 4.6 is its published score). HLE judged by GPT-4o. Muse Glimmer-30B's CharXiv cell (78.8) does not say which setting, so it is left out. Qwen3.7-Plus figures here differ slightly from the Flash-Next card (e.g. SWE-bench Pro 57.6 vs 55.8).

• Seed 2.0 Pro Tables 3, 8, 11 (pp.9, 13, 16)

  ByteDance's own runs. Competitor columns are as printed in the Seed2.0 card (frontier models available in early 2026). Codeforces is an Elo rating. HLE-text under Agentic is with search tools.

• Claude Opus 4.5 Table 2.3.A, p.19

  Average of 5 trials with a 64k thinking budget, 200k context, default (high) effort and default sampling. SWE-bench, τ²-bench and MCP Atlas are without extended thinking; Terminal-bench uses a 128k budget (57.8% at 64k). GPT-5.1 MMMU is from the MMMU leaderboard.

• Gemini 2.5 Pro p.5–6

  Results as of June 2025, as published in Google's model card. Single-attempt pass@1. Competitor figures are providers' own numbers as listed by Google; Claude 4 Opus is 32K thinking. Aider Polyglot uses diff-fenced for 2.5 Pro and diff for the others. MRCR v2 for Claude 4 Opus is with no thinking.

• Seed 2.0 Lite Tables 4, 8, 12 (pp.10, 13, 17)

  ByteDance's own runs from the Seed2.0 model card, which predates the 260428 snapshot served now. Codeforces is an Elo rating. Seed 2.0 Mini is shown for comparison (same table).

• Gemma 4 31B Benchmark results

  Instruction-tuned models, as published in Google's model card. OmniDocBench is edit distance (lower is better). Audio benchmarks are omitted because 31B has no audio input.

• Gemini 2.5 Flash p.5–6

  Results as of May 2025 for Gemini 2.5 Flash GA with thinking, as published in Google's model card. Competitor figures are as listed by Google; Claude 3.7 Sonnet uses 64K extended thinking where available, Grok 3 Beta extended thinking. MRCR v2 here is the earlier (non 8-needle) version. Aider Polyglot and SWE-bench rows are left out because their columns are ambiguous in the PDF text.

• Mistral Small 4 HF model card, launch post

  Vendor-reported. With reasoning, Mistral says it matches or beats GPT-OSS 120B on AA LCR, LiveCodeBench and AIME25 with shorter outputs; those comparisons are charts without numbers.

• Llama 4 Maverick Benchmarks: Instruction tuned models

  Meta's own 0-shot results on bf16 weights. Llama 3.x models have no image support and a 128K context, so those cells are not reported.

• Gemini 2.5 Flash-Lite p.5

  Results for the 06-17 release (the stable model) as of June 2025, as published in Google's model card. Results for the shut-down 09-2025 preview are left out.

• Llama 4 Scout Benchmarks: Instruction tuned models

  Meta's own 0-shot results on bf16 weights. Llama 3.x models have no image support and a 128K context, so those cells are not reported.