SWE-bench Verified

How each maker reported it

• Claude Fable 5 Table 8.1.A, p.251–252

  Rows where the card reports a Fable 5 score. Fable 5 scores include production safeguards and fallback to Opus 4.8, so some are slightly below Mythos 5. Mythos 5 uses adaptive thinking at max effort, averaged over 5 trials. Some Fable rows were run by external evaluators (Surge, Andon Labs, Artificial Analysis, Databricks, Zapier, Harvey). Terminal-Bench 2.1 GPT-5.5 is with Codex CLI, Gemini with Gemini CLI. GDPval-AA Elo as of 6 Jun 2026. Competitor figures from developers' published results or leaderboards. Competitor values for AutomationBench, OfficeQA Pro and Harvey are left blank: they couldn't be matched to columns reliably from the PDF.

• Claude Opus 4.8 Table 8.1.A, p.194–195

  Opus 4.8 uses adaptive thinking at max effort with default sampling, averaged over 5 trials. Competitor figures come from the developers' own published cards or leaderboards.

• Claude Opus 4.7 Table 8.1.A, p.191–192

  Opus 4.7 uses adaptive thinking at max effort with default sampling, averaged over 5 trials (Terminal-Bench 2.0 with thinking disabled; Finance Agent at high effort). Competitor figures come from the developers' own published cards or leaderboards.

• Claude Sonnet 5 Table 8.1.A, p.115–116; SWE-bench variants §8.2, p.116

  Claude Sonnet 5 results use adaptive thinking at max effort, averaged over five trials. BrowseComp is the single-agent score (86.6% multi-agent) with a 10M-token limit and compaction. GPT-5.5 Terminal-Bench 2.1 figure uses Codex CLI. Competitor figures come from the developers' own published results.

• Claude Opus 4.5 Table 2.3.A, p.19

  Average of 5 trials with a 64k thinking budget, 200k context, default (high) effort and default sampling. SWE-bench, τ²-bench and MCP Atlas are without extended thinking; Terminal-bench uses a 128k budget (57.8% at 64k). GPT-5.1 MMMU is from the MMMU leaderboard.

• Claude Opus 4.6 Table 2.3.A, p.18

  Opus 4.6 results average 5 trials (SWE-bench: 25) with adaptive thinking at max effort and default sampling. MCP-Atlas is the max-effort score (62.7% at high). Gemini 3 Pro ARC-AGI-2 is Deep Thinking.

• Gemini 3.1 Pro p.4

  Results as of February 2026, Gemini at Thinking (High); competitors at their highest reported settings, as listed by Google.

• Claude Sonnet 4.6 Table 2.1.A, p.15

  Claude Sonnet 4.6 results are averaged over 10 trials (SWE-bench over 25) with adaptive thinking at max effort and default sampling; Terminal-Bench 2.0 at default thinking. ARC-AGI-2 is 60.4% at high effort. Competitor columns as reported in the card.

• Gemini 3 Flash launch post (blog.google, 17 Dec 2025)

  The model card and evaluation-methodology PDFs show the results table only as an image. These four scores are the ones Google states in text in the launch post (benchmarks as of December 2025, pass@1, default API settings).

• Mistral Medium 3.5 HF model card, launch post

  Vendor-reported. Comparisons with other models appear only in charts without numbers. LEXam-hard is asterisked on the card.

• Seed 2.0 Pro Tables 3, 8, 11 (pp.9, 13, 16)

  ByteDance's own runs. Competitor columns are as printed in the Seed2.0 card (frontier models available in early 2026). Codeforces is an Elo rating. HLE-text under Agentic is with search tools.

• Muse Glimmer 30B Benchmarks

  Meta's own results. Muse Glimmer at high reasoning; Gemma4-31B and Qwen3.6-27B in thinking mode. GDPVal-AA v2 is an Elo-style score.

• Seed 2.0 Lite Tables 4, 8, 12 (pp.10, 13, 17)

  ByteDance's own runs from the Seed2.0 model card, which predates the 260428 snapshot served now. Codeforces is an Elo rating. Seed 2.0 Mini is shown for comparison (same table).

• gpt-oss-120b Figures 1, 2, 4; Table 2

  OpenAI's own pass@1 results at reasoning level high. AIME and HLE with tools; Codeforces with a terminal tool. o4-mini and o3 values read from OpenAI's charts.

• gpt-oss-20b Figures 1, 2, 4; Table 2

  OpenAI's own pass@1 results at reasoning level high. AIME and HLE with tools; Codeforces with a terminal tool. o3-mini was run without tools on AIME, HLE and Codeforces, so those cells are left empty. Baseline values read from OpenAI's charts.