SWE-bench Pro

How each maker reported it

• Claude Opus 5.5 Table 8.1.A, p.174

  Claude results use adaptive thinking at max effort, averaged over five trials (Terminal-Bench 4.0 at xhigh). Competitor figures come from the developers' own published results.

• Claude Sonnet 5.5 Table 8.1.A, p.109

  Claude results use adaptive thinking at max effort with default sampling, averaged over five trials. Competitor figures come from the developers' own published system cards or leaderboards.

• Claude Fable 5.1 Table 8.1.A, p.167

  Fable 5.1 and Mythos 5.1 share weights, so column 0 covers both (likewise Fable 5 / Mythos 5). Claude results use adaptive thinking at max effort, averaged over five trials. Terminal-Bench 4.0 values are from §8.6, p.171 (Mythos 5.1: 60.9%; GPT-5.6 Sol from the public leaderboard). Competitor figures come from the developers' own published results or leaderboards.

• Claude Fable 5 Table 8.1.A, p.251–252

  Rows where the card reports a Fable 5 score. Fable 5 scores include production safeguards and fallback to Opus 4.8, so some are slightly below Mythos 5. Mythos 5 uses adaptive thinking at max effort, averaged over 5 trials. Some Fable rows were run by external evaluators (Surge, Andon Labs, Artificial Analysis, Databricks, Zapier, Harvey). Terminal-Bench 2.1 GPT-5.5 is with Codex CLI, Gemini with Gemini CLI. GDPval-AA Elo as of 6 Jun 2026. Competitor figures from developers' published results or leaderboards. Competitor values for AutomationBench, OfficeQA Pro and Harvey are left blank: they couldn't be matched to columns reliably from the PDF.

• Claude Opus 5 Table 8.1.A, p.148–149

  Claude Opus 5 results use adaptive thinking at max effort, averaged over five trials (ARC-AGI-3 at high effort). GPT-5.6 Sol's FrontierBench figure uses Codex and its ARC-AGI-1 figure is at xhigh. The Fable 5 HealthBench Professional figure carries a footnote reading "Mythos 5". Competitor figures come from the developers' own published results or leaderboards.

• Claude Opus 4.8 Table 8.1.A, p.194–195

  Opus 4.8 uses adaptive thinking at max effort with default sampling, averaged over 5 trials. Competitor figures come from the developers' own published cards or leaderboards.

• Qwen3.8-Max model card (Benchmark Results)

  Qwen's own table. Terminal Bench 2.1 run with Claude Code (avg@10); competitor scores are the best published across harnesses. SWE-bench Pro uses a corrected task set. Rows marked in-house are Qwen benchmarks. Fable 5 results may involve fallbacks. Agents' Last Exam is split into pass rate and score.

• Claude Haiku 5.5 Table 8.1.A, p.111

  Haiku 5.5 results use adaptive thinking at max effort with default sampling, averaged over five trials. FrontierCode is shown at max effort (Sonnet 5.5 scores 52.1 at xhigh). OSWorld is the offline subset; GPT-6 Luna was run by Anthropic on the same tasks. Other competitor figures come from the developers' own published results.

• Grok 4.5 §2–7, p.5–19

  Grok 4.5 at high effort; competitors at max (Fable 5 with fallback), GPT-5.5 at xhigh. Results come from mixed harnesses and evaluators; some competitor figures are taken from developers' own cards. CyberGym is run without production safeguards.

• Claude Opus 4.7 Table 8.1.A, p.191–192

  Opus 4.7 uses adaptive thinking at max effort with default sampling, averaged over 5 trials (Terminal-Bench 2.0 with thinking disabled; Finance Agent at high effort). Competitor figures come from the developers' own published cards or leaderboards.

• Claude Sonnet 5 Table 8.1.A, p.115–116; SWE-bench variants §8.2, p.116

  Claude Sonnet 5 results use adaptive thinking at max effort, averaged over five trials. BrowseComp is the single-agent score (86.6% multi-agent) with a 10M-token limit and compaction. GPT-5.5 Terminal-Bench 2.1 figure uses Codex CLI. Competitor figures come from the developers' own published results.

• Qwen3.8-Flash model card (Benchmark Results)

  Qwen's own tables. SWE tasks use Claude Code or mini-SWE-agent at temperature 1.0 and 256K context; HLE judged by GPT-4o. OSWorld 2.0 and two-setting rows are split. DeepSeek-V4-Flash is not in the vision table.

• Qwen3.8-27B model card (Text Performance, VL Performance)

  Qwen's own tables. SWE-bench Pro uses Claude Code on a corrected task set (Opus 4.6 is its published score). HLE judged by GPT-4o. Muse Glimmer-30B's CharXiv cell (78.8) does not say which setting, so it is left out. Qwen3.7-Plus figures here differ slightly from the Flash-Next card (e.g. SWE-bench Pro 57.6 vs 55.8).

• Muse Spark 1.1 model page; report §5

  Highest reasoning effort for every model (high for Gemini 3.1 Pro, max for Opus 4.8). "Muse Spark (1.0)" is the earlier model. Third-party scores are Meta's best-effort runs.

• Qwen3.7-Plus Qwen3.8-Flash-Next model card (Benchmark Results)

  Qwen published Qwen3.7-Plus scores as a comparison column in its Qwen3.8 cards. The Qwen3.8-27B card reports slightly different Qwen3.7-Plus numbers for some rows (e.g. SWE-bench Pro 57.6, DeepSWE 14.2).

• Muse Glimmer 30B Benchmarks

  Meta's own results. Muse Glimmer at high reasoning; Gemma4-31B and Qwen3.6-27B in thinking mode. GDPVal-AA v2 is an Elo-style score.

• Seed 2.0 Pro Tables 3, 8, 11 (pp.9, 13, 16)

  ByteDance's own runs. Competitor columns are as printed in the Seed2.0 card (frontier models available in early 2026). Codeforces is an Elo rating. HLE-text under Agentic is with search tools.