AutomationBench

How each maker reported it

• Mistral Large 4 launch post

  Preliminary, vendor-reported preview results. Many charts in the post give no numbers; only figures stated in text are listed. Blind human coding eval run by Surge AI on a 1–5 scale.

• DeepSeek-V4.1-Flash Table 3, p.33

  All models at max reasoning effort. Code-agent benchmarks use DeepSeek Harness Minimal mode with 1M context; DeepSWE uses mini-SWE; visual agent tasks use the Claude Code harness with 512K context. HLE for some models is the text-only subset. The Hugging Face card lists NL2Repo-Bench as 64.0; the report and change log say 65.4.

• Muse Spark 1.3 model page; Muse Spark 1.3 evaluation methodology

  Muse Spark 1.3, GPT 5.6 Sol and Opus 5 at max reasoning effort; Muse Spark 1.2 at xhigh. Meta reports the highest comparable score from its own runs, official leaderboards or the other makers' own reports. OSWorld 2.0 version 08.08, except 1.2 on 06.24. Agentic IF Index is a Meta-internal composite. Meta's table has no MRCR scores for Opus 5.

• Claude Sonnet 5.5 Table 8.1.A, p.109

  Claude results use adaptive thinking at max effort with default sampling, averaged over five trials. Competitor figures come from the developers' own published system cards or leaderboards.

• Claude Opus 5.5 Table 8.1.A, p.174

  Claude results use adaptive thinking at max effort, averaged over five trials (Terminal-Bench 4.0 at xhigh). Competitor figures come from the developers' own published results.

• Claude Fable 5.1 Table 8.1.A, p.167

  Fable 5.1 and Mythos 5.1 share weights, so column 0 covers both (likewise Fable 5 / Mythos 5). Claude results use adaptive thinking at max effort, averaged over five trials. Terminal-Bench 4.0 values are from §8.6, p.171 (Mythos 5.1: 60.9%; GPT-5.6 Sol from the public leaderboard). Competitor figures come from the developers' own published results or leaderboards.

• Claude Opus 5 Table 8.1.A, p.148–149

  Claude Opus 5 results use adaptive thinking at max effort, averaged over five trials (ARC-AGI-3 at high effort). GPT-5.6 Sol's FrontierBench figure uses Codex and its ARC-AGI-1 figure is at xhigh. The Fable 5 HealthBench Professional figure carries a footnote reading "Mythos 5". Competitor figures come from the developers' own published results or leaderboards.

• Claude Fable 5 Table 8.1.A, p.251–252

  Rows where the card reports a Fable 5 score. Fable 5 scores include production safeguards and fallback to Opus 4.8, so some are slightly below Mythos 5. Mythos 5 uses adaptive thinking at max effort, averaged over 5 trials. Some Fable rows were run by external evaluators (Surge, Andon Labs, Artificial Analysis, Databricks, Zapier, Harvey). Terminal-Bench 2.1 GPT-5.5 is with Codex CLI, Gemini with Gemini CLI. GDPval-AA Elo as of 6 Jun 2026. Competitor figures from developers' published results or leaderboards. Competitor values for AutomationBench, OfficeQA Pro and Harvey are left blank: they couldn't be matched to columns reliably from the PDF.

• Claude Opus 4.8 Table 8.1.A, p.194–195

  Opus 4.8 uses adaptive thinking at max effort with default sampling, averaged over 5 trials. Competitor figures come from the developers' own published cards or leaderboards.

• Claude Sonnet 5 Table 8.1.A, p.115–116; SWE-bench variants §8.2, p.116

  Claude Sonnet 5 results use adaptive thinking at max effort, averaged over five trials. BrowseComp is the single-agent score (86.6% multi-agent) with a 10M-token limit and compaction. GPT-5.5 Terminal-Bench 2.1 figure uses Codex CLI. Competitor figures come from the developers' own published results.