DeepSWE v1.1

How each maker reported it

• Muse Spark 1.3 model page; Muse Spark 1.3 evaluation methodology

  Muse Spark 1.3, GPT 5.6 Sol and Opus 5 at max reasoning effort; Muse Spark 1.2 at xhigh. Meta reports the highest comparable score from its own runs, official leaderboards or the other makers' own reports. OSWorld 2.0 version 08.08, except 1.2 on 06.24. Agentic IF Index is a Meta-internal composite. Meta's table has no MRCR scores for Opus 5.

• DeepSeek-V4.1-Flash Table 3, p.33

  All models at max reasoning effort. Code-agent benchmarks use DeepSeek Harness Minimal mode with 1M context; DeepSWE uses mini-SWE; visual agent tasks use the Claude Code harness with 512K context. HLE for some models is the text-only subset. The Hugging Face card lists NL2Repo-Bench as 64.0; the report and change log say 65.4.

• GPT-6 Astra Launch post (openai.com/index/gpt-6-astra); system card §6, §10.1; GPT-6.1 Sol addendum §5, §9

  Coding and reasoning rows come from the GPT-6 Astra launch post (3 Sep 2026); OpenAI's own comparison table, settings as published there. The system card reports safety and Preparedness evaluations, not general capability benchmarks. Cyber and bio scores are at maximum reasoning effort; bio scores are refusal-adjusted. HealthBench scores are length-adjusted. GPT-6.1 Sol and some GPT-5.6 Sol values come from the GPT-6.1 Sol addendum (29 Sep 2026). Unverified: openai.com blocks automated access, so the coding and reasoning figures were read from copies of the launch post, and they don't appear in the system card or developer docs.

• Gemini 3.8 Flash p.5

  Results as of September 2026, as published in Google's model card. Prices in the card list $0.75 / $3.75 with $1.50 / $7.50 as the regular rate. LVBench value for 3.8 Flash is the agentic setting (static: 87.1%).

• Grok 4.7 §2–6, p.5–16

  Grok 4.7 at xhigh effort except DeepSWE, SWE-Marathon, CADGenBench and CyberGym (high). Grok 4.6 at high except FrontierSWE, EEBench and HealthBench (xhigh). Most results come from third-party evaluators, each running its own harness; competitor results use max effort (CADGenBench GPT-5.6 Sol at xhigh). CyberGym is run without production safeguards.

• Claude Opus 5 Table 8.1.A, p.148–149

  Claude Opus 5 results use adaptive thinking at max effort, averaged over five trials (ARC-AGI-3 at high effort). GPT-5.6 Sol's FrontierBench figure uses Codex and its ARC-AGI-1 figure is at xhigh. The Fable 5 HealthBench Professional figure carries a footnote reading "Mythos 5". Competitor figures come from the developers' own published results or leaderboards.

• Grok 4.6 §2–7, p.7–28

  Grok models at high effort (EEBench Grok 4.6 xhigh is 60.0%). Competitors at max effort (Fable 5 with fallback); GPT-5.6 Sol at xhigh on APEX-SWE and CADGenBench, high on DeepSearchQA. Most results come from third-party evaluators, each with its own harness. CyberGym is run without production safeguards.

• Mistral Large 4 launch post

  Preliminary, vendor-reported preview results. Many charts in the post give no numbers; only figures stated in text are listed. Blind human coding eval run by Surge AI on a 1–5 scale.

• Muse Spark 1.2 launch post charts; Muse Spark 1.2 & Muse Code evaluation methodology

  Values read from Meta's published bar charts. Terminal-Bench 2.1 and DeepSWE v1.1 run each model in its own agent product (Muse Code for 1.2, mini-swe-agent for 1.1, Claude Code, Codex, Grok Build, Antigravity CLI), pass@1 averaged over five attempts, so DeepSWE is not harness-identical to the official leaderboard. Muse Spark at xhigh, Opus 5 and GPT at max, Grok and Gemini at high. The MCP Atlas chart shows Opus 5 at xhigh and omits GPT 5.6 Terra, Grok 4.5 and Gemini 3.6 Flash. Meta Internal Coding Bench is 440 tasks from Meta's own codebase.

• Grok 4.5 §2–7, p.5–19

  Grok 4.5 at high effort; competitors at max (Fable 5 with fallback), GPT-5.5 at xhigh. Results come from mixed harnesses and evaluators; some competitor figures are taken from developers' own cards. CyberGym is run without production safeguards.

• Gemini 3.6 Flash p.4 (table also on the model card web page)

  Results as of July 2026, as published in Google's model card. Competitor figures are as listed by Google.