aiaimodel.cards
← All benchmarks

SWE-bench Multilingual

Fixing real issues in real open-source repositories; a patch counts only if the project's own tests pass. Variants differ in task set and languages.

Your workloadRAG over docs · 12K in · 600 out · 50K requests/mo · 30% cached
  1. 1Claude Opus 5.5 · Anthropicbest value
    93.9%
  2. 2Claude Sonnet 5.5 · Anthropicbest value
    90.3%
  3. 3Claude Opus 5 · Anthropic
    89.5%
  4. 4Claude Fable 5.1 · Anthropic
    89.1%
  5. 5Claude Opus 4.8 · Anthropic
    84.4%
  6. 6Claude Haiku 5.5 · Anthropicbest value
    83.7%
  7. 7Qwen3.8-Flash · Alibaba
    81.0%
  8. 8Claude Opus 4.7 · Anthropic
    80.5%
  9. 9Claude Sonnet 5 · Anthropic
    78.3%
  10. 10Grok 4.5 · xAI
    78.0%
  11. 11Qwen3.7-Plus · Alibaba
    75.8%

“Best value”: no other model here reports a higher score at a lower cost for your workload. Scores are self-reported by each maker with different settings, so treat close calls as ties.

How each maker reported it

  • Claude Opus 5.5 Table 8.1.A, p.174

    Claude results use adaptive thinking at max effort, averaged over five trials (Terminal-Bench 4.0 at xhigh). Competitor figures come from the developers' own published results.

  • Claude Sonnet 5.5 Table 8.1.A, p.109

    Claude results use adaptive thinking at max effort with default sampling, averaged over five trials. Competitor figures come from the developers' own published system cards or leaderboards.

  • Claude Opus 5 Table 8.1.A, p.148–149

    Claude Opus 5 results use adaptive thinking at max effort, averaged over five trials (ARC-AGI-3 at high effort). GPT-5.6 Sol's FrontierBench figure uses Codex and its ARC-AGI-1 figure is at xhigh. The Fable 5 HealthBench Professional figure carries a footnote reading "Mythos 5". Competitor figures come from the developers' own published results or leaderboards.

  • Claude Fable 5.1 Table 8.1.A, p.167

    Fable 5.1 and Mythos 5.1 share weights, so column 0 covers both (likewise Fable 5 / Mythos 5). Claude results use adaptive thinking at max effort, averaged over five trials. Terminal-Bench 4.0 values are from §8.6, p.171 (Mythos 5.1: 60.9%; GPT-5.6 Sol from the public leaderboard). Competitor figures come from the developers' own published results or leaderboards.

  • Claude Opus 4.8 Table 8.1.A, p.194–195

    Opus 4.8 uses adaptive thinking at max effort with default sampling, averaged over 5 trials. Competitor figures come from the developers' own published cards or leaderboards.

  • Claude Haiku 5.5 Table 8.1.A, p.111

    Haiku 5.5 results use adaptive thinking at max effort with default sampling, averaged over five trials. FrontierCode is shown at max effort (Sonnet 5.5 scores 52.1 at xhigh). OSWorld is the offline subset; GPT-6 Luna was run by Anthropic on the same tasks. Other competitor figures come from the developers' own published results.

  • Qwen3.8-Flash model card (Benchmark Results)

    Qwen's own tables. SWE tasks use Claude Code or mini-SWE-agent at temperature 1.0 and 256K context; HLE judged by GPT-4o. OSWorld 2.0 and two-setting rows are split. DeepSeek-V4-Flash is not in the vision table.

  • Claude Opus 4.7 Table 8.1.A, p.191–192

    Opus 4.7 uses adaptive thinking at max effort with default sampling, averaged over 5 trials (Terminal-Bench 2.0 with thinking disabled; Finance Agent at high effort). Competitor figures come from the developers' own published cards or leaderboards.

  • Claude Sonnet 5 Table 8.1.A, p.115–116; SWE-bench variants §8.2, p.116

    Claude Sonnet 5 results use adaptive thinking at max effort, averaged over five trials. BrowseComp is the single-agent score (86.6% multi-agent) with a 10M-token limit and compaction. GPT-5.5 Terminal-Bench 2.1 figure uses Codex CLI. Competitor figures come from the developers' own published results.

  • Grok 4.5 §2–7, p.5–19

    Grok 4.5 at high effort; competitors at max (Fable 5 with fallback), GPT-5.5 at xhigh. Results come from mixed harnesses and evaluators; some competitor figures are taken from developers' own cards. CyberGym is run without production safeguards.

  • Qwen3.7-Plus Qwen3.8-Flash-Next model card (Benchmark Results)

    Qwen published Qwen3.7-Plus scores as a comparison column in its Qwen3.8 cards. The Qwen3.8-27B card reports slightly different Qwen3.7-Plus numbers for some rows (e.g. SWE-bench Pro 57.6, DeepSWE 14.2).

Only each model's own maker's number is shown; scores a maker reports for rival models are left out.