ProgramBench task detail

esubaalew__run.0fb9dec

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
1212generated behavioral tests
88.0%official best score
82.2%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 82.2% ok 996/1212 $3.77 75 0.21h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 80.3% ok 973/1212 $3.44 62 0.19h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 79.2% ok 960/1212 $3.83 72 0.20h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

82.2% from 996/1212 ProgramBench-scored tests. Raw public eval statuses: error: 26, failure: 269, passed: 1047, skipped: 165.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • c88ecf8e564f eval.tests.test_flag_combinations.test_python_all_aliases (failure)
  • c88ecf8e564f eval.tests.test_language_aliases.test_python_aliases (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

80.3% from 973/1212 ProgramBench-scored tests. Raw public eval statuses: error: 26, failure: 295, passed: 1021, skipped: 165.

Agent trace: 165 shell command(s), 135 executable-observation command(s), 9 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch.

  • c88ecf8e564f eval.tests.test_detection_comprehensive.test_detect_python_by_import (failure)
  • c88ecf8e564f eval.tests.test_detection_comprehensive.test_python_specific_syntax_detection (failure)
  • c88ecf8e564f eval.tests.test_error_handling.test_code_and_file_conflict (failure)
  • c88ecf8e564f eval.tests.test_detection_comprehensive.test_bash_specific_syntax_detection (failure)
  • c88ecf8e564f eval.tests.test_flag_combinations.test_python_all_aliases (failure)
  • c88ecf8e564f eval.tests.test_language_detection.test_python_syntax_detection (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 Claude Opus 4.7 (xhigh) Anthropic 88.0% $19.96 282
2 Claude Sonnet 4.6 Anthropic 85.2% $32.29 669
3 Claude Opus 4.6 Anthropic 85.1% $14.72 342
4 GPT 5.5 (xhigh) OpenAI 84.4% $6.28 78
5 GPT 5.5 (high) OpenAI 80.8% $3.33 44
6 Claude Opus 4.7 Anthropic 73.2% $1.91 73
7 GPT 5.5 OpenAI 72.6% $0.82 14
8 GPT 5.4 OpenAI 69.8% $0.21 11
9 Gemini 3.1 Pro Google 58.5% $3.42 156
10 Gemini 3 Flash Google 54.5% $0.26 85
11 Claude Haiku 4.5 Anthropic 42.3% $0.32 104
12 GPT 5.4 mini OpenAI 37.6% $0.03 18
13 GPT 5 mini OpenAI 1.7% $0.03 16