ProgramBench task detail

dalance__amber.69a0f52

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
565generated behavioral tests
74.3%official best score
72.2%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 72.2% ok 408/565 $1.96 38 0.12h manifest · eval json · eval summary · agent summary · usage audit
2 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 72.0% ok 407/565 $2.36 42 0.15h not exported
3 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 19.3% ok 109/565 $2.68 47 0.15h manifest · eval json · eval summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260518T232006Z

72.2% from 408/565 ProgramBench-scored tests. Raw public eval statuses: failure: 184, not_run: 19, passed: 577, skipped: 5.

Agent trace: 51 shell command(s), 32 executable-observation command(s), 5 build command(s), 0 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, interactive terminal behavior mismatch.

  • 12565bd8e77e eval.tests.test_comprehensive.test_binary_file_skipped_by_default (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_statistics_output (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_key_from_empty_file_error (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_skip_vcs_directories (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_no_matches_found_behavior (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_file_output_enabled (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260517T094700Z

19.3% from 109/565 ProgramBench-scored tests. Raw public eval statuses: failure: 121, not_run: 434, passed: 230.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch, interactive terminal behavior mismatch.

  • 12565bd8e77e eval.tests.test_comprehensive.test_non_interactive_mode (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_no_color_output (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_directory_recursive_search (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_regex_named_capture_groups (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_key_from_file (failure)
  • 12565bd8e77e eval.tests.test_comprehensive.test_single_file_replacement (failure)

manifest · eval json · eval summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 Claude Opus 4.7 (xhigh) Anthropic 74.3% $16.71 329
2 GPT 5.5 OpenAI 72.6% $0.78 20
3 Claude Sonnet 4.6 Anthropic 71.1% $23.27 507
4 Claude Opus 4.6 Anthropic 68.6% $14.47 329
5 Claude Opus 4.7 Anthropic 67.9% $2.97 96
6 Gemini 3 Flash Google 64.3% $0.30 126
7 GPT 5.4 OpenAI 63.4% $4.75 1,000
8 Gemini 3.1 Pro Google 60.7% $0.93 91
9 Claude Haiku 4.5 Anthropic 47.4% $0.33 81
10 GPT 5.5 (xhigh) OpenAI 23.2% $4.70 77
11 GPT 5.5 (high) OpenAI 22.3% $2.07 44
12 GPT 5 mini OpenAI 12.4% $0.01 8
13 GPT 5.4 mini OpenAI 3.4% $0.02 7