ProgramBench task detail

jarun__nnn.cb2c535

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
477generated behavioral tests
98.1%official best score
98.1%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 98.1% ok 468/477 $3.71 73 0.18h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 93.5% ok 446/477 n/a 58 0.14h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 62.1% ok 296/477 n/a 77 0.20h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

98.1% from 468/477 ProgramBench-scored tests. Raw public eval statuses: failure: 590, passed: 1201, skipped: 5.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • 316c1d019b0a eval.tests.test_with_terminal.test_tmux_multiple_files (failure)
  • 316c1d019b0a eval.tests.test_with_terminal.test_tmux_shows_file_count (failure)
  • 316c1d019b0a eval.tests.test_with_terminal.test_tmux_detail_mode (failure)
  • 316c1d019b0a eval.tests.test_more_coverage.test_run_3 (failure)
  • 316c1d019b0a eval.tests.test_with_terminal.test_tmux_basic_launch (failure)
  • 316c1d019b0a eval.tests.test_with_terminal_advanced.test_delete_file (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

93.5% from 446/477 ProgramBench-scored tests. Raw public eval statuses: failure: 588, passed: 1198, skipped: 10.

Agent trace: 55 shell command(s), 31 executable-observation command(s), 3 build command(s), 1 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch.

  • 316c1d019b0a eval.tests.test_additional_flags.test_internal_editor_flag (failure)
  • 316c1d019b0a eval.tests.test_additional_flags.test_all_flags_combined (failure)
  • 316c1d019b0a eval.tests.test_with_terminal.test_tmux_multiple_files (failure)
  • 316c1d019b0a eval.tests.test_with_terminal.test_tmux_detail_mode (failure)
  • 316c1d019b0a eval.tests.test_with_terminal.test_tmux_shows_file_count (failure)
  • 316c1d019b0a eval.tests.test_more_coverage.test_run_3 (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (high) OpenAI 98.1% $3.95 36
2 Claude Opus 4.6 Anthropic 98.1% $14.07 334
3 Claude Opus 4.7 (xhigh) Anthropic 97.9% $1.88 70
4 Claude Opus 4.7 Anthropic 96.6% $1.32 56
5 Claude Sonnet 4.6 Anthropic 96.6% $10.47 347
6 GPT 5.5 OpenAI 96.4% $1.22 17
7 Claude Haiku 4.5 Anthropic 88.7% $0.52 123
8 GPT 5.4 OpenAI 76.7% $0.17 8
9 GPT 5.4 mini OpenAI 69.8% $0.03 13
10 GPT 5.5 (xhigh) OpenAI 64.8% $7.36 53
11 Gemini 3.1 Pro Google 59.7% $0.88 70
12 GPT 5 mini OpenAI 51.4% $0.02 10
13 Gemini 3 Flash Google 13.6% $0.24 87