ProgramBench task detail

oppiliappan__statix.e9df54c

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
815generated behavioral tests
69.0%official best score
67.6%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 67.6% ok 551/815 $5.82 101 0.30h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 53.0% ok 432/815 $4.21 63 0.22h not exported
3 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 52.9% ok 431/815 $3.05 59 0.17h manifest · eval json · eval summary · agent summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

67.6% from 551/815 ProgramBench-scored tests. Raw public eval statuses: failure: 265, passed: 703, skipped: 4.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • 5abb696a5d96 eval.tests.test_basic_invocation.test_no_args (failure)
  • 5abb696a5d96 eval.tests.test_config_handling.test_config_nonexistent_path (failure)
  • 5abb696a5d96 eval.tests.test_dump_subcommand.test_dump_no_extra_args (failure)
  • 5abb696a5d96 eval.tests.test_config_handling.test_config_invalid_toml (failure)
  • 5abb696a5d96 eval.tests.test_check_subcommand.test_check_nonexistent_file (failure)
  • 5abb696a5d96 eval.tests.test_edge_cases.test_stdin_empty (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260518T232006Z

52.9% from 431/815 ProgramBench-scored tests. Raw public eval statuses: failure: 390, passed: 578, skipped: 4.

Agent trace: 68 shell command(s), 48 executable-observation command(s), 6 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch.

  • 5abb696a5d96 eval.tests.test_basic_invocation.test_no_args (failure)
  • 5abb696a5d96 eval.tests.test_check_subcommand.test_check_stdin_mode (failure)
  • 5abb696a5d96 eval.tests.test_config_handling.test_config_nonexistent_path (failure)
  • 5abb696a5d96 eval.tests.test_dump_subcommand.test_dump_no_extra_args (failure)
  • 5abb696a5d96 eval.tests.test_config_handling.test_config_invalid_toml (failure)
  • 5abb696a5d96 eval.tests.test_edge_cases.test_fix_stdin_no_issues (failure)

manifest · eval json · eval summary · agent summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 69.0% $9.28 83
2 GPT 5.5 (high) OpenAI 65.2% $4.29 48
3 GPT 5.5 OpenAI 62.1% $1.19 18
4 Claude Opus 4.7 Anthropic 42.8% $3.28 89
5 Claude Opus 4.7 (xhigh) Anthropic 40.7% $6.92 142
6 Gemini 3 Flash Google 30.8% $0.20 51
7 Gemini 3.1 Pro Google 29.3% $1.18 60
8 GPT 5.4 OpenAI 25.4% $0.30 11
9 GPT 5.4 mini OpenAI 24.0% $0.02 8
10 GPT 5 mini OpenAI 21.7% $0.01 18
11 Claude Opus 4.6 Anthropic 11.3% $14.17 282
12 Claude Sonnet 4.6 Anthropic 11.3% $24.17 391
13 Claude Haiku 4.5 Anthropic 0.0% $0.94 140