ProgramBench task detail

ekzhang__bore.8e059cd

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
394generated behavioral tests
93.1%official best score
87.8%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 87.8% ok 346/394 $3.72 77 0.22h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 77.2% ok 304/394 $3.69 81 0.19h not exported
3 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 30.5% ok 120/394 $3.44 75 0.17h manifest · eval json · eval summary · agent summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

87.8% from 346/394 ProgramBench-scored tests. Raw public eval statuses: failure: 60, passed: 390.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • 8f4b78a7e9eb tests.test_cli_basics.test_env_bore_server_used_when_no_to_flag (failure)
  • 8f4b78a7e9eb tests.test_cli_basics.test_server_min_port_explicit_1024 (failure)
  • 8f4b78a7e9eb tests.test_cli_basics.test_local_long_host_name (failure)
  • 8f4b78a7e9eb tests.test_client.test_port_boundary_minimum (failure)
  • 8f4b78a7e9eb tests.test_client.test_env_server_invalid (failure)
  • 8f4b78a7e9eb tests.test_client.test_port_boundary_maximum (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260518T232006Z

30.5% from 120/394 ProgramBench-scored tests. Raw public eval statuses: error: 1, failure: 177, not_run: 116, passed: 156.

Agent trace: 116 shell command(s), 75 executable-observation command(s), 5 build command(s), 1 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch.

  • d95dad695c95 eval.tests.test_01_basic_invocation.TestBasicInvocation.test_no_arguments (failure)
  • d95dad695c95 eval.tests.test_01_basic_invocation.TestBasicInvocation.test_invalid_flag (not_run)
  • d95dad695c95 eval.tests.test_01_basic_invocation.TestBasicInvocation.test_unknown_command (not_run)
  • d95dad695c95 eval.tests.test_01_basic_invocation.TestHelpSubcommand.test_help_command (not_run)
  • d95dad695c95 eval.tests.test_01_basic_invocation.TestHelpSubcommand.test_help_local (not_run)
  • d95dad695c95 eval.tests.test_01_basic_invocation.TestHelpSubcommand.test_help_server (not_run)

manifest · eval json · eval summary · agent summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 Claude Opus 4.7 (xhigh) Anthropic 93.1% $13.90 184
2 GPT 5.5 (high) OpenAI 92.4% $3.71 39
3 GPT 5.5 (xhigh) OpenAI 86.2% $4.88 60
4 Claude Opus 4.7 Anthropic 74.6% $5.59 112
5 Claude Opus 4.6 Anthropic 68.8% $8.20 193
6 GPT 5.5 OpenAI 65.5% $1.79 28
7 Claude Sonnet 4.6 Anthropic 48.2% $8.25 225
8 Gemini 3.1 Pro Google 44.2% $1.28 70
9 GPT 5.4 OpenAI 42.1% $0.56 14
10 Claude Haiku 4.5 Anthropic 33.5% $0.54 117
11 Gemini 3 Flash Google 28.7% $0.28 81
12 GPT 5 mini OpenAI 11.7% $0.02 16
13 GPT 5.4 mini OpenAI 11.4% $0.04 11