ProgramBench task detail

agourlay__zip-password-finder.704700d

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
680generated behavioral tests
98.7%official best score
94.1%Codex best score
2Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 94.1% ok 640/680 n/a 48 0.14h manifest · eval json · eval summary · agent summary · usage audit
2 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 92.5% ok 629/680 $1.86 37 0.22h manifest · eval json · eval summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260518T232006Z

94.1% from 640/680 ProgramBench-scored tests. Raw public eval statuses: failure: 40, passed: 751, skipped: 1.

Agent trace: 97 shell command(s), 65 executable-observation command(s), 6 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch.

  • 3001c1482b82 eval.tests.test_error_handling.test_starting_password_with_dictionary_error (failure)
  • 3001c1482b82 eval.tests.test_error_handling.test_starting_password_too_long_error (failure)
  • 3001c1482b82 eval.tests.test_error_handling.test_starting_password_too_short_error (failure)
  • 3001c1482b82 eval.tests.test_error_handling.test_starting_password_out_of_charset_error (failure)
  • 644934f68973 eval.tests.test_starting_password.test_starting_password_outside_charset (failure)
  • 644934f68973 eval.tests.test_starting_password.test_starting_password_with_dictionary (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260517T094700Z

92.5% from 629/680 ProgramBench-scored tests. Raw public eval statuses: failure: 51, passed: 740, skipped: 1.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • 3001c1482b82 eval.tests.test_error_handling.test_max_password_length_zero_error (failure)
  • 3001c1482b82 eval.tests.test_error_handling.test_starting_password_too_long_error (failure)
  • 3001c1482b82 eval.tests.test_error_handling.test_starting_password_with_dictionary_error (failure)
  • 3001c1482b82 eval.tests.test_error_handling.test_starting_password_too_short_error (failure)
  • 3001c1482b82 eval.tests.test_error_handling.test_starting_password_out_of_charset_error (failure)
  • 644934f68973 eval.tests.test_password_length.test_max_password_len_zero (failure)

manifest · eval json · eval summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 98.7% $4.94 59
2 Claude Opus 4.6 Anthropic 97.9% $3.51 129
3 GPT 5.5 (high) OpenAI 97.4% $1.90 29
4 Claude Opus 4.7 Anthropic 97.4% $2.53 78
5 GPT 5.5 OpenAI 90.3% $1.19 22
6 Gemini 3.1 Pro Google 90.3% $1.16 86
7 Claude Opus 4.7 (xhigh) Anthropic 79.4% $6.96 143
8 Claude Sonnet 4.6 Anthropic 79.1% $8.63 295
9 Gemini 3 Flash Google 72.9% $0.26 118
10 Claude Haiku 4.5 Anthropic 39.0% $0.53 94
11 GPT 5.4 mini OpenAI 27.4% $0.05 9
12 GPT 5.4 OpenAI 27.2% $0.19 10
13 GPT 5 mini OpenAI 5.4% $0.02 18