ProgramBench task detail

hush-shell__hush.560c33a

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
1201generated behavioral tests
83.3%official best score
12.0%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 12.0% ok 144/1201 $6.39 112 0.32h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 8.3% ok 100/1201 $6.14 83 0.30h not exported
3 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 7.2% ok 87/1201 $4.04 56 0.24h manifest · eval json · eval summary · agent summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

12.0% from 144/1201 ProgramBench-scored tests. Raw public eval statuses: failure: 1117, passed: 181.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • ad3448d10aa5 eval.tests.test_advanced_features.test_function_with_no_params (failure)
  • ad3448d10aa5 eval.tests.test_advanced_features.test_string_indexing (failure)
  • ad3448d10aa5 eval.tests.test_commands.test_command_with_variable (failure)
  • ad3448d10aa5 eval.tests.test_comprehensive_coverage.test_sorting (failure)
  • ad3448d10aa5 eval.tests.test_basic_invocation.test_invalid_flag (failure)
  • ad3448d10aa5 eval.tests.test_command_features.test_command_multiple_commands (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260518T232006Z

7.2% from 87/1201 ProgramBench-scored tests. Raw public eval statuses: failure: 1169, passed: 129.

Agent trace: 134 shell command(s), 94 executable-observation command(s), 16 build command(s), 1 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • ad3448d10aa5 eval.tests.test_commands.test_command_with_numbers (failure)
  • ad3448d10aa5 eval.tests.test_advanced_features.test_string_indexing (failure)
  • ad3448d10aa5 eval.tests.test_comprehensive_coverage.test_dict_operations (failure)
  • ad3448d10aa5 eval.tests.test_basic_invocation.test_invalid_flag (failure)
  • ad3448d10aa5 eval.tests.test_commands.test_command_with_variable (failure)
  • ad3448d10aa5 eval.tests.test_advanced_features.test_string_length (failure)

manifest · eval json · eval summary · agent summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 Claude Opus 4.6 Anthropic 83.3% $16.84 374
2 Claude Sonnet 4.6 Anthropic 76.6% $67.34 1,000
3 GPT 5.5 (high) OpenAI 68.2% $5.33 56
4 Claude Opus 4.7 (xhigh) Anthropic 37.2% $2.15 57
5 Claude Opus 4.7 Anthropic 15.7% $0.75 28
6 Gemini 3.1 Pro Google 15.1% $1.01 61
7 GPT 5.5 OpenAI 13.1% $1.83 33
8 Gemini 3 Flash Google 8.7% $0.48 158
9 GPT 5.4 mini OpenAI 8.4% $0.03 19
10 GPT 5 mini OpenAI 6.9% $0.03 12
11 GPT 5.4 OpenAI 6.2% $0.23 10
12 Claude Haiku 4.5 Anthropic 4.6% $0.72 87
13 GPT 5.5 (xhigh) OpenAI 0.8% $15.30 119