ProgramBench task detail

mkj__dropbear.75f699b

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
678generated behavioral tests
65.5%official best score
48.5%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 48.5% ok 329/678 $3.02 54 0.16h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 48.5% ok 329/678 $2.44 53 0.17h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 46.8% ok 317/678 $2.77 52 0.15h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

48.5% from 329/678 ProgramBench-scored tests. Raw public eval statuses: error: 243, failure: 261, passed: 564, skipped: 7.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch, interactive terminal behavior mismatch.

  • 564b53891395 eval.tests.test_dropbear_cli.test_invalid_port_non_numeric (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_multiple_ports (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_address_port_format (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_pid_file_creation (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_foreground_mode (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_receive_window_buffer_option (error)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

48.5% from 329/678 ProgramBench-scored tests. Raw public eval statuses: error: 198, failure: 294, passed: 576, skipped: 7.

Agent trace: 117 shell command(s), 91 executable-observation command(s), 5 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, interactive terminal behavior mismatch.

  • 564b53891395 eval.tests.test_dropbear_cli.test_invalid_port_non_numeric (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_multiple_ports (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_address_port_format (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_pid_file_creation (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_foreground_mode (error)
  • 564b53891395 eval.tests.test_dropbear_cli.test_receive_window_buffer_option (error)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 65.5% $5.49 64
2 Claude Opus 4.7 (xhigh) Anthropic 65.0% $3.00 72
3 GPT 5.4 OpenAI 58.1% $0.15 10
4 Claude Opus 4.6 Anthropic 51.8% $8.15 152
5 GPT 5.5 (high) OpenAI 50.1% $2.08 30
6 GPT 5.5 OpenAI 46.5% $0.96 18
7 Gemini 3 Flash Google 44.5% $0.19 56
8 Claude Sonnet 4.6 Anthropic 41.6% $67.81 600
9 Claude Opus 4.7 Anthropic 40.6% $1.62 56
10 Gemini 3.1 Pro Google 38.3% $1.05 69
11 Claude Haiku 4.5 Anthropic 26.5% $0.26 78
12 GPT 5.4 mini OpenAI 22.0% $0.02 7
13 GPT 5 mini OpenAI 17.0% $0.01 12