ProgramBench task detail

foriequal0__git-trim.07c2f50

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
509generated behavioral tests
79.0%official best score
71.3%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 71.3% ok 363/509 $4.09 71 0.22h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 56.0% ok 285/509 n/a 64 0.25h not exported
3 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 37.7% ok 192/509 n/a 79 0.23h manifest · eval json · eval summary · agent summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

71.3% from 363/509 ProgramBench-scored tests. Raw public eval statuses: failure: 151, passed: 559.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • 80d1d2869975 eval.tests.test_bases_configuration.test_bases_with_alias_base (failure)
  • 80d1d2869975 eval.tests.test_actual_deletion.test_actual_merged_branch_deletion (failure)
  • 80d1d2869975 eval.tests.test_actual_deletion.test_dry_run_vs_actual_deletion (failure)
  • a03959b491c9 tests.test_conflicting_flags.test_conflicting_confirm_flags (failure)
  • a03959b491c9 tests.test_conflicting_flags.test_base_alias (failure)
  • a03959b491c9 tests.test_conflicting_flags.test_conflicting_detach_flags (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260518T232006Z

37.7% from 192/509 ProgramBench-scored tests. Raw public eval statuses: failure: 333, passed: 377.

Agent trace: 121 shell command(s), 45 executable-observation command(s), 4 build command(s), 5 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch.

  • 80d1d2869975 eval.tests.test_bases_configuration.test_bases_with_alias_base (failure)
  • 80d1d2869975 eval.tests.test_detach_control.test_no_detach_flag (failure)
  • 80d1d2869975 eval.tests.test_delete_ranges.test_delete_merged_with_remote (failure)
  • 80d1d2869975 eval.tests.test_bases_configuration.test_bases_flag_multiple_branches (failure)
  • 80d1d2869975 eval.tests.test_dry_run.test_dry_run_output_format (failure)
  • 80d1d2869975 eval.tests.test_dry_run.test_dry_run_flag (failure)

manifest · eval json · eval summary · agent summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 79.0% $7.04 59
2 GPT 5.5 (high) OpenAI 65.0% $2.95 41
3 Claude Opus 4.6 Anthropic 64.6% $11.85 321
4 Claude Sonnet 4.6 Anthropic 57.0% $13.30 334
5 Claude Opus 4.7 Anthropic 52.7% $2.67 82
6 GPT 5.5 OpenAI 50.7% $0.91 16
7 Claude Opus 4.7 (xhigh) Anthropic 48.5% $4.29 114
8 GPT 5.4 OpenAI 45.4% $0.28 15
9 Claude Haiku 4.5 Anthropic 41.1% $0.70 138
10 Gemini 3 Flash Google 38.9% $0.37 75
11 GPT 5.4 mini OpenAI 11.6% $0.06 13
12 GPT 5 mini OpenAI 10.4% $0.02 17
13 Gemini 3.1 Pro Google n/a $1.57 98