ProgramBench task detail

kyoh86__richgo.313114f

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
546generated behavioral tests
88.1%official best score
86.6%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 86.6% ok 473/546 $3.50 83 0.17h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 20.0% ok 109/546 $3.48 60 0.21h not exported
3 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 17.8% ok 97/546 $2.38 43 0.14h manifest · eval json · eval summary · agent summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

86.6% from 473/546 ProgramBench-scored tests. Raw public eval statuses: failure: 90, passed: 696, skipped: 1.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • aedc381a5a8d tests.test_config.test_config_invalid_yaml_logs_error_continues (failure)
  • aedc381a5a8d tests.test_config_gaps.test_label_type_short_single_char_labels (failure)
  • aedc381a5a8d tests.test_config_gaps.test_label_type_none_suppresses_labels (failure)
  • aedc381a5a8d tests.test_edge_cases.test_testfilter_with_only_carriage_returns (failure)
  • aedc381a5a8d tests.test_edge_cases.test_testfilter_empty_stdin (failure)
  • aedc381a5a8d tests.test_core.test_test_command_with_short_flag (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260518T232006Z

17.8% from 97/546 ProgramBench-scored tests. Raw public eval statuses: failure: 584, passed: 202, skipped: 1.

Agent trace: 84 shell command(s), 56 executable-observation command(s), 12 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch.

  • 7e3fefff015d eval.tests.test_basic_invocation.TestTestFilterMode.test_testfilter_basic_pass (failure)
  • 7e3fefff015d eval.tests.test_configuration.TestConfigurationLoading.test_config_from_cwd_yml (failure)
  • 7e3fefff015d eval.tests.test_advanced_config.TestConfigFileFormats.test_yml_extension (failure)
  • 7e3fefff015d eval.tests.test_configuration.TestConfigurationOptions.test_label_type_short (failure)
  • 7e3fefff015d eval.tests.test_basic_invocation.TestTestFilterMode.test_testfilter_coverage_output (failure)
  • 7e3fefff015d eval.tests.test_advanced_config.TestStyleCombinations.test_all_style_types (failure)

manifest · eval json · eval summary · agent summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (high) OpenAI 88.1% $3.00 54
2 Claude Opus 4.7 (xhigh) Anthropic 87.4% $16.10 232
3 GPT 5.5 OpenAI 87.2% $1.25 20
4 Claude Opus 4.6 Anthropic 85.0% $12.87 312
5 Claude Opus 4.7 Anthropic 83.7% $6.01 166
6 Claude Sonnet 4.6 Anthropic 77.7% $8.23 313
7 Gemini 3 Flash Google 59.2% $0.36 120
8 Gemini 3.1 Pro Google 56.6% $1.48 121
9 GPT 5.4 OpenAI 54.4% $0.26 10
10 GPT 5.5 (xhigh) OpenAI 51.8% $4.11 56
11 Claude Haiku 4.5 Anthropic 48.7% $1.04 156
12 GPT 5.4 mini OpenAI 8.2% $0.02 8
13 GPT 5 mini OpenAI 6.4% $0.02 14