ProgramBench task detail

bootandy__dust.62bf1e1

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
584generated behavioral tests
87.3%official best score
75.3%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 75.3% ok 440/584 $6.85 107 0.35h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 68.5% ok 400/584 n/a 62 0.21h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 36.5% ok 213/584 n/a 92 0.31h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

75.3% from 440/584 ProgramBench-scored tests. Raw public eval statuses: failure: 184, passed: 765, skipped: 16.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch, terminal rendering mismatch.

  • 2670a863a199 eval.tests.test_size_filtering.test_depth_zero (failure)
  • 24c039380685 eval.tests.test_error_handling.test_invalid_regex_pattern (failure)
  • 24c039380685 eval.tests.test_error_handling.test_nonexistent_ignore_file (failure)
  • 746762709b06 tests.test_cli.test_full_paths_flag (failure)
  • 746762709b06 tests.test_cli.test_depth_zero (failure)
  • 746762709b06 tests.test_cli.test_conflicting_file_types_depth (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

68.5% from 400/584 ProgramBench-scored tests. Raw public eval statuses: failure: 229, passed: 720, skipped: 16.

Agent trace: 138 shell command(s), 104 executable-observation command(s), 13 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, terminal rendering mismatch.

  • 2670a863a199 eval.tests.test_input_handling.test_invalid_path (failure)
  • 2670a863a199 eval.tests.test_error_handling.test_print_errors_flag (failure)
  • 2670a863a199 eval.tests.test_size_filtering.test_depth_zero (failure)
  • 24c039380685 eval.tests.test_basic_invocation.test_nonexistent_directory (failure)
  • 24c039380685 eval.tests.test_config.test_invalid_config_file (failure)
  • 24c039380685 eval.tests.test_error_handling.test_conflicting_only_dir_and_only_file (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 87.3% $12.40 89
2 Claude Opus 4.7 (xhigh) Anthropic 81.3% $42.44 255
3 GPT 5.5 (high) OpenAI 80.5% $3.25 28
4 Claude Opus 4.7 Anthropic 70.9% $12.12 171
5 Claude Opus 4.6 Anthropic 70.0% $15.78 245
6 Claude Sonnet 4.6 Anthropic 68.7% $45.15 499
7 GPT 5.5 OpenAI 61.8% $1.19 14
8 GPT 5.4 OpenAI 52.2% $0.46 11
9 Gemini 3.1 Pro Google 45.7% $1.32 88
10 Claude Haiku 4.5 Anthropic 42.8% $1.46 164
11 Gemini 3 Flash Google 28.3% $0.21 53
12 GPT 5.4 mini OpenAI 18.3% $0.04 8
13 GPT 5 mini OpenAI 2.1% $0.05 18