ProgramBench task detail

burntsushi__xsv.f430466

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
1182generated behavioral tests
90.7%official best score
88.8%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 88.8% ok 1050/1182 $4.00 85 0.25h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 81.9% ok 968/1182 $2.55 48 0.17h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 74.5% ok 881/1182 $3.35 57 0.19h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

88.8% from 1050/1182 ProgramBench-scored tests. Raw public eval statuses: failure: 133, passed: 1183, skipped: 1.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch, exact version/output mismatch, interactive terminal behavior mismatch.

  • e75089074c1a tests.test_basic.test_no_arguments (failure)
  • e75089074c1a tests.test_edge_cases.test_conflicting_slice_options (failure)
  • e75089074c1a tests.test_index.test_index_with_output (failure)
  • 8b960294e03b tests.test_basic_invocation.test_no_arguments (failure)
  • 8b960294e03b tests.test_cat.test_cat_help (failure)
  • 8b960294e03b tests.test_count.test_count_help (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

81.9% from 968/1182 ProgramBench-scored tests. Raw public eval statuses: error: 1, failure: 218, passed: 1097, skipped: 1.

Agent trace: 40 shell command(s), 27 executable-observation command(s), 5 build command(s), 1 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • e75089074c1a tests.test_basic.test_no_arguments (failure)
  • e75089074c1a tests.test_basic.test_invalid_command (failure)
  • e75089074c1a tests.test_edge_cases.test_invalid_delimiter (failure)
  • e75089074c1a tests.test_edge_cases.test_conflicting_slice_options (failure)
  • e75089074c1a tests.test_frequency.test_frequency_no_nulls (failure)
  • e75089074c1a tests.test_index.test_index_with_output (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 90.7% $11.17 93
2 GPT 5.5 (high) OpenAI 87.1% $2.70 40
3 Claude Opus 4.7 (xhigh) Anthropic 85.8% $24.54 350
4 Claude Opus 4.6 Anthropic 82.7% $17.25 288
5 Claude Opus 4.7 Anthropic 82.0% $4.51 106
6 Claude Sonnet 4.6 Anthropic 80.6% $15.53 367
7 GPT 5.5 OpenAI 77.2% $1.18 21
8 GPT 5.4 OpenAI 65.6% $0.32 8
9 Gemini 3 Flash Google 39.6% $0.33 97
10 Gemini 3.1 Pro Google 32.2% $1.10 78
11 GPT 5.4 mini OpenAI 6.3% $0.02 11
12 GPT 5 mini OpenAI 2.7% $0.03 18
13 Claude Haiku 4.5 Anthropic 0.0% $0.89 137