ProgramBench task detail

chirlu__sox.42b3557

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
1202generated behavioral tests
38.9%official best score
50.3%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 50.3% ok 605/1202 $5.04 65 0.27h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 49.3% ok 592/1202 $3.82 44 0.19h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 42.0% ok 505/1202 $3.22 38 0.19h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

50.3% from 605/1202 ProgramBench-scored tests. Raw public eval statuses: failure: 598, passed: 636, skipped: 26.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • 86f3178a4b28 eval.tests.test_effects.test_gain_reduces_volume (failure)
  • 86f3178a4b28 eval.tests.test_effects.test_highpass_filter (failure)
  • 86f3178a4b28 eval.tests.test_effects.test_dcshift_effect (failure)
  • 86f3178a4b28 eval.tests.test_effects.test_echos_effect (failure)
  • 86f3178a4b28 eval.tests.test_error_handling.test_invalid_format_name (failure)
  • 86f3178a4b28 eval.tests.test_error_handling.test_missing_effect_argument (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

49.3% from 592/1202 ProgramBench-scored tests. Raw public eval statuses: failure: 611, passed: 623, skipped: 26.

Agent trace: 87 shell command(s), 61 executable-observation command(s), 5 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • 86f3178a4b28 eval.tests.test_audio_formats.test_add_comment_aiff (failure)
  • 86f3178a4b28 eval.tests.test_audio_formats.test_endian_big (failure)
  • 86f3178a4b28 eval.tests.test_audio_formats.test_au_ulaw (failure)
  • 86f3178a4b28 eval.tests.test_effects.test_highpass_filter (failure)
  • 86f3178a4b28 eval.tests.test_effects.test_echos_effect (failure)
  • 86f3178a4b28 eval.tests.test_effects.test_dcshift_effect (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (high) OpenAI 38.9% $4.14 39
2 Claude Opus 4.6 Anthropic 37.9% $10.13 233
3 GPT 5.5 OpenAI 29.7% $1.30 16
4 Claude Sonnet 4.6 Anthropic 29.2% $13.13 287
5 GPT 5.4 OpenAI 13.3% $0.23 7
6 Gemini 3 Flash Google 8.6% $0.29 66
7 Claude Opus 4.7 (xhigh) Anthropic 8.1% $7.08 125
8 Gemini 3.1 Pro Google 7.2% $1.15 53
9 Claude Opus 4.7 Anthropic 3.9% $1.25 49
10 GPT 5 mini OpenAI 3.6% $0.03 10
11 Claude Haiku 4.5 Anthropic 3.1% $0.86 125
12 GPT 5.4 mini OpenAI 1.3% $0.04 16
13 GPT 5.5 (xhigh) OpenAI n/a $79.42 1,000