ProgramBench task detail

abishekvashok__cmatrix.5c082c6

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
507generated behavioral tests
100.0%official best score
100.0%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 100.0% ok 507/507 $3.54 55 0.17h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 97.4% ok 494/507 $1.81 41 0.10h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 84.4% ok 428/507 $3.03 73 0.17h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

100.0% from 507/507 ProgramBench-scored tests. Raw public eval statuses: failure: 1, passed: 768.

Agent trace summary unavailable.

Likely miss class: exact version/output mismatch.

  • 60fce8bbb0c8 eval.tests.test_help_and_version.test_version_flag (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

97.4% from 494/507 ProgramBench-scored tests. Raw public eval statuses: failure: 15, passed: 754.

Agent trace: 70 shell command(s), 46 executable-observation command(s), 6 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • 79b69dd3fd98 tests.test_error_handling.test_invalid_color (failure)
  • 79b69dd3fd98 tests.test_error_handling.test_invalid_color_numeric (failure)
  • 79b69dd3fd98 tests.test_error_handling.test_invalid_color_typo (failure)
  • 1b991a57d4e9 eval.tests.test_cmatrix.TestColorOptions.test_color_invalid_shows_error_message (failure)
  • f53d318371c8 eval.tests.test_args_parsing.test_u_valid_values_start_matrix_render_loop[args3] (failure)
  • f53d318371c8 eval.tests.test_args_parsing.test_u_valid_values_start_matrix_render_loop[args1] (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 100.0% $4.84 40
2 GPT 5.5 (high) OpenAI 100.0% $3.17 34
3 GPT 5.5 OpenAI 99.2% $1.04 17
4 Claude Opus 4.6 Anthropic 97.0% $5.99 139
5 Claude Opus 4.7 (xhigh) Anthropic 96.3% $10.74 178
6 Claude Sonnet 4.6 Anthropic 95.9% $13.49 292
7 Claude Opus 4.7 Anthropic 94.9% $4.44 130
8 Gemini 3.1 Pro Google 93.9% $2.94 84
9 GPT 5.4 OpenAI 91.7% $0.37 8
10 Claude Haiku 4.5 Anthropic 85.6% $0.44 72
11 GPT 5.4 mini OpenAI 79.7% $0.03 28
12 GPT 5 mini OpenAI 72.2% $0.01 7
13 Gemini 3 Flash Google 65.5% $0.17 51