ProgramBench task detail

rcoh__angle-grinder.9c2fc88

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
1130generated behavioral tests
73.3%official best score
62.4%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 62.4% ok 705/1130 $7.08 120 0.38h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 55.8% ok 630/1130 $5.22 90 0.26h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 12.6% ok 142/1130 $5.07 74 0.25h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

62.4% from 705/1130 ProgramBench-scored tests. Raw public eval statuses: failure: 428, passed: 715.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • 9eded8203941 eval.tests.test_agrind.test_no_args_returns_error (failure)
  • 9eded8203941 eval.tests.test_agrind.test_output_json_sorted_agg (failure)
  • 9eded8203941 eval.tests.test_agrind.test_sort_by_multiple_fields (failure)
  • 9eded8203941 eval.tests.test_agrind.test_nonexistent_file_returns_error (failure)
  • 9eded8203941 eval.tests.test_agrind.test_alias_nginx (failure)
  • 9eded8203941 eval.tests.test_agrind.test_count_distinct_missing_arg_error (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

55.8% from 630/1130 ProgramBench-scored tests. Raw public eval statuses: failure: 503, passed: 640.

Agent trace: 194 shell command(s), 156 executable-observation command(s), 10 build command(s), 1 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, interactive terminal behavior mismatch.

  • 9eded8203941 eval.tests.test_agrind.test_parse_operator_with_from (failure)
  • 9eded8203941 eval.tests.test_agrind.test_no_args_returns_error (failure)
  • 9eded8203941 eval.tests.test_agrind.test_avg_alias_for_average (failure)
  • 9eded8203941 eval.tests.test_agrind.test_sort_by_multiple_fields (failure)
  • 9eded8203941 eval.tests.test_agrind.test_output_json_sorted_agg (failure)
  • 9eded8203941 eval.tests.test_agrind.test_json_from_field (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 73.3% $10.07 111
2 GPT 5.5 (high) OpenAI 64.8% $4.46 44
3 Claude Opus 4.7 (xhigh) Anthropic 50.4% $11.18 211
4 GPT 5.5 OpenAI 44.2% $1.23 20
5 Claude Opus 4.7 Anthropic 38.0% $3.55 84
6 GPT 5.4 OpenAI 21.9% $0.49 15
7 Gemini 3 Flash Google 20.2% $0.25 73
8 Gemini 3.1 Pro Google 18.8% $1.46 114
9 Claude Haiku 4.5 Anthropic 13.1% $1.18 147
10 GPT 5 mini OpenAI 6.9% $0.03 17
11 GPT 5.4 mini OpenAI 6.8% $0.07 12
12 Claude Sonnet 4.6 Anthropic 0.4% $37.01 660
13 Claude Opus 4.6 Anthropic 0.0% $3.83 303