ProgramBench task detail

ariga__atlas.6d81150

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
1318generated behavioral tests
72.5%official best score
66.8%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 66.8% ok 881/1318 $5.36 94 0.25h manifest · eval json · eval summary · agent summary · usage audit
2 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 65.7% ok 866/1318 $6.61 116 0.35h manifest · eval json · eval summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 9.2% ok 121/1318 $4.47 71 0.23h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260518T232006Z

66.8% from 881/1318 ProgramBench-scored tests. Raw public eval statuses: failure: 569, passed: 1160, skipped: 3.

Agent trace: 195 shell command(s), 125 executable-observation command(s), 10 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • 5e483d7465e7 eval.tests.test_basic_invocation.test_unknown_flag (failure)
  • 5e483d7465e7 eval.tests.test_completion.test_completion_bash_no_descriptions (failure)
  • 5e483d7465e7 eval.tests.test_completion.test_completion_bash (failure)
  • 5e483d7465e7 eval.tests.test_completion.test_completion_powershell (failure)
  • 5e483d7465e7 eval.tests.test_completion.test_completion_fish (failure)
  • 5e483d7465e7 eval.tests.test_error_conditions.test_completion_invalid_shell (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260517T094700Z

65.7% from 866/1318 ProgramBench-scored tests. Raw public eval statuses: failure: 570, passed: 1159, skipped: 3.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • 5e483d7465e7 eval.tests.test_completion.test_completion_fish (failure)
  • 5e483d7465e7 eval.tests.test_completion.test_completion_powershell (failure)
  • 5e483d7465e7 eval.tests.test_error_conditions.test_schema_inspect_with_bad_url (failure)
  • 5e483d7465e7 eval.tests.test_migrate_commands.test_migrate_status_missing_url (failure)
  • 5e483d7465e7 eval.tests.test_error_conditions.test_invalid_url_format (failure)
  • 040ee76f34fd tests.test_completion.test_completion_powershell (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 72.5% $9.62 80
2 Claude Opus 4.7 (xhigh) Anthropic 55.4% $11.04 163
3 Claude Opus 4.7 Anthropic 54.8% $8.11 173
4 GPT 5.5 (high) OpenAI 54.6% $3.23 28
5 GPT 5.5 OpenAI 45.7% $1.58 31
6 Claude Haiku 4.5 Anthropic 28.4% $0.61 127
7 GPT 5.4 OpenAI 26.3% $0.43 16
8 Gemini 3 Flash Google 22.5% $0.27 104
9 Gemini 3.1 Pro Google 21.6% $1.62 114
10 GPT 5 mini OpenAI 20.4% $0.02 15
11 Claude Sonnet 4.6 Anthropic 9.3% $35.07 626
12 Claude Opus 4.6 Anthropic 0.0% $5.00 296
13 GPT 5.4 mini OpenAI 0.0% $0.06 18