ProgramBench task detail

noborus__trdsql.d8c5ff6

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
1312generated behavioral tests
76.4%official best score
70.9%Codex best score
2Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 70.9% ok 930/1312 $2.96 55 0.16h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 70.7% ok 928/1312 n/a 54 0.18h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

70.9% from 930/1312 ProgramBench-scored tests. Raw public eval statuses: failure: 393, passed: 1009, skipped: 1.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • d560cdefe188 eval.tests.test_basic_invocation.test_debug_flag (failure)
  • d560cdefe188 eval.tests.test_compression.test_read_gzip_file (failure)
  • d560cdefe188 eval.tests.test_compression.test_query_testdata_bz2 (failure)
  • d560cdefe188 eval.tests.test_configuration.test_config_with_debug (failure)
  • d560cdefe188 eval.tests.test_compression.test_query_testdata_xz (failure)
  • d560cdefe188 eval.tests.test_compression.test_read_bz2_file (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 76.4% $7.07 70
2 GPT 5.5 (high) OpenAI 71.0% $2.11 22
3 Claude Opus 4.6 Anthropic 66.8% $9.89 307
4 Claude Opus 4.7 Anthropic 61.7% $3.13 73
5 GPT 5.5 OpenAI 54.0% $0.74 13
6 GPT 5.4 OpenAI 49.1% $0.38 14
7 Gemini 3.1 Pro Google 46.4% $1.10 96
8 Claude Haiku 4.5 Anthropic 25.2% $1.70 207
9 Gemini 3 Flash Google 14.9% $0.26 109
10 Claude Opus 4.7 (xhigh) Anthropic 3.4% $8.71 175
11 GPT 5.4 mini OpenAI 3.4% $0.03 12
12 Claude Sonnet 4.6 Anthropic 3.0% $12.60 413
13 GPT 5 mini OpenAI 2.9% $0.01 14