ProgramBench task detail

drew-alleman__datasurgeon.d257cee

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
502generated behavioral tests
80.7%official best score
72.9%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 72.9% ok 366/502 $4.61 79 0.21h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 36.3% ok 182/502 $4.01 71 0.20h manifest · eval json · eval summary · agent summary · usage audit
3 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 30.9% ok 155/502 $2.89 56 0.17h not exported

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

72.9% from 366/502 ProgramBench-scored tests. Raw public eval statuses: failure: 154, passed: 410.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • f5ab65c66ad6 eval.tests.test_extraction_features.test_aws_long_flag (failure)
  • 61d5d5110768 tests.test_extraction_basic.test_no_extraction_flags_shows_first_match_all_types (failure)
  • 61d5d5110768 tests.test_extraction_extended.test_google_keys_from_file (failure)
  • 61d5d5110768 tests.test_extraction_extended.test_phone_numbers_from_file (failure)
  • 61d5d5110768 tests.test_extraction_basic.test_email_in_log_context_suppress_stdin_msg (failure)
  • 61d5d5110768 tests.test_extraction_basic.test_email_tld_length_restrictions (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

36.3% from 182/502 ProgramBench-scored tests. Raw public eval statuses: failure: 338, passed: 226.

Agent trace: 185 shell command(s), 159 executable-observation command(s), 10 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, interactive terminal behavior mismatch.

  • f5ab65c66ad6 eval.tests.test_additional_coverage.test_aws_key_formats (failure)
  • f5ab65c66ad6 eval.tests.test_additional_coverage.test_credit_card_various_formats (failure)
  • f5ab65c66ad6 eval.tests.test_edge_cases.test_very_long_line (failure)
  • f5ab65c66ad6 eval.tests.test_additional_coverage.test_phone_various_formats (failure)
  • f5ab65c66ad6 eval.tests.test_additional_coverage.test_mac_various_formats (failure)
  • f5ab65c66ad6 eval.tests.test_additional_coverage.test_drop_filter_on_same_content (failure)

manifest · eval json · eval summary · agent summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 80.7% $6.08 69
2 GPT 5.5 (high) OpenAI 77.1% $2.24 31
3 Claude Opus 4.7 (xhigh) Anthropic 76.1% $17.05 263
4 Claude Opus 4.6 Anthropic 74.3% $13.45 345
5 Gemini 3 Flash Google 73.3% $0.30 112
6 Claude Opus 4.7 Anthropic 72.1% $8.64 213
7 GPT 5.5 OpenAI 70.1% $0.91 14
8 Claude Sonnet 4.6 Anthropic 69.1% $13.76 462
9 Gemini 3.1 Pro Google 44.8% $1.08 71
10 GPT 5.4 OpenAI 39.8% $0.20 9
11 Claude Haiku 4.5 Anthropic 35.7% $0.67 88
12 GPT 5 mini OpenAI 27.5% $0.03 21
13 GPT 5.4 mini OpenAI 3.0% $0.05 12