ProgramBench task detail

ivanceras__svgbob.6d00ad9

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
472generated behavioral tests
47.7%official best score
40.3%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 40.3% ok 190/472 $3.47 72 0.19h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 34.7% ok 164/472 $4.26 62 0.21h not exported
3 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 25.4% ok 120/472 $4.53 71 0.20h manifest · eval json · eval summary · agent summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

40.3% from 190/472 ProgramBench-scored tests. Raw public eval statuses: failure: 282, passed: 192.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_inline_string_with_newline_escape (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_output_to_file_error (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_invalid_stroke_width_error (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_invalid_font_size_error (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_invalid_scale_error (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_build_converts_bob_files (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260518T232006Z

25.4% from 120/472 ProgramBench-scored tests. Raw public eval statuses: failure: 353, passed: 121.

Agent trace: 90 shell command(s), 55 executable-observation command(s), 3 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_inline_string_with_newline_escape (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_output_to_file_error (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_invalid_font_size_error (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_build_default_output_same_dir (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_build_only_converts_bob_extension (failure)
  • 1b6f0d2f3f1f eval.tests.test_svgbob.test_build_creates_output_dir (failure)

manifest · eval json · eval summary · agent summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 47.7% $5.84 56
2 GPT 5.5 (high) OpenAI 46.0% $3.58 40
3 Claude Sonnet 4.6 Anthropic 41.3% $43.75 744
4 GPT 5.5 OpenAI 35.2% $1.00 17
5 Claude Opus 4.7 (xhigh) Anthropic 31.6% $4.11 95
6 Gemini 3.1 Pro Google 24.8% $0.97 38
7 Claude Opus 4.7 Anthropic 23.1% $0.75 36
8 Claude Opus 4.6 Anthropic 23.1% $14.16 268
9 Gemini 3 Flash Google 21.8% $0.27 52
10 GPT 5.4 OpenAI 18.6% $0.15 6
11 Claude Haiku 4.5 Anthropic 18.0% $1.28 187
12 GPT 5.4 mini OpenAI 13.3% $0.09 23
13 GPT 5 mini OpenAI 2.3% $0.03 16