ProgramBench task detail

codesnap-rs__codesnap.f81e4f3

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
571generated behavioral tests
82.7%official best score
78.5%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 78.5% ok 448/571 $4.88 55 0.22h not exported
2 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 76.9% ok 439/571 $3.32 36 0.16h manifest · eval json · eval summary · usage audit
3 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 76.4% ok 436/571 $5.10 59 0.18h manifest · eval json · eval summary · agent summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260517T094700Z

76.9% from 439/571 ProgramBench-scored tests. Raw public eval statuses: failure: 293, passed: 556, skipped: 4.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • ab87cc3978a9 eval.tests.test_edge_cases.test_malformed_json_highlight_lines (failure)
  • ab87cc3978a9 eval.tests.test_input_sources.test_from_code_stdin (failure)
  • ab87cc3978a9 eval.tests.test_output_destinations.test_output_raw_to_stdout (failure)
  • df37316a82f6 tests.test_configuration.test_silent_mode_with_first_run_shows_info (failure)
  • df37316a82f6 tests.test_config_gaps.test_solid_background_color_string (failure)
  • df37316a82f6 tests.test_configuration.test_custom_config_file_with_config_flag (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260518T232006Z

76.4% from 436/571 ProgramBench-scored tests. Raw public eval statuses: error: 1, failure: 296, passed: 552, skipped: 4.

Agent trace: 139 shell command(s), 106 executable-observation command(s), 4 build command(s), 2 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch.

  • ab87cc3978a9 eval.tests.test_additional_coverage.test_svg_output_with_options (failure)
  • ab87cc3978a9 eval.tests.test_additional_coverage.test_all_code_options_combined (failure)
  • ab87cc3978a9 eval.tests.test_additional_coverage.test_scale_factor_one (failure)
  • ab87cc3978a9 eval.tests.test_comprehensive_coverage.test_scale_factors (failure)
  • ab87cc3978a9 eval.tests.test_configuration.test_config_with_all_options (failure)
  • ab87cc3978a9 eval.tests.test_configuration.test_config_file_auto_created (failure)

manifest · eval json · eval summary · agent summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 82.7% $7.25 63
2 GPT 5.5 (high) OpenAI 79.0% $3.04 34
3 GPT 5.4 OpenAI 75.7% $0.79 11
4 Gemini 3.1 Pro Google 69.7% $1.40 83
5 GPT 5.4 mini OpenAI 62.5% $0.08 41
6 GPT 5.5 OpenAI 59.4% $0.69 15
7 Claude Haiku 4.5 Anthropic 51.0% $0.49 86
8 Claude Opus 4.7 Anthropic 47.1% $1.88 54
9 Gemini 3 Flash Google 29.2% $0.51 116
10 GPT 5 mini OpenAI 8.7% $0.01 9
11 Claude Opus 4.7 (xhigh) Anthropic 7.5% $4.86 120
12 Claude Opus 4.6 Anthropic 7.4% $8.54 172
13 Claude Sonnet 4.6 Anthropic 7.4% $20.92 395