ProgramBench task detail

google__brotli.b3dc9cc

Task-level results for this Codex /goal scaffold. ProgramBench baseline context is cached from the official task page; Codex scored tests are after active-branch and ignored-test filtering.

Back to leaderboard Official ProgramBench task
441generated behavioral tests
95.5%official best score
24.9%Codex best score
3Codex result rows

Codex Results by Model

#ModelRunAgentModeScoreEvalTestsEst. costCallsWallEvidence
1 GPT 5.5 (xhigh) 20260517T094700Z Codex /goal Mini-SWE-compatible no internet 24.9% ok 110/441 $4.01 64 0.20h manifest · eval json · eval summary · usage audit
2 GPT 5.5 (xhigh) 20260519T215046Z Codex /goal Paper prompt + Goal contract no internet 19.5% ok 86/441 n/a 96 0.28h not exported
3 GPT 5.5 (xhigh) 20260518T232006Z Codex /goal Paper prompt + /goal no internet 1.4% ok 6/441 n/a 72 0.24h manifest · eval json · eval summary · agent summary · usage audit

Why Scores Differ

Official ProgramBench rows are the public mini-SWE-agent submissions. GoalBench rows are separate Codex /goal submissions. Same model label does not mean the same agent, prompt, tool loop, or generated implementation. A GoalBench row is resolved only when the ProgramBench-scored pass rate is exactly 100%.

GPT 5.5 (xhigh) · 20260517T094700Z

24.9% from 110/441 ProgramBench-scored tests. Raw public eval statuses: failure: 458, passed: 148.

Agent trace summary unavailable.

Likely miss class: behavioral mismatch.

  • f8d762f9e408 eval.tests.test_brotli.TestBasicInvocation.test_help_mentions_coalesced_options (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestFileDecompression.test_decompress_precompressed_alice (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestFileCompression.test_compress_large_text_file_smaller (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestFileDecompression.test_decompress_precompressed_zeros (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestFileCompression.test_compress_file_with_keep_flag (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestFileCompression.test_compress_file_creates_valid_br (failure)

manifest · eval json · eval summary · usage audit

GPT 5.5 (xhigh) · 20260519T215046Z

Public eval evidence has not been exported for this row yet.

GPT 5.5 (xhigh) · 20260518T232006Z

1.4% from 6/441 ProgramBench-scored tests. Raw public eval statuses: failure: 597, passed: 9.

Agent trace: 108 shell command(s), 71 executable-observation command(s), 5 build command(s), 1 blocked attempt(s). Raw Codex logs are not published.

Likely miss class: behavioral mismatch, exact version/output mismatch.

  • f8d762f9e408 eval.tests.test_brotli.TestFileDecompression.test_decompress_suffix_mismatch_error (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestBasicInvocation.test_invalid_option_long_format_help_to_stdout (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestFileDecompression.test_decompress_precompressed_alice (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestFileCompression.test_compress_large_text_file_smaller (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestBasicInvocation.test_help_short_goes_to_stderr (failure)
  • f8d762f9e408 eval.tests.test_brotli.TestBasicInvocation.test_help_mentions_coalesced_options (failure)

manifest · eval json · eval summary · agent summary · usage audit

Official ProgramBench Results by Model

#ModelProviderScoreCostCalls
1 GPT 5.5 (xhigh) OpenAI 95.5% $14.46 116
2 Claude Sonnet 4.6 Anthropic 90.7% $19.93 394
3 Claude Opus 4.7 Anthropic 84.6% $5.68 110
4 GPT 5.5 (high) OpenAI 82.1% $4.86 59
5 Claude Opus 4.6 Anthropic 80.3% $10.59 262
6 Gemini 3.1 Pro Google 52.8% $1.03 69
7 Gemini 3 Flash Google 46.7% $0.34 100
8 GPT 5 mini OpenAI 1.6% $0.02 14
9 GPT 5.5 OpenAI 1.1% $1.13 22
10 GPT 5.4 OpenAI 0.9% $0.52 18
11 Claude Haiku 4.5 Anthropic 0.9% $0.63 122
12 GPT 5.4 mini OpenAI 0.7% $0.05 11
13 Claude Opus 4.7 (xhigh) Anthropic 0.5% $6.32 94