Issues

• Invitation to review PaperBench in the AI Auto-Research survey collection

  openai/frontier-evals#183 · Filter by author ldkong1205 opened on Oct 15, 2026

• JudgeEval drops leaves with valid_score=False from y_true/y_pred, so judge accuracy/F1 is computed only over leaves the judge managed to grade

  openai/frontier-evals#178 · Filter by author shaurya416 opened on Sep 26, 2026

• PaperBench: resources_provided=True scores a subtree whose leaves are all "Dataset and Model Acquisition" as 0 at full weight, so a perfect submission scores below 1.0

  openai/frontier-evals#177 · Filter by author shaurya416 opened on Sep 24, 2026
  1

• SWE-Lancer grades all-missing pytest exit codes as a model failure instead of rollout error

  openai/frontier-evals#151 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• PaperBench metrics can mix papers from different evaluation runs into one seed

  openai/frontier-evals#149 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• PaperBench token aggregation crashes on explicit null usage metadata

  openai/frontier-evals#146 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• PaperBench aggregate nodes can hide invalid child grades

  openai/frontier-evals#144 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• Cached PaperBench reproduction metadata cannot load older optional schemas

  openai/frontier-evals#142 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• PaperBench run parsing crashes on uneven seed counts

  openai/frontier-evals#140 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• SWE-Lancer token totals include one phantom token per category

  openai/frontier-evals#139 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• System-error metric validity ignores missing rollout attempts

  openai/frontier-evals#137 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• Metadata stripping mutates the original computer-task conversation

  openai/frontier-evals#134 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• End-of-run computer-task logging treats partial scores as correct

  openai/frontier-evals#133 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• Computer-task action limit rejects the final allowed action

  openai/frontier-evals#131 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• ShortAnswerEval records every sampled answer as incorrect

  openai/frontier-evals#127 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• MCQ extraction scores the first answer after model self-correction

  openai/frontier-evals#125 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• Multi-answer MCQ parse failures return an int and violate the evaluator contract

  openai/frontier-evals#124 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• MCQ API prompt suppresses multi-answer responses and leaves answer_choices unexpanded

  openai/frontier-evals#123 · Filter by author sylvesterkaczmarek opened on Aug 16, 2026

• Tempo audit task metadata is inconsistent with config.yaml and finding files

  openai/frontier-evals#122 · Filter by author Salieri-128 opened on Aug 6, 2026

• No SecurityPolicy

  openai/frontier-evals#113 · Filter by author infycore opened on May 10, 2026

• BasicAgent: missing elif in _construct_periodic_reminder makes use_real_time_limit a no-op for the agent's progress reminder

  openai/frontier-evals#112 · Filter by author jzheng17 opened on Apr 26, 2026

• Broken relative links in audit finding "Code Location" sections

  openai/frontier-evals#103 · Filter by author josepchetrit12 opened on Feb 27, 2026
  2

• Overly-strict test in swelancer sample 16912_4

  openai/frontier-evals#100 · Filter by author MattFisher opened on Dec 31, 2025

• swelancer user_tool files call non-existent playwright function

  openai/frontier-evals#99 · Filter by author MattFisher opened on Dec 31, 2025
  1

• Release PaperBench on Hugging Face

  paperbench
  openai/frontier-evals#3 · Filter by author NielsRogge opened on Apr 4, 2025
  1