FrontierPhysics

FrontierPhysics is an end-to-end physics research agent benchmark. Paper: https://arxiv.org/abs/2611.06910 (arXiv:2611.06910).

Active physicists (PhD students, postdocs, research scientists) author tasks from their own research: 56 tasks, 42 physicists, 26 institutions, across theoretical, computational, and experimental physics in nine primary domains. Given a PI-style high-level prompt, agents must review literature, plan, run research/experiments, and produce deterministically verifiable numerical outputs plus a scientific manuscript (e.g. paper.pdf).

Highlights:

• Combines deterministic verifiers (scripts check numerical outputs) with expert-authored rubrics (blockers / non-blockers);
• A pass requires the verifier and every blocker; non-blockers weight manuscript/research quality;
• About 22 models in 26 configurations; early results: no configuration passes more than 46 of 168 rollouts; Claude Opus 5.5 overall Pass@1 ≈ 27.4%;
• Manual trajectory review: main weaknesses are insufficiently rigorous reasoning and limited expert intuition, not missing physics facts or pure execution errors.

Problem

Prior physics benchmarks mostly use fixed QA, isolated sub-tasks, or paper reproduction with the original paper given. FrontierPhysics asks agents to do open-ended, multi-stage research without a reference solution, grading both correctness and publishable quality.

Tasks & scoring

• Tasks come from contributors’ own projects (typically ≥ ~two weeks of real work); some underlie Nature / PRL / PRX papers;
• Deliverables: deterministic artifacts + manuscript; global rubrics cover reward hacking, hallucination, literature synthesis; task rubrics are expert-written;
• Score is 0 if the verifier or any blocker fails; otherwise a weighted non-blocker score measures quality.

Limits

• v0.1 with 56 tasks—per-domain gaps remain uncertain;
• Agent-as-judge grading can err (authors report and corrected some false negatives);
• Failure-mode analysis focuses on a strong-model cohort; other models/harnesses may differ.

Who it's for

• Physics / AI4Science agent researchers;
• Evaluators who need dual signal on end-to-end research and write-up quality;
• Physicists willing to contribute real projects as tasks.

Getting started

1. Read https://arxiv.org/abs/2611.06910 and the task/eval protocol;
2. Follow BenchFlow and official task-release channels;
3. Reproduce rollouts in the documented harnesses (Codex CLI, Claude Code, Grok Build, Antigravity, etc.);
4. Track Pass@1, deterministic pass rate, and rubric scores, and improve agents on rigor, intuition, and literature synthesis.