Copyright 2026 The Johns Hopkins University Applied Physics Laboratory LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

SFBench: The SciFy Scientific Feasibility Benchmark

This dataset was created for the research described in this paper:

Cash Costello, James Mayfield, Elsbeth Turcan, Christine Piatko, Christina K. Pikas, Justin Rokisky, Sam Scheck, Chris Ribaudo, Ritwik Bose and Alex Memory, 'SFBench: The SciFy Scientific Feasibility Benchmark' (2026). arXiv preprint 2606.29630

In addition to the collection of claims and associated gold standard labels referenced in the paper, this project contains tools to validate assessment outputs and calculate metrics.

Install

This project relies on the uv package manager.

Install uv and run: uv sync

Validation

validate.py contains pydantic definitions for Gold Standards, Problems, and Assessments and can be run to validate jsonl collections of those data types.

usage: validate.py [-h] -i INPUT

Validate a JSONL file.

options:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        The JSONL file to validate (default: None)

Metrics

metrics.py generates metrics for a jsonl collection of assessments given a jsonl collection of gold standards.

usage: metrics.py [-h] -i INPUT -g GOLD [-o OUTPUT] [--verbose]

Compute agreement metrics between a submission and gold standard.

options:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Submission JSONL file
  -g GOLD, --gold GOLD  Gold standard JSONL file
  -o OUTPUT, --output OUTPUT
                        Output JSON file path
  --verbose             Verbose output

Example output:

{
  "team": "team_1",
  "run_id": "run_id_1",
  "krippendorff_alpha": 0.6347173280220622,
  "quadratic_cohen_kappa": 0.6346275349026801,
  "pearson": 0.6381623644648151,
  "mean_absolute_error": 0.8730964467005076,
  "accuracy": 0.467005076142132,
  "off_by_one_accuracy": 0.7918781725888325,
  "balanced_accuracy": 0.38818760024334936,
  "sign_accuracy": 0.700507614213198,
  "wallclock": {
    "min": 328.8922975063324,
    "mean": 2179.072237656756,
    "max": 6363.285372495651,
    "std_dev": 1495.9622531120697
  },
  "confidence": {
    "min": 0.05,
    "mean": 0.7349268292682928,
    "max": 0.98,
    "std_dev": 0.16515364468688268
  }
}

Citation

@article{sfbench,
      title={SFBench: The SciFy Scientific Feasibility Benchmark}, 
      author={Cash Costello and James Mayfield and Elsbeth Turcan and Christine Piatko and Christina K. Pikas and Justin Rokisky and Sam Scheck and Chris Ribaudo and Ritwik Bose and Alex Memory},
      year={2026},
      eprint={2606.29630},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2606.29630},
}

Acknowledgements

This material is based upon work supported by the DARPA Scientific Feasibility (SciFy) Program under Contract HR001122D0001. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Department of War or the U.S. Government.