DetectionBench

DetectionBench exists to make benchmark results on real-world and underrepresented object detection datasets as reproducible, comparable, and trustworthy as benchmarks on COCO have become.

Why DetectionBench?

Modern object detection research is overwhelmingly evaluated on a small number of canonical datasets such as COCO. In practice, however, computer vision systems are deployed in domains such as aerial robotics, maritime search and rescue, agriculture, underwater inspection, autonomous driving, document understanding, and many others where datasets are smaller, more specialized, and benchmark results are often difficult to compare.

DetectionBench provides a standardized benchmarking framework for these real-world and underrepresented datasets. By using common dataset adapters, reproducible training recipes, identical evaluation protocols, and unified hardware profiling, DetectionBench enables fair comparison of modern object detectors across diverse application domains.

DetectionBench is a Hydra-driven framework for preparing datasets, training models, evaluating performance, and benchmarking modern object detectors under standardized experimental conditions.

Supported Models

DetectionBench wraps two model families behind one CLI, trained and evaluated with identical recipes (same augmentation, early stopping, and metrics) regardless of family:

Family	Backend	Example checkpoints	Entrypoints
YOLO	Ultralytics	yolov8n/s/m, yolov9c/e, yolo11n/s/m, ...	detectionbench-train, detectionbench-evaluate, detectionbench-infer
RT-DETR	Ultralytics	rtdetr-l, rtdetr-x	detectionbench-train, detectionbench-evaluate, detectionbench-infer
RF-DETR	Roboflow rfdetr	rfdetr-nano, rfdetr-small, rfdetr-medium, rfdetr-large	detectionbench-train, detectionbench-evaluate, detectionbench-infer

Any Ultralytics-registered YOLO or RT-DETR checkpoint name works out of the box — the YOLO family isn't a fixed enum, YOLOTrainer just passes the name straight through to Ultralytics. Every model, regardless of family, gets hardware profiling (latency, FPS, VRAM, parameters, FLOPs) via detectionbench-benchmark and per-class evaluation metrics via detectionbench-evaluate.

detectionbench-train and detectionbench-evaluate are each one command for every model family: both inspect model.name=... / --model and dispatch to the Ultralytics or RF-DETR implementation automatically, so adding a new checkpoint name (or a new RF-DETR size) never needs a new entrypoint.

Supported Datasets

Dataset	Primary Task	Domain	Classes	Images	License	Source
DocLayNet	Object Detection / Layout Analysis	Document	11	80,863	CDLA-Permissive-1.0	Hugging Face
ExDark	Object Detection	Low-light Robustness	12	7,344	BSD-3-Clause 1	Hugging Face
GWHD 2021	Object Detection	Agriculture (Wheat Heads)	1	6,515	CC BY 4.0	Hugging Face
SeaDronesSee	Object Detection / Tracking	Maritime UAV / Search & Rescue	5	10,477 2	CC0-1.0	Dataset Card
Brackish Underwater	Object Detection	Marine Animal Detection	6	14,674	CC BY 4.0	Hugging Face
LISA Traffic Lights	Object Detection	Autonomous Driving	7	43,017	CC BY-NC-SA 4.0	Hugging Face
VisDrone-DET	Object Detection	Aerial / UAV Surveillance	11	8,629 3	CC BY-NC-SA 3.0	Hugging Face

Each dataset is a self-contained adapter under src/detectionbench/datasets/ that converts its raw format into a canonical COCO layout — everything downstream (COCO↔YOLO conversion, training, evaluation, inference, benchmarking) is dataset-agnostic. See src/detectionbench/datasets/doclaynet.py for a fully worked adapter.

Installation

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"          # core + dev tooling
pip install -e ".[rfdetr]"       # + RF-DETR training/eval
pip install -e ".[coco]"         # + pycocotools
pip install -e ".[benchmark]"    # + hardware-profiling extras (psutil, thop, fvcore, torchinfo, nvidia-ml-py)

Usage

# 1. Convert a raw dataset download into the canonical COCO layout
detectionbench-prepare-coco --dataset doclaynet --raw-dir /path/to/DocLayNet_core --output-dir /path/to/doclaynet_coco

# 2. Bridge into Ultralytics YOLO format (for YOLO/RT-DETR training)
detectionbench-convert-coco-to-yolo --input-dir /path/to/doclaynet_coco --output-dir /path/to/doclaynet_yolo

# 3. Train + evaluate (Hydra config group `dataset=<key>` selects the dataset;
#    one entrypoint for every model family, dispatched by `model.name=`)
detectionbench-train dataset=doclaynet model.name=yolov8n
detectionbench-train dataset=doclaynet model.name=rfdetr-nano

# 4. Evaluate / infer / benchmark a checkpoint directly (--dataset-yaml is
#    optional -- it auto-resolves from configs/dataset/<key>.yaml if omitted;
#    the same command works unchanged for model=rfdetr-nano)
detectionbench-evaluate --checkpoint experiments/yolov8n/weights/best.pt --model yolov8n --dataset doclaynet
detectionbench-infer --checkpoint experiments/yolov8n/weights/best.pt --model yolov8n --dataset doclaynet --input /path/to/images
detectionbench-benchmark --model yolov8n --checkpoint experiments/yolov8n/weights/best.pt

License

Code is licensed under Apache-2.0 (see LICENSE). Each benchmarked dataset retains its own original license — see the corresponding adapter's docstring under src/detectionbench/datasets/, CITATION.cff, and (once published) its own dataset card under dataset_cards/.

Footnotes

1. BSD-3-Clause is the license text itself; the original authors separately request non-commercial use. See the dataset card for compliance details. ↩

2. No public test-set labels are available for this dataset; this count reflects train and validation splits only. ↩

3. Train + val + test-dev splits only (6,471 + 548 + 1,610); the official test-challenge split (1,580 images) has no public ground truth. ↩