Rank LLMs with a fraction of the benchmark items — same ranking, a lot less compute.
Running a full benchmark (thousands of items) for every model and every checkpoint is slow and expensive, and most of those items are uninformative — everyone passes them or everyone fails them, so they cost tokens and add no signal. adaptive-eval borrows Computerized Adaptive Testing (the method behind the GRE): model each item with a difficulty and a discrimination, then administer only the items that carry the most Fisher information about this model's ability, and stop as soon as the ranking is pinned.
Small, readable, zero dependencies (pure Python). A drop-in library, not a framework.
python -m adaptive_eval.simitem bank: 300 items | models ranked: 40
adaptive used on average 13.9 items (4.7% of the bank)
ranking fidelity vs full bank (Spearman):
ADAPTIVE : 0.949
RANDOM (same budget): 0.843
On this synthetic bank, adaptive testing reproduces 95% of the full-bank ranking using under 5% of the items, and clearly beats random subsampling at the same budget.
Honest scope: this is a synthetic, well-specified demonstration (responses are generated from the same one-dimensional IRT model the estimator assumes). Real benchmarks are multi-skill and noisier, so expect to spend more items and to need good calibration. This proves the mechanism; it is not a claim about any specific real benchmark. See Limits.
git clone https://github.com/moses607/adaptive-eval && cd adaptive-eval
pip install -e .from adaptive_eval import calibrate, adaptive_test
# 1) Calibrate item params ONCE from models you already ran in full (0/1 matrix).
items = calibrate(response_matrix) # -> [(discrimination, difficulty), ...]
# 2) Evaluate a NEW model adaptively. answer_fn(i) runs item i, returns 1/0.
result = adaptive_test(items, answer_fn, se_threshold=0.33)
print(result.theta, result.se, result.n_items) # ability, error, items used- Item Response Theory (2PL):
p(correct) = 1 / (1 + e^-a(θ − b))— item difficultyb, discriminationa, model abilityθ. - Fisher information: an item tells you most about
θwhen its difficulty sits nearθ→I = a²·p·(1−p). - Adaptive loop: administer the max-information item, re-estimate
θ, stop when the standard error is small enough. The model never sees the whole bank.
The math lives in adaptive_eval/core.py (~120 lines) and adaptive_eval/adaptive.py. Read it in one sitting.
The idea is not new, and usable tools already exist. Be clear-eyed:
- Research: ATLAS — Adaptive Testing for LLM Evaluation (2025, Fisher-info selection, up to 90% fewer items — arXiv:2511.04689) and Confident Rankings with Fewer Items (2026 — arXiv:2601.13885).
- Tools: CAT4AI (a full adaptive-testing framework for AI models), adaptivetesting (a broad Bayesian CAT package), tinyBenchmarks (pre-selected static subsets + an IRT tool), plus general CAT libs (catsim, mirtCAT).
So why adaptive-eval? Minimalism. It's the smallest version — ~120 readable lines, zero dependencies, a two-function drop-in (calibrate, adaptive_test) you can read in one sitting and vendor straight into your own eval loop, selecting items adaptively per run (not a fixed pre-chosen subset). Want a full framework? Use CAT4AI. Want the thing you can read, trust, and paste in? Use this. Think "the nanoGPT of adaptive LLM eval" — not "the first."
- One-dimensional ability. Real benchmarks are multi-skill; a single
θblurs that. Multidimensional IRT is on the roadmap. - Needs calibration. Adaptive selection needs item parameters.
calibrate()is a solid JMLE baseline (recovers item difficulty at Spearman ≈ 0.99 on synthetic data) — supply your own for production. - Assumes stable items. If an item's behavior drifts, its calibration goes stale.
- The demo is idealized. See the scope note above.
Multidimensional (per-skill) ability · an adapter for lm-eval-harness / HELM result matrices · continuous-score items (not just right/wrong) · exposure control so you don't overuse the same items.
MIT © 2026 Cherry FRANCOIS. From the maker of Aether OS — whose kernel has an eval harness this extends.