Four models run five real-world task families on QoderCLI under the same prompt bytes, the same pass/fail gates, and the same rubric. The prompts, the artifacts, the evaluation harnesses, and the reports are all published here.
This is the first round of the QoderCLI benchmark series, with Qwen3.8-Max as the subject under test. It is not a leaderboard. It is a side-by-side comparison built to be re-checked: every figure below traces back to a file in this repository.
| Model | Aliases used in paths and filenames |
|---|---|
| Qwen3.8-Max (subject) | qwen, qwen38, qwen3.8max |
| Claude Opus 4.8 | opus48, opus4.8 |
| Claude Fable 5 | fable, fable5 |
| GPT-5.6 sol | sol, sol56 |
All four ran on QoderCLI; the version recorded in track 01 is 1.1.12.
| # | Track | Task | Round winner | Qwen3.8-Max |
|---|---|---|---|---|
| 1 | Deep Research | End-to-end analysis of the OpenAlex global AI-paper dataset, with verifiable deliverables | Qwen3.8-Max (100 pts) | 🥇 1/4 |
| 2 | F2E | 18 front-end visual tasks (plus 9 L4/L5 hard cases) | GPT-5.6 sol (69.1% median %-of-max over the shared 6 cases) | Tied with both Claude models (53.8%) |
| 3 | Along the River During Qingming | From an empty directory, hand-write a true-3D scroll that runs offline | Fable 5 (visual quality); Opus 4.8 (combined, including wall-clock) | 🥉 3/4 |
| 4 | Database Development | Fixed language (Rust) and fixed protocol; build an in-memory KV store from scratch | Three different winners across six metrics | Swept all three throughput metrics |
| 5 | The Lord of the Rings | A 92-second single-file Three.js narrative animation | Fable 5 (90 pts) | 🥈 2/4 (89 pts) |
| Track | Primary report | Also available |
|---|---|---|
| 01-deepresearch | REPORT.md |
prompts/TASK.md, the shared task brief; resources/, four interactive web artifacts; a side-by-side comparison deck |
| 02-f2e | README.md, all four models case by case |
prompts/, all 27 prompts verbatim; resources/, side-by-side screenshots, four-viewport grids, and screen recordings |
| 03-qingmingshanghetu | Per-model reports: qwen, fable, opus48 — sol56 shipped an artifact but no report |
input/, the shared prompt and gate pack; output/, four artifacts plus flythrough videos |
| 04-database | EVAL-GUIDE.md |
arena/, the task and four compilable submissions; harness/, the load and conformance engines; Dockerfile |
| 05-lotr | evaluation/REPORT.md |
input/PROMPT.md; output/, four artifacts; evaluation/, same-timestamp screenshots, score charts, exported video |
01-deepresearch/ prompts/TASK.md · resources/ (4 interactive HTML artifacts) · report · comparison deck
02-f2e/ prompts/ (27 cases) · resources/ (screenshots, viewport grids, clips/) · README.md
03-qingmingshanghetu/ input/ (prompt + gate pack) · output/ (4 artifacts, per-model reports, flythrough videos)
04-database/ arena/ · harness/ · run-eval.sh · Dockerfile · EVAL-GUIDE.md
05-lotr/ input/PROMPT.md · output/ (4 artifacts) · evaluation/ (REPORT.md, screenshots, videos)
Ordered from cheapest to most expensive.
04-database — fully reproducible with one command.
cd 04-database
bash run-eval.sh # 4 models × 6 metric groups
bash run-eval.sh --rounds 3 # median of 3 rounds (more stable)The other four tracks — complete prompts and all artifacts are published; re-running requires your own model access. Each prompt is the exact byte sequence sent to the model and can be reused directly: 01-deepresearch/prompts/TASK.md, 02-f2e/prompts/, 03-qingmingshanghetu/input/prompt.txt, 05-lotr/input/PROMPT.md.
Note: the frozen dataset that 01-deepresearch depends on,
openalex-ai-landscape/(60 country-year cells, 1,198 papers, 290 PDF pages across 17 CC-BY papers), is not distributed with this repository. You will need to obtain it separately to re-run that track in full.
These numbers describe one specific run under one specific configuration. They are not a general ranking of model capability.
- The scores are presentation metrics. We chose the dimension weights ourselves, and a different weighting produces a different ordering. The raw artifacts are all in this repository; recompute them with your own rubric.
- Only order-of-magnitude gaps support strong claims. In 04-database the top three finish within 0.5–9% of each other, which is smaller than the cross-round variance and not statistically significant. What can be asserted strongly are structural results such as the batching-gain tiers and the fact that throughput rank ≠ latency rank.
- Per-track sample sizes are small. F2E has 18 cases; Qingming and LOTR have one each. Conclusions from single-case tasks are case studies, not statistics.
- Wall-clock time and token usage do not count toward capability scores. They are reported as resource cost only, and they reflect this session — routing, concurrency, and cache hits all affect them.
- Cross-machine comparison requires re-running all four baselines on the same machine. The 04-database reference data was collected under WSL2 and its absolute values are conservative.
Each track documents its own limitations in its report — see in particular section 7 of 01-deepresearch/REPORT.md and the scope-of-claims section of 04-database/README.md.