[Product Gap] Add real-audio MIR accuracy acceptance benchmarks

[seonghobae]

Buyer-visible gap

BandScope has strong unit, contract, parity, and security gates, but the protected merge path does not yet prove that an actual audio file traversing the production intake and analysis pipeline yields the expected musical result. A buyer cannot distinguish “the code executes” from “the rehearsal guidance is measurably accurate.”

This issue establishes a reproducible, rights-safe real-audio acceptance layer for harmony, tempo/beat, structure, source separation, role range, and rehearsal cue accuracy. The test must exercise decoded PCM audio through the same public analysis boundary used by the desktop product; mocked feature matrices alone are not acceptance evidence.

Product outcome

Every release candidate publishes an accuracy manifest answering:

• what audio was analyzed;
• what ground truth was expected;
• which exact engine/backend/version produced the result;
• which recognized MIR metric was used;
• whether accuracy improved, regressed, or remained within the registered tolerance;
• whether CPU and GPU paths agree where both exist; and
• which known limitations require human confirmation in the rehearsal UI.

Test tiers

Tier 1 — deterministic redistributable PCM fixtures

Generate or check in tiny, license-clean WAV/FLAC fixtures with immutable SHA-256 manifests. These must be real decoded waveforms, not direct chroma/onset arrays.

Required cases:

• sustained single notes across instrument/vocal register boundaries;
• major, minor, seventh, no-chord, inversion, and enharmonic chord segments;
• a known chord progression with exact change times;
• stable tempo, half/double-tempo ambiguity, swing, pickup, stop, and tempo-change cases;
• verse/chorus/bridge-style structural boundaries and repeated sections;
• bass, vocal, drums, and harmonic stems mixed at known gains;
• simultaneous roles whose ranges overlap and then separate;
• silence, clipping, low level, short input, sample-rate conversion, mono/stereo, and bounded background noise.

The fixture generator, waveform parameters, annotations, checksums, and expected metric ranges must be versioned. Generation should be deterministic and independent of network access.

Tier 2 — redistributable public corpus slice

Use only audio whose redistribution and automated evaluation rights are documented. Record the license, source URL/DOI, exact file hash, annotation provenance, split, and any transformation. Do not commit copyrighted commercial recordings or annotations that require unavailable audio.

Tier 3 — private commercial-readiness benchmark

Run a scheduled/manual benchmark against a separately licensed private corpus. Store only aggregate metrics, bounded error exemplars, configuration hashes, and provenance-safe artifacts in GitHub. The workflow must fail closed when the corpus credential or manifest is absent and must never substitute synthetic evidence while claiming private-corpus success.

Metrics and acceptance contracts

Harmony

• Report segment-based chord symbol recall and corpus-duration-weighted WCSR.
• Report at least root, major/minor, and seventh-vocabulary mappings rather than one opaque score.
• Preserve no-chord handling and exact annotation/estimate time intervals.
• Add boundary error summaries so a temporally shifted but harmonically correct output is visible.

Beat and tempo

• Report beat precision, recall, and F-measure using a documented tolerance, plus continuity-aware beat metrics when available.
• Report tempo Acc1 and Acc2 separately so octave-related errors are not hidden.
• Include explicit half-time/double-time fixtures and confidence calibration.

Structure

• Report boundary precision/recall/F-measure at documented strict and relaxed windows.
• Report a segment-label agreement metric for repeated verse/chorus/bridge regions.
• Verify section ordering, repeated-section identity, and pickup/tag preservation.

Source separation

• Report BSSEval v4-style SDR or the repository’s pinned equivalent per stem, but do not treat one energy-ratio metric as complete perceptual evidence.
• Record stem-specific artifacts and add a bounded human-listening validation protocol before making perceptual-quality claims.
• Verify mixture consistency and absence of NaN/Inf outputs.

Rehearsal-specific outputs

• Range endpoints: note/semitone error and exact out-of-range classification.
• Entry/dropout/stop/pickup cues: event precision, recall, and timing error.
• Role overlap: interval IoU or equivalent overlap error against known role activity.
• Confidence: calibration curve/Brier-style score where a probabilistic output exists.

Regression policy

• Establish the first protected baseline without inventing unsupported “industry standard” thresholds.
• Thereafter fail the protected gate on a statistically or practically meaningful regression defined per metric and fixture family.
• Publish bootstrap or repeated-run uncertainty for nondeterministic stages.
• Keep dataset-level and track-level metrics so aggregate improvements cannot hide severe song-specific failures.
• Require an explicit reviewed waiver with expiry for any accepted regression; no silent threshold reduction.

CPU/GPU and implementation boundary

• Production numeric kernels remain Rust-owned where the corresponding production path is Rust-backed.
• Run CPU reference and available GPU backend on the same exact fixtures and compare outputs/metrics within registered numerical tolerances.
• Do not skip a configured GPU acceptance lane and report it as passing.
• Minimize thread oversubscription and record backend, thread count, device, precision, peak RSS/VRAM, and elapsed time in the manifest.

Security and rights notes

• Treat audio, annotations, metadata, decoders, model artifacts, and benchmark manifests as untrusted input.
• Enforce bounded duration, channels, sample rate, decoded size, file count, and output artifact size.
• Use no shell interpolation or generic execution capability.
• Never publish private audio, absolute local paths, credentials, or copyrighted excerpts in logs/artifacts.
• Keep local-first runtime behavior; the acceptance workflow may use separately authorized benchmark storage but must not add a network dependency to ordinary analysis.

Required repository changes

• Add an accuracy manifest schema and deterministic fixture manifest.
• Add a single documented command for the complete acceptance run.
• Add permanent tests proving actual file decode and end-to-end analysis, metric correctness, malformed-manifest failure, checksum mismatch failure, and report determinism.
• Add production statement and branch coverage 100% plus public docstrings 100% for new repository-owned code.
• Add docs/doctoring/real-audio-accuracy-acceptance.md with metric definitions, claim boundaries, licenses, operational interpretation, rollback policy, and APA 7th references.
• Update AGENTS.md, ARCHITECTURE.md, CHANGELOG.md, and release acceptance documentation.
• Export machine-readable JSON and accessible HTML exact-value reports.

Merge and release gate

Do not claim release readiness from unit tests alone after this acceptance layer exists. Merge the implementation only after exact-head repository CI, central coverage, security/supply-chain checks, realistic audio acceptance, CPU/GPU parity where configured, current-head automated review, zero unresolved threads, and qualifying independent approval all succeed without bypass.

References — APA 7th

Odekerken, D., Koops, H. V., & Volk, A. (2021). Improving audio chord estimation by alignment and integration of crowd-sourced symbolic music. Transactions of the International Society for Music Information Retrieval, 4(1), 141–155. https://doi.org/10.5334/tismir.81

Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., & Ellis, D. P. W. (2014). MIR_EVAL: A transparent implementation of common MIR metrics. In Proceedings of the 15th International Society for Music Information Retrieval Conference (pp. 367–372).

Schreiber, H., & Müller, M. (2020). Music tempo estimation: Are we done yet? Transactions of the International Society for Music Information Retrieval, 3(1), 111–125. https://doi.org/10.5334/tismir.43

Stöter, F.-R., Liutkus, A., & Ito, N. (2018). The 2018 signal separation evaluation campaign. In Latent Variable Analysis and Signal Separation. https://doi.org/10.1007/978-3-319-93764-9_35

Claim boundary

Passing this issue’s benchmark will support specific, versioned accuracy claims on the registered fixtures and corpora. It will not establish universal musical correctness, genre/culture invariance, or perceptual superiority without separate representative data and human validation.

[seonghobae]

Progress update (2026-08-10): PR #828 now publishes exact head 8e34c0e563fafa08909307109b358cbbebf5537f (tree a4e77ef4998ac7b30fea38aa02edb8a74b5c3489) for one bounded known-vocal slice of this issue.

Delivered and reviewable:

• production YouTube intake → pinned creator-master identity → one composed global alignment → deterministic real htdemucs → zero-mean vocal SI-SDR improvement and semantic assignment margin;
• fail-closed full filename/size/SHA-256 model verification before restricted weights-only same-byte loading, with no runtime model retrieval;
• full ffmpeg+ffprobe path/hash/execute/sibling identity, bounded download/decode/cleanup, deterministic offline metric/alignment/integrity/failure tests, and an explicit authorized live opt-in;
• canonical PRD (11 requirements), TRD (13 requirements), three ADRs, root/discoverable Architecture, sequence/state/deployment/class UML, a logical run/evidence model, traceability, acceptance, operations, release, Security Notes, and [Product Gap] Add real-audio MIR accuracy acceptance benchmarks #770 doctoring;
• a shared cross-platform Python launcher for root checks and quickcheck rather than assuming either python3 or python exists.

Documentation fitness is now design-sufficient for this bounded known-vocal slice. A physical database ERD remains intentionally not applicable because fixtures, aligned windows, and separated stems are ephemeral and no benchmark database exists; the logical BenchmarkRun/evidence relationships are documented instead.

Exact-head publication evidence: all 9 hosted workflows completed successfully, CodeRabbit combined status is success, its exact-head command review reported no actionable issue, and all 29 review threads are resolved. Formal independent exact-head approval is still absent, so the PR is not merge-ready.

The only live production-path attempt remains historical negative evidence: archive/member/master/model integrity passed, then YouTube intake failed closed with HTTP 502 after 65.49 seconds before separation. It produced no identity correlation or SI-SDR score and is not a live pass. Creator-master-only calibration (+1.752 dB SI-SDRi, +7.631 dB assignment margin) is not YouTube validation.

This still does not close #770. Remaining work includes formal content/platform authorization; model rights/delivery and approved-pickle risk decisions; authorized exact-candidate YouTube calibration with drift ownership; accepted schema-v1 evidence persistence/access/TTL/deletion controls and an emitter; every advertised OS/architecture pass or verified fallback; recoverable failure UX; multi-fixture/four-stem coverage; the other MIR families; machine-readable/accessibility reports; and qualifying independent review.