P5a: jc reliability/validity metric battery (Pearson / Spearman / Cronbach α / ICC)#709
Merged
Merged
Conversation
…h α/ICC)
crates/jc was a FORMAL-SCAFFOLD pillar-prover with NO callable
reliability/validity metrics — only a private spearman_rho on rankings
in probe_p1, and a separately hand-rolled one in the stockfish chess
probe. The D-TRI-2 (12-family vs 12-step agreement), D-TRI-4
(chess<->thinking transfer) and D-TRI-5 (emulation vs resonance) gates
all name a battery that did not exist as an API.
New jc::reliability module:
- pearson(x, y): product-moment correlation.
- spearman(x, y): Pearson on average-rank (tie-corrected) transform.
- cronbach_alpha(items): internal-consistency reliability of a k-item scale.
- icc(ratings, IccForm::{Icc2_1, Icc3_1}): Shrout-Fleiss 1979 two-way ANOVA
ICC — absolute-agreement and consistency single-measure forms.
All return Option<f64> (None on degenerate input — no panics/NaN, per the
no-unwrap guideline). Point estimates ONLY; significance calibration stays
jc::jirak's job (I-NOISE-FLOOR-JIRAK: weak dependence, not IID Berry-Esseen).
Formula correctness pinned to reference values: Shrout-Fleiss worked example
ICC(2,1)=0.290 / ICC(3,1)=0.715 (matches R psych::ICC / pingouin), plus
Pearson/Spearman/Cronbach textbook checks. 12 unit tests + 3 doctests green;
clippy exit 0.
Board: EPIPHANIES E-JC-IS-NOT-A-METRIC-BATTERY-1.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Awg6TXocHcwTtc6eGsHcdD
📝 WalkthroughWalkthroughChangesReliability metrics
Shipped board notes
Estimated code review effort: 3 (Moderate) | ~30 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_c5aeccd3-fd89-44f5-9473-acd94ca9aa65) |
AdaWorldAPI
marked this pull request as ready for review
July 17, 2026 13:11
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/jc/src/reliability.rs`:
- Line 7: The documented pillar count is stale: update the “ten pillars”
reference in crates/jc/src/reliability.rs at lines 7-7 to reflect the 12
registered pillars or remove the hard-coded count, and correct the shipped
inventory in .claude/board/EPIPHANIES.md at lines 5-5 without reordering its
ledger.
- Around line 76-96: Update pearson, the rank-correlation API, and the
reliability APIs at crates/jc/src/reliability.rs lines 76-96, 102-149, 181-203,
and 239-287 to reject non-finite inputs before calculations and return None
whenever intermediate values or the final result are non-finite. Add regression
tests covering NaN, positive and negative infinity, and large finite values for
all four APIs, ensuring none can return Some(NaN).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 746501cd-c336-42cc-8d1e-2af1150f036f
📒 Files selected for processing (3)
.claude/board/EPIPHANIES.mdcrates/jc/src/lib.rscrates/jc/src/reliability.rs
| //! | ||
| //! # Why this lives in `jc` | ||
| //! | ||
| //! `jc` is otherwise a proof-in-code harness (the ten pillars). But the |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Synchronize the documented pillar count with the 12 registered pillars.
crates/jc/src/reliability.rs#L7-L7: replace “ten pillars” with the current count or avoid a hard-coded count..claude/board/EPIPHANIES.md#L5-L5: correct the shipped inventory without reordering the ledger.
📍 Affects 2 files
crates/jc/src/reliability.rs#L7-L7(this comment).claude/board/EPIPHANIES.md#L5-L5
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/jc/src/reliability.rs` at line 7, The documented pillar count is
stale: update the “ten pillars” reference in crates/jc/src/reliability.rs at
lines 7-7 to reflect the 12 registered pillars or remove the hard-coded count,
and correct the shipped inventory in .claude/board/EPIPHANIES.md at lines 5-5
without reordering its ledger.
Comment on lines
+76
to
+96
| pub fn pearson(x: &[f64], y: &[f64]) -> Option<f64> { | ||
| if x.len() != y.len() || x.len() < 2 { | ||
| return None; | ||
| } | ||
| let mx = mean(x)?; | ||
| let my = mean(y)?; | ||
| let mut sxy = 0.0; | ||
| let mut sxx = 0.0; | ||
| let mut syy = 0.0; | ||
| for (&xi, &yi) in x.iter().zip(y.iter()) { | ||
| let dx = xi - mx; | ||
| let dy = yi - my; | ||
| sxy += dx * dy; | ||
| sxx += dx * dx; | ||
| syy += dy * dy; | ||
| } | ||
| let denom = (sxx * syy).sqrt(); | ||
| if denom == 0.0 { | ||
| return None; // at least one series is constant | ||
| } | ||
| Some(sxy / denom) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Enforce the promised no-NaN API contract.
All four APIs can currently return Some(NaN) for non-finite inputs or overflowing intermediate calculations.
crates/jc/src/reliability.rs#L76-L96: validate both vectors and require a finite Pearson result.crates/jc/src/reliability.rs#L102-L149: reject non-finite values before sorting ranks.crates/jc/src/reliability.rs#L181-L203: validate every item score and require finite variances/result.crates/jc/src/reliability.rs#L239-L287: validate every rating and require finite mean squares/denominator/result.
Add regression tests using NaN, infinities, and large finite values.
📍 Affects 1 file
crates/jc/src/reliability.rs#L76-L96(this comment)crates/jc/src/reliability.rs#L102-L149crates/jc/src/reliability.rs#L181-L203crates/jc/src/reliability.rs#L239-L287
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/jc/src/reliability.rs` around lines 76 - 96, Update pearson, the
rank-correlation API, and the reliability APIs at crates/jc/src/reliability.rs
lines 76-96, 102-149, 181-203, and 239-287 to reject non-finite inputs before
calculations and return None whenever intermediate values or the final result
are non-finite. Add regression tests covering NaN, positive and negative
infinity, and large finite values for all four APIs, ensuring none can return
Some(NaN).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 08f96b8882
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+107
to
+110
| idx.sort_by(|&a, &b| { | ||
| xs[a] | ||
| .partial_cmp(&xs[b]) | ||
| .unwrap_or(std::cmp::Ordering::Equal) |
There was a problem hiding this comment.
When an upstream metric vector contains a non-finite value (for example a missing/overflow score encoded as NaN), this comparator treats it as equal for sorting, but the later tie grouping uses ==, so the NaN receives an ordinary rank; spearman(&[1.0, f64::NAN, 2.0], &[1.0, 2.0, 3.0]) returns Some(1.0). Pearson/Cronbach/ICC similarly return Some(NaN), contradicting the module contract that undefined estimates return None rather than NaN/garbage and allowing a bad measurement to falsely pass or poison agreement gates.
Useful? React with 👍 / 👎.
AdaWorldAPI
pushed a commit
that referenced
this pull request
Jul 17, 2026
The battery's module doc promised `None` "rather than returning NaN", but non-finite input slipped through: spearman(&[1.0, NAN, 2.0], ...) returned Some(1.0) (average_ranks maps NaN to a finite rank, so it never reached the delegated pearson guard) and pearson/cronbach_alpha/icc returned Some(NaN). Add an `all_finite` helper + up-front finite guards on all four public APIs (spearman guards the raw values before ranking), plus trailing result guards so finite-but-overflowing magnitudes (deviations -> +/-inf -> ratio NaN) also return None. Two regression tests cover NaN/+/-inf (incl. the reported reproducer) and 1e308-overflow inputs. Also soften the "ten pillars" aside in the module doc to "registered pillars" (the registry holds 12; a hard count in a doc-comment is a drift magnet). Found post-merge by CodeRabbit (Major) and Codex independently on #709. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Awg6TXocHcwTtc6eGsHcdD
AdaWorldAPI
added a commit
that referenced
this pull request
Jul 17, 2026
…iles-nhqgx1 jc reliability: enforce the no-NaN contract on non-finite input (follow-up to #709)
AdaWorldAPI
pushed a commit
that referenced
this pull request
Jul 17, 2026
…::reliability) D-TRI-2 (12-step ↔ 12-family agreement) is NO-GO pre-mint: both 12-vectors are codebook views of the unbuilt minted register (D-TRI-1). Scoped and recorded (E-D-TRI-2-MINT-BLOCKED-1) so no future session re-attempts it. The mint-free cousin: measure whether the three SHIPPED 12-family param tables actually agree — the exact D-TSC-1 "five tables diverged silently" failure mode, using the jc::reliability battery (#709/#710). New probe crates/jc/examples/style_table_agreement.rs (transcribes UNIFIED_STYLES, thinking-engine StyleParams, planner FieldModulation with per-value // SOURCE provenance; verified against source). Runs ICC(2,1)/ICC(3,1)/Cronbach α/pairwise Pearson-Spearman per shared dim in all-12 and 7-explicit modes; pre-registered ICC bands 0.75/0.50; exit-0 (DISTINCT is a valid finding, never a build failure). Measured verdict: - UNIFIED_STYLES ≡ thinking-engine StyleParams: PERFECT (r=ρ=1.0) — M9-dedup confirmed, the two are one table in two homes. - Planner FieldModulation's 7 explicitly-calibrated families: IDENTITY (ICC 0.90-0.98). - ONLY drift: the 5 planner default_modulation fallbacks (convergent/systematic/ divergent/diffuse/peripheral → flat 0.7/6/0.3) drop all-12 ICC to 0.71 (AMBIGUOUS). → TD-PLANNER-STYLE-DEFAULT-DRIFT-1: fill them from the canonical table. Board: EPIPHANIES E-D-TRI-2-MINT-BLOCKED-1, STATUS_BOARD D-TSC-1b, TECH_DEBT, triangle plan §6 annotation, AGENT_LOG. Measurement-only, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Awg6TXocHcwTtc6eGsHcdD
AdaWorldAPI
pushed a commit
that referenced
this pull request
Jul 17, 2026
IMPLEMENT — arigraph/community.rs: TripletGraph::communities() — multi-level Louvain modularity over the triplet adjacency (NARS-confidence-weighted edges, self-loops dropped), a carrier method (litmus-compliant), deterministic (BTreeMap-ordered moves + sorted-entity index), 5 inline tests. The STRUCTURAL partition complementing the episodic/family (part_of:is_a) basins AriGraph owns. Louvain core verified standalone before the in-crate build (two-triangle -> 2 comms Q=0.357, clique -> 1, determinism, empty-safe, weighted-cohesion); built against jc/examples/splat_louvain_modularity.rs (parallel-session reference). Leiden refinement pass + PPR/community fusion into retrieval.rs are D-GR-3b. WRITE — plan v1.2: - The base is a MEANING MEASUREMENT: COCA + NARS are co-equal core faculties; agnosticism is scoped to raw data + consumer modelling only (NOT meaning). The 12-byte facet is 6x2x8bit (CLAM), never f32 (f32 is build-time source only); 256 = ranking (needs cosine-replacement), 256^2 = distribution (as accurate as content-blind f32); the 6x(8:8) register is polymorphically part_of:is_a family-identity OR palette256^2 centroid (one register, two lenses) (S3a). - Leiden community synergies: distributional-meaning modes, NARS-truth-weighted (6 axes; the rung-3 layer of the ascent) (S3b). - DocumentID-KV / witness-handle seam: a stable DocumentID keys the consumer KV; the witness/node hold handles only, never raw bytes (S4a, D-GR-6). - New probes: P-COMMUNITY-BASIN-AGREE, P-HIER-LEIDEN-HHTL, P-MEASUREMENT-DETERMINISM, P-PQ-RANK (all via jc::reliability, merged #709/#710). - Meaning-substrate hazard (S7.6): do not reintroduce "keep language out"; the only real constraint is crate-layering (core can't dep the downstream deepnsm crate; use the in-core nsm/ copy). Board hygiene: INTEGRATION_PLANS entry updated to v1.2. Co-Authored-By: Claude <noreply@anthropic.com>
AdaWorldAPI
pushed a commit
that referenced
this pull request
Jul 17, 2026
IMPLEMENT — arigraph/community.rs: TripletGraph::communities() — multi-level Louvain modularity over the triplet adjacency (NARS-confidence-weighted edges, self-loops dropped), a carrier method (litmus-compliant), deterministic (BTreeMap-ordered moves + sorted-entity index), 5 inline tests. The STRUCTURAL partition complementing the episodic/family (part_of:is_a) basins AriGraph owns. Louvain core verified standalone before the in-crate build (two-triangle -> 2 comms Q=0.357, clique -> 1, determinism, empty-safe, weighted-cohesion); built against jc/examples/splat_louvain_modularity.rs (parallel-session reference). Leiden refinement pass + PPR/community fusion into retrieval.rs are D-GR-3b. WRITE — plan v1.2: - The base is a MEANING MEASUREMENT: COCA + NARS are co-equal core faculties; agnosticism is scoped to raw data + consumer modelling only (NOT meaning). The 12-byte facet is 6x2x8bit (CLAM), never f32 (f32 is build-time source only); 256 = ranking (needs cosine-replacement), 256^2 = distribution (as accurate as content-blind f32); the 6x(8:8) register is polymorphically part_of:is_a family-identity OR palette256^2 centroid (one register, two lenses) (S3a). - Leiden community synergies: distributional-meaning modes, NARS-truth-weighted (6 axes; the rung-3 layer of the ascent) (S3b). - DocumentID-KV / witness-handle seam: a stable DocumentID keys the consumer KV; the witness/node hold handles only, never raw bytes (S4a, D-GR-6). - New probes: P-COMMUNITY-BASIN-AGREE, P-HIER-LEIDEN-HHTL, P-MEASUREMENT-DETERMINISM, P-PQ-RANK (all via jc::reliability, merged #709/#710). - Meaning-substrate hazard (S7.6): do not reintroduce "keep language out"; the only real constraint is crate-layering (core can't dep the downstream deepnsm crate; use the in-core nsm/ copy). Board hygiene: INTEGRATION_PLANS entry updated to v1.2. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
crates/jc/src/reliability.rs— the four reliability/validity measures the workspace repeatedly names but had no callable implementation of:pearson(x, y)— product-moment correlation.spearman(x, y)— Pearson on the average-rank (tie-corrected) transform.cronbach_alpha(items)— internal-consistency reliability of ak-item scale (α = (k/(k-1))(1 − Σσ²ᵢ/σ²_total)).icc(ratings, IccForm::{Icc2_1, Icc3_1})— Shrout-Fleiss 1979 two-way ANOVA ICC, single-measure absolute-agreement and consistency forms.All return
Option<f64>(Noneon degenerate input — too few observations, zero variance, ragged matrices — no panics/NaN, per the no-unwrapguideline).Why
The chess-seam orientation surfaced that
crates/jcis a FORMAL-SCAFFOLD pillar-prover (tenprove() -> PillarResultpillars) with no callable ICC/Pearson/Cronbach-α/Spearman — only a privatespearman_rhoon rankings inprobe_p1_gamma_phase.rs, and a separately hand-rolled one in the stockfish-rs chess probe. The D-TRI measurement gates all name a battery that didn't exist as an API:Operator brief: "measure with lance-graph/crates/JC ICC Pearson Cronbach alpha spearman."
Correctness
Point estimates ONLY — significance calibration stays
jc::jirak's job under I-NOISE-FLOOR-JIRAK (the fingerprint bits are weakly dependent by construction; classical IID Berry-Esseen is wrong). Formulas pinned to reference values:psych::ICC/ Pythonpingouin.12 unit tests + 3 doctests green;
cargo clippy --libexit 0.Follow-up (not in this PR)
The two existing hand-rolled
spearman_rhocopies (probe_p1, stockfish-rs) can migrate tojc::reliability::spearman;thinking-engine/src/cronbach.rs(excluded crate) is separate prior art. Left as a follow-up, not forced here.Board
EPIPHANIES
E-JC-IS-NOT-A-METRIC-BATTERY-1(same commit).🤖 Generated with Claude Code
https://claude.ai/code/session_01Awg6TXocHcwTtc6eGsHcdD
Generated by Claude Code
Summary by CodeRabbit
New Features
Documentation