diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 75745ac6..f15c1df6 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,11 @@ +## 2026-07-17 — E-EMPIRICAL-VS-SCIENTIFIC-JC-1 — layer boundary: jc is the SCIENTIFIC crate (consume, never extend to fit a probe); graphrag probes are EMPIRICAL (measure, consume jc's science). P-COMMUNITY-BASIN-AGREE (S1) harness shipped — mechanism green, verdict awaits a real is_a corpus + +**Status:** SHIPPED (branch `claude/happy-hamilton-0azlw4`, post-#719). `crates/lance-graph/examples/p_community_basin_agree.rs` + `jc` dev-dep on lance-graph. Runs clean; fmt + clippy clean. + +- **The boundary (operator, 2026-07-17):** "JC is a scientific crate. Yours are empirical." jc (`reliability::{pearson,spearman,cronbach_alpha,icc}`, `weyl`, `jirak`, `cartan`) is the certified SCIENTIFIC surface — an empirical probe **consumes** it, never extends it. The instinct to "add ARI/NMI to jc so my probe has the right partition metric" is a **layer violation** (empirical need reaching into the scientific crate). Correct posture: the empirical probe constructs its own measurement (partitions → pairwise co-membership) and feeds it through jc's EXISTING science; a missing metric is an OBSERVATION reported to the scientific track, not a bolt-on. Probes live on the empirical side (`lance-graph/examples/`, dev-depping jc), never inside jc. +- **P-COMMUNITY-BASIN-AGREE (S1) — the mechanism:** two partitions of one entity set — Leiden `communities()` over ALL edges vs basin = connected components under ONLY `is_a`/`part_of` edges — compared by φ = `jc::pearson` of their pairwise co-membership vectors (φ=1 iff identical up to relabeling). Empirical result on the synthetic fixture: aligned → **φ=1.0000** (the identity the ruling predicts, cleanly shown); one deliberate cross-basin entity (`robot` is_a pet but structurally in the tree cluster) → **φ=0.5500**, `robot` correctly named as the single bridge / revision candidate. +- **The gate it feeds (S1 handoff, #719):** on a REAL is_a-annotated corpus this φ decides the D-TRI-1 community-id mint — identity (φ→1) ⇒ community-id collapses into the is_a rail, NO tenant minted; distinct (φ<1) ⇒ community-id earns the batched mint. Harness = the mechanism; verdict = the real-corpus run (like G0's KILL/PASS). Cross-ref `E-GRAPHRAG-DGR3B-1` (§3b.1 identity ruling); #719 (mint train waiting on S1). + ## 2026-07-17 — E-GRAPHRAG-DGR3B-1 — D-GR-1 + D-GR-3b shipped (DocGraphQuery, PPR, Leiden refinement, BM25, G0 harness); two operator sharpenings — (a) the cosine-replacement distance ALREADY EXISTS (grep it, don't build it); (b) part_of:is_a category ≡ Leiden community ≡ episodic-witness basin ≡ HHTL family identity — one concept **Status:** SHIPPED (branch `claude/happy-hamilton-0azlw4`, fresh post-#714 main). `lance-graph` 945 lib + 18 D-GR module tests green; `lance-graph-contract` 10 green + clippy `-D warnings` exit 0; G0 harness runs. All D-GR-1/3b capabilities are pure/reversible/no-write-path — they land ahead of G0 exactly as D-GR-3a (#714) did; the D-GR-2 retrieval WIRING stays gated on the G0 real-corpus verdict. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index bd1dbeb9..c20df613 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -4,14 +4,15 @@ Plan: `.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md` (v1.2). Pure/ | D-id | Title | Repo | Status | Evidence | |---|---|---|---|---| -| D-GR-1 | `DocGraphQuery` zero-dep contract trait + `ScoredId` (rung→walk dispatch) + D-GR-2 design | lance-graph | Shipped (this PR) — `doc_graph.rs`, 9 tests | plan §5 | +| D-GR-1 | `DocGraphQuery` zero-dep contract trait + `ScoredId` (rung→walk dispatch) + D-GR-2 design | lance-graph | Shipped (#716) — `doc_graph.rs`, 9 tests | plan §5 | | D-GR-3a | `TripletGraph::communities()` multi-level Louvain, deterministic | lance-graph | Shipped (#714) | plan §3b | -| D-GR-3b | PPR (`personalized_pagerank`) + Leiden `refine_connected` + BM25 (`Bm25Index`) — pure capabilities | lance-graph | Shipped (this PR) — 13 tests | plan §3b, §5 | -| G0 | P-GRAPH-LOADBEARING harness (vector-only vs vector+PPR+community) | lance-graph | Harness shipped (this PR); real-corpus verdict OPEN | plan §5, §6 | +| D-GR-3b | PPR (`personalized_pagerank`) + Leiden `refine_connected` + BM25 (`Bm25Index`) — pure capabilities | lance-graph | Shipped (#716) — 13 tests | plan §3b, §5 | +| G0 | P-GRAPH-LOADBEARING harness (vector-only vs vector+PPR+community) | lance-graph | Harness shipped (#716); real-corpus verdict OPEN | plan §5, §6 | | D-GR-2 | Fuse CAM-PQ+SPO-G+PPR+community into `retrieval.rs` under the #708 RungElevator | lance-graph | Design done (in `doc_graph.rs` module-doc); impl GATED on G0 | plan §5 | | D-GR-4 | Community summaries (no-LLM DeepNSM; Rig-oracle tail) | lance-graph | Deferred (W3-coupled) | plan §5 | | D-GR-5 | `ogar-doc` reconstruct/related-docs → `DocGraphQuery` seam | lance-graph + OGAR | Deferred (mint-gated, doc-W4 council) | plan §5 | | D-GR-6 | Witness-KV separation (DocumentID handle → consumer KV) | lance-graph | Deferred (doc-W4 council) | plan §4a, §5 | +| P-COMMUNITY-BASIN-AGREE (S1) | Empirical probe: Leiden community vs `is_a`-basin agreement, φ via `jc::pearson` (consumes jc science, doesn't extend it) | lance-graph | Harness SHIPPED — φ=1.0 aligned / 0.55 bridged (`robot` = the bridge); real-corpus verdict OPEN, **gates the D-TRI-1 community-id mint** | plan §6, #719 | ## triangle-tenants-gestalt-separation-v1 — triangle tenants, surface separation, chess quarantine diff --git a/Cargo.lock b/Cargo.lock index 29a475e3..8e17e3e3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4337,6 +4337,10 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jc" +version = "0.1.0" + [[package]] name = "jieba-macros" version = "0.9.0" @@ -4821,6 +4825,7 @@ dependencies = [ "datafusion-sql", "deltalake", "futures", + "jc", "lance", "lance-arrow", "lance-graph-catalog", diff --git a/crates/lance-graph/Cargo.toml b/crates/lance-graph/Cargo.toml index 89280fee..f8fecc2d 100644 --- a/crates/lance-graph/Cargo.toml +++ b/crates/lance-graph/Cargo.toml @@ -103,4 +103,5 @@ lance-index = "=7.0.0" tempfile = "3" tokio = { version = "1.37", features = ["macros", "rt-multi-thread"] } causal-edge = { path = "../causal-edge" } +jc = { path = "../jc" } lance-graph-planner = { path = "../lance-graph-planner" } diff --git a/crates/lance-graph/examples/p_community_basin_agree.rs b/crates/lance-graph/examples/p_community_basin_agree.rs new file mode 100644 index 00000000..a50f7a07 --- /dev/null +++ b/crates/lance-graph/examples/p_community_basin_agree.rs @@ -0,0 +1,262 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! P-COMMUNITY-BASIN-AGREE — the **empirical** probe for the operator's identity +//! ruling: *`part_of:is_a` category ≡ Leiden community ≡ episodic-witness basin — +//! one concept* (graphrag plan §3b.1). +//! +//! # Empirical, not scientific (the layer boundary) +//! +//! This harness is **empirical**: it sets up a measurement over real graph +//! structure and reports an observation. The **statistic** it reports is +//! `jc`'s — the certified scientific crate ([`jc::reliability::pearson`] / +//! [`spearman`]). The empirical construction (the two partitions, their pairwise +//! co-membership vectors) is this probe's; the science is consumed, never +//! re-implemented or extended here. jc has no partition-agreement metric +//! (Rand/ARI/NMI); if the empirical reading turns out to need one, that is an +//! **observation reported to the scientific track**, not a metric bolted on from +//! the empirical side. +//! +//! # What it measures +//! +//! Two partitions of the same entity set: +//! - **Community** — [`TripletGraph::communities`] (multi-level Louvain + +//! Leiden refinement) over the *whole* relational field (all edges). +//! - **Basin** — the connected components under **only** the `is_a`/`part_of` +//! edges (the inherited/taxonomic grouping the plan names as the basin rail). +//! +//! Agreement = Pearson (φ) of the two partitions' pairwise co-membership vectors +//! (`same-community?` vs `same-basin?` over every entity pair). φ = 1.0 iff the +//! partitions are identical up to relabeling — the **identity** the ruling +//! predicts. φ < 1.0 localises the disagreements: those entities are exactly the +//! *discovered bridges / revision candidates* (a community that crosses a basin). +//! +//! # The gate it feeds +//! +//! On a real `is_a`-annotated corpus this number is the S1 gate the other +//! session's D-TRI-1 classid-half mint waits on: **identity (φ→1) ⇒ community-id +//! collapses into the `is_a` rail, no new tenant minted; distinct (φ<1) ⇒ +//! community-id earns its place in the batched mint.** This synthetic fixture +//! exercises the *mechanism*; the verdict needs the real corpus. +//! +//! Run: `cargo run -p lance-graph --example p_community_basin_agree` + +use std::collections::HashMap; + +use jc::reliability::{pearson, spearman}; +use lance_graph::graph::arigraph::triplet_graph::{Triplet, TripletGraph}; + +/// A `(subject, relation, object)` fact. +type Fact = (&'static str, &'static str, &'static str); + +/// Relations that define the **basin** rail (the inherited/taxonomic grouping). +fn is_taxonomic(relation: &str) -> bool { + let r = relation.to_lowercase(); + r.contains("is_a") || r.contains("is a") || r.contains("part_of") || r.contains("part of") +} + +fn build_graph(facts: &[Fact]) -> TripletGraph { + let mut g = TripletGraph::new(); + let ts: Vec = facts + .iter() + .enumerate() + .map(|(i, (s, r, o))| Triplet::new(s, o, r, i as u64)) + .collect(); + g.add_triplets(&ts); + g +} + +/// Basin partition: dense connected-component ids over ONLY the `is_a`/`part_of` +/// edges, aligned to `entities` order. Entities in no taxonomic edge are their +/// own basin. A deterministic union-find (find with path compression, union by +/// lower root so ids are stable given the sorted `entities`). +fn basin_labels(graph: &TripletGraph, entities: &[String]) -> Vec { + let index: HashMap<&str, usize> = entities + .iter() + .enumerate() + .map(|(i, e)| (e.as_str(), i)) + .collect(); + let mut parent: Vec = (0..entities.len()).collect(); + + fn find(parent: &mut [usize], mut x: usize) -> usize { + while parent[x] != x { + parent[x] = parent[parent[x]]; + x = parent[x]; + } + x + } + + for t in &graph.triplets { + if t.is_deleted() || !is_taxonomic(&t.relation) { + continue; + } + let (Some(&a), Some(&b)) = (index.get(t.subject.as_str()), index.get(t.object.as_str())) + else { + continue; + }; + let (ra, rb) = (find(&mut parent, a), find(&mut parent, b)); + if ra != rb { + // union by lower root → deterministic given sorted entities. + let (lo, hi) = (ra.min(rb), ra.max(rb)); + parent[hi] = lo; + } + } + + // Densify roots to 0..k by first appearance (deterministic). + let mut root_to_dense: HashMap = HashMap::new(); + let mut next = 0u32; + (0..entities.len()) + .map(|i| { + let r = find(&mut parent, i); + *root_to_dense.entry(r).or_insert_with(|| { + let d = next; + next += 1; + d + }) + }) + .collect() +} + +/// Pairwise co-membership over all `i Vec { + let n = labels.len(); + let mut v = Vec::with_capacity(n * (n - 1) / 2); + for i in 0..n { + for j in (i + 1)..n { + v.push(if labels[i] == labels[j] { 1.0 } else { 0.0 }); + } + } + v +} + +fn distinct(labels: &[u32]) -> usize { + let mut s = labels.to_vec(); + s.sort_unstable(); + s.dedup(); + s.len() +} + +fn measure(title: &str, facts: &[Fact]) { + let graph = build_graph(facts); + let comms = graph.communities(); + let community = comms.labels.clone(); + let basin = basin_labels(&graph, &comms.entities); + + let comm_co = comembership(&community); + let basin_co = comembership(&basin); + + // Scientific statistic — jc, consumed as-is. + let phi = pearson(&comm_co, &basin_co); + let rho = spearman(&comm_co, &basin_co); + // Empirical descriptive — raw fraction of pairs that agree (Rand-like). + let agree_frac = if comm_co.is_empty() { + f64::NAN + } else { + comm_co + .iter() + .zip(&basin_co) + .filter(|(a, b)| a == b) + .count() as f64 + / comm_co.len() as f64 + }; + + // Bridges = entities whose basin differs from the MAJORITY basin of their + // community — the minority member a community pulled across a basin line + // (the discovered bridge / revision candidate, plan §3b.1). + let mut comm_basin_counts: HashMap> = HashMap::new(); + for i in 0..community.len() { + *comm_basin_counts + .entry(community[i]) + .or_default() + .entry(basin[i]) + .or_insert(0) += 1; + } + let majority_basin: HashMap = comm_basin_counts + .iter() + .map(|(&c, counts)| { + let maj = counts + .iter() + .max_by_key(|(_, &n)| n) + .map(|(&b, _)| b) + .unwrap_or(0); + (c, maj) + }) + .collect(); + let bridges: Vec<&str> = comms + .entities + .iter() + .enumerate() + .filter(|&(i, _)| basin[i] != majority_basin[&community[i]]) + .map(|(_, e)| e.as_str()) + .collect(); + + println!("── {title} ──"); + println!( + " entities={} communities={} basins={}", + comms.entities.len(), + distinct(&community), + distinct(&basin) + ); + println!( + " φ (jc::pearson) = {} ρ (jc::spearman) = {} pair-agreement = {:.3}", + phi.map(|x| format!("{x:.4}")) + .unwrap_or_else(|| "n/a (constant)".into()), + rho.map(|x| format!("{x:.4}")) + .unwrap_or_else(|| "n/a (constant)".into()), + agree_frac, + ); + match phi { + Some(p) if p > 0.999 => { + println!(" → IDENTITY on this fixture (φ≈1): community ≡ basin, no bridges.") + } + Some(_) => println!( + " → DISTINCT-with-bridges: {:?} cross a basin (the revision candidates).", + bridges + ), + None => println!(" → degenerate (a partition is trivial); not informative."), + } + println!(); +} + +fn main() { + println!("== P-COMMUNITY-BASIN-AGREE (empirical probe; statistic = jc science) ==\n"); + + // Scenario A — clean: two taxonomic basins, structure matches exactly. + // Expect φ≈1.0 (the identity the ruling predicts, cleanly demonstrated). + let clean: &[Fact] = &[ + ("cat", "is_a", "pet"), + ("dog", "is_a", "pet"), + ("hamster", "is_a", "pet"), + ("cat", "relates_to", "dog"), + ("dog", "relates_to", "hamster"), + ("hamster", "relates_to", "cat"), + ("oak", "is_a", "tree"), + ("pine", "is_a", "tree"), + ("birch", "is_a", "tree"), + ("oak", "relates_to", "pine"), + ("pine", "relates_to", "birch"), + ("birch", "relates_to", "oak"), + ]; + measure("A · aligned (identity expected)", clean); + + // Scenario B — one deliberate cross: `robot` is_a pet (basin=pet) but is + // structurally pulled entirely into the tree cluster (community=tree). + // Expect φ<1.0 with `robot` named as the single bridge / revision candidate. + let mut bridged: Vec = clean.to_vec(); + bridged.extend_from_slice(&[ + ("robot", "is_a", "pet"), + ("robot", "relates_to", "oak"), + ("robot", "relates_to", "pine"), + ("robot", "relates_to", "birch"), + ]); + measure("B · one cross-basin bridge (distinct expected)", &bridged); + + println!( + "NOTE: synthetic mechanism, NOT the verdict. The S1 gate reads φ on a REAL\n\ + is_a-annotated corpus: identity (φ→1) ⇒ community-id collapses into the\n\ + is_a rail, NO new tenant minted; distinct (φ<1) ⇒ community-id earns the\n\ + batched D-TRI-1 mint. jc supplies the statistic; if a Rand/ARI/NMI metric\n\ + is wanted, that is an observation for the scientific (jc) track." + ); +}