diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index aaaf60ab..f425e35b 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,44 @@ +## 2026-07-17 — graphrag-doc-retrieval-soa-integration v1 — DESIGN + FIRST CODE (probe-gated) + +**Plan:** `.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md` +**v1.2 (2026-07-17, operator-directed):** (a) the base is a *meaning measurement* — +COCA + NARS are co-equal core faculties; agnosticism is scoped to raw data + +consumer modelling only; the 12-byte facet is `6×2×8bit` (CLAM), never f32; 256 = +ranking (needs cosine-replacement), 256² = distribution (as accurate as +content-blind f32); the `6×(8:8)` register is polymorphically `part_of:is_a` +family-identity OR palette256² centroid (§3a). (b) Leiden community synergies — +distributional-meaning modes, NARS-truth-weighted (§3b). (c) the DocumentID-KV / +witness-handle seam (§4a). **FIRST CODE SHIPPED (D-GR-3a):** `arigraph/community.rs` +— `TripletGraph::communities()`, multi-level Louvain, deterministic, 5 tests. +**v1.1 (2026-07-17, operator-directed):** NO standalone `crates/graphrag` — +**expand AriGraph in place** (`arigraph/community.rs` Leiden + PPR into the +existing `arigraph/retrieval.rs`), complementing the episodic-witness basins +AriGraph already owns (`witness_corpus.rs`/`episodic.rs`). A crate would be a +free-function-on-carrier's-state (litmus reject) + a parallel retrieval layer +beside the existing one; AriGraph has no community/partition type today, so +Leiden fills the gap. Dissolves the §10 dep-weight feature-gate. The +`DocGraphQuery` contract trait + a small out-of-graph BM25 leg are the only +zero-dep additions. +**Scope (v1):** evaluation + wave plan for a query-side retrieval surface + a +`DocGraphQuery` contract surface, both reading the *calcified* +document-graph substrate. Reframes the task's "lance-graph feeds ogar-doc": OGAR +owns doc ingestion + the `document 0x080B` mint (`ogar-from-docv1` → +`ogar-doc-ir` → `ogar-doc`, all built); lance-graph provides the contract types +OGAR consumes + query capability. graphrag is query-side ONLY — never ingests +(door-knocking-compiler test; assembler-vs-storage boundary). REUSE: +`causal_edge::CausalEdge64` (SPO edge), native NARS `(freq,conf)`, +`WitnessTable`/`EpisodicEdges64` (provenance; `EpisodicWitness64` = a *queued* +column), AriGraph `TripletGraph`+`spo_bridge` (fact store), `MailboxSoaView`, +CAM-PQ, `lance-graph-arm-discovery` (extraction proposer). BUILD only the 3 +gaps: hierarchical Leiden, HippoRAG-PPR reset-distribution, BM25. **Retrieval = +rung ascent** (#708 / D-TRI-6 `RungElevator` + canonical `RungLevel` + +predicate-plane widen; BLOCK ascends, FLOW at base) up the §3a Maslow pyramid. +Probe-first: **G0 P-GRAPH-LOADBEARING** (graph beats vector-only?) gates all +Leiden/PPR code. graphrag-rs (automataIA) = REUSE-AS-REFERENCE +(`E-V3-GRAPHRAG-INV-1`). Composes atop +`oxigraph-arigraph-cognitive-shader-soa-merge-v1`. D-GR-1..5; D-GR-5 (OGAR seam) +mint-gated + baton-audited. No bytes land. + ## 2026-07-17 — triangle-tenants-gestalt-separation v1 — DESIGN (mint-gated) **Plan:** `.claude/plans/triangle-tenants-gestalt-separation-v1.md` diff --git a/.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md b/.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md new file mode 100644 index 00000000..aacf6428 --- /dev/null +++ b/.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md @@ -0,0 +1,557 @@ +# graphrag + doc-retrieval on the V3 SoA — integration plan v1 + +> **Status:** DESIGN + FIRST CODE. **v1.2 (2026-07-17):** the base is a *meaning +> measurement* — COCA + NARS are co-equal core faculties, agnosticism is scoped to +> raw data + consumer modelling only (§3a); Leiden community synergies (§3b); the +> DocumentID-KV / witness-handle seam (§4a). **v1.1:** expand AriGraph in place — +> NO standalone `crates/graphrag` (§1). **First code shipped: `arigraph/community.rs` +> (D-GR-3a) — `TripletGraph::communities()`, multi-level Louvain, deterministic.** +> Aligned to +> **#708 / D-TRI-6 — now MERGED (`8d3209c`; `E-RUNG-ASCENT-WIRED-1` SHIPPED)** + +> `E-MASLOW-PYRAMID-OF-COGNITION-1` and the `triangle-tenants-gestalt-separation-v1` +> Maslow pyramid. The rung-ascent loop is landed code (D-GR-2's dependency is +> satisfied). Any OGAR classid touch is **mint-gated** (batched, baton-audited). +> Probe-first: the **graph-load-bearing baseline** is the FIRST deliverable, +> before any Leiden/PPR code. +> **Grounding:** three Opus reader-agents (v3 waves+census · CausalEdge64/witness +> substrate · OGAR↔lance-graph boundary) + `.claude/knowledge/graphrag-rs-inventory.md` +> (`E-V3-GRAPHRAG-INV-1`, the pre-existing automataIA/graphrag-rs audit) + the +> automataIA repo survey (graphrag-rs ★519, graph-librarian-rs, wasm-typst-studio-rs, +> lodviz-rs). Citations are as-of `efd21d2` (#707) — verify before coding. + +## §0. The reframe — direction correction (READ FIRST) + +The task said "create crates **inside lance-graph feeding into** ogar-doc." Run +through `assembler-vs-storage-substrate.md` + `compilation-vs-runtime-substrate.md`, +that **inverts the dependency** and is WRONG-SHELF. The corrected direction: + +- **OGAR = assembler.** It OWNS document ingestion, the `document 0x080B` mint, + and persistence — already built as `ogar-from-docv1` → `ogar-doc-ir` → + `ogar-doc` (doc-W4), all in `/home/user/OGAR/crates/`. A lance-graph + doc-transcode crate would **duplicate `ogar-from-docv1`** and invert the dep. +- **lance-graph = spine.** It provides **contract types** (`ClassView`, facet / + envelope, masks) that `ogar-doc` builds against — OGAR consumes + `lance-graph-contract`, **never the reverse** — **plus query capability** over + the *calcified* substrate. +- **Lance = calcification.** "There is no ingest API." The SoA/Lance write is + calcification by the batch writer on-behalf of a mailbox owner. +- **The door-knocking test forbids** any build step or typed-value constructor + asking the running substrate for permission. `doc.v1 → ogar_doc_ir` is a plain + Rust fn, unit-testable with nothing running. + +**Therefore the honest shape of "feeds both ogar-doc and graphrag":** a single +**query/view surface** over the calcified document-graph, consumed by two +readers — `crates/graphrag` (retrieval) and OGAR `ogar-doc`'s +`reconstruct_document` ("re-issue with updated knowledge" / "documents in this +community"). `ogar-doc` calls it **via a `lance-graph-contract` trait** +(`DocGraphQuery`), so OGAR still depends only on the contract. graphrag provides +the impl. **Neither ingests; both read.** + +## §1. Evaluation — what to build, what to reject + +> **★ v1.1 REVISION (2026-07-17, operator-directed): NO `crates/graphrag`.** +> Expand **AriGraph** in place instead. AriGraph already owns `retrieval.rs` +> (OsintRetriever — BFS+episodic fusion), `witness_corpus.rs`, `episodic.rs`, +> `spo_bridge.rs`, `markov_soa.rs` — the retrieval brain + the episodic-witness +> basins are already there. A separate crate is a **free function on AriGraph's +> state** (the litmus-test reject) and a **parallel retrieval layer** beside an +> existing one. There is **no** community/basin/cluster/Leiden/partition type in +> AriGraph today (grep-confirmed) — so Leiden fills the one *structural-partition* +> gap that **complements** the episodic-witness partition AriGraph owns. This +> dissolves the §10 dep-weight feature-gate: the graph capabilities live where the +> graph lives (core). §4/§10 updated; the #708 rung-ascent alignment (§3), the +> reuse census (§2), the boundary (§0), and the probes (§6) all carry over. + +| Candidate | Verdict | Why | +|---|---|---| +| lance-graph doc-transcode / perceptual-IR / `document`-mint crate | **REJECT** | Duplicates OGAR `ogar-from-docv1`/`ogar-doc-ir`/`ogar-doc`; inverts the dep; WRONG-SHELF. Doc ingestion is assembler-side. | +| Monolithic `graphrag` (extraction→graph→retrieval from scratch, à la graphrag-rs) | **REJECT** | Extraction, SPO edges, fact store, vector/CAM-PQ retrieval already EXIST (§2). graphrag-rs = REUSE-AS-REFERENCE only; its `LanceDBStore` is a stub, its `InferenceEngine` is the no-singleton anti-exhibit. | +| **`crates/graphrag`** — thin retrieval orchestrator crate | **REJECT (v1.1)** | Duplicates AriGraph's existing `retrieval.rs`; a free-function-on-carrier's-state (litmus reject); a new *layer* where the rule is new *method*. **Expand AriGraph instead.** | +| **Leiden community → `arigraph/community.rs`** (NEW module) | **BUILD** | The structural partition; complements `witness_corpus`/`episodic` basins. First partition type in AriGraph — fills the gap, doesn't clutter. | +| **PPR / HippoRAG → extend `arigraph/retrieval.rs`** | **BUILD** | A graph-ranking sibling of the existing `find_path`/`get_associated`/BFS+episodic fusion; community = a third fusion signal. Backed by in-core `blasgraph::hdr_pagerank`. | +| **`DocGraphQuery` trait** in `lance-graph-contract` (impl = AriGraph methods) | **BUILD (thin)** | The zero-dep read surface `ogar-doc` consumes; keeps OGAR on the contract, not the impl. | +| BM25 / lexical leg | **BUILD (small, OUT of AriGraph)** | Text-index, not a graph op; a tiny module elsewhere (or reuse existing). | +| New lance-graph OGAR-seam crate | **REJECT** | `lance-graph-ogar` already IS the seam. Extend if needed. | + +**Net (v1.1): expand AriGraph** — `arigraph/community.rs` (Leiden) + PPR into +`arigraph/retrieval.rs`, complementing the witness/episodic basins it already +owns; a thin `DocGraphQuery` contract trait (impl = those methods); a small +out-of-graph BM25 leg. **No new crate.** Everything else is wiring. + +## §2. Substrate reuse — the census (build only the gaps) + +REUSE (do **not** rebuild — the substrate provides these): + +- **SPO edge = `causal_edge::CausalEdge64`** (8 B, `#[repr(transparent)]`). v2 + layout (default `causal-edge-v2-layout`): S/P/O palette idx (bits 0-23), NARS + **frequency**/**confidence** u8 (24-39), Pearl-2³ causal mask (40-42), + direction triad (43-45), inference mantissa i4 (46-49), plasticity (50-52), + **W-slot witness handle** (53-58), truth-band lens (59-60), spare (61-63). + **Disambiguation hazard:** two `CausalEdge64` exist — use `causal_edge` (SPO + palette), NOT `thinking_engine::layered` (8×u8 strength vector). +- **NARS truth = native `(freq u8, conf u8)`** in the edge (bits 24-39); + `expectation()`, `evidence_weight()` on `CausalEdge64`. Don't add a + confidence/weight field. +- **Provenance/witness.** `EpisodicWitness64` is **not a symbol** — it names a + *queued* SoA column (`soa_view.rs` `episodic_witness()` deferred). Today use + `EpisodicEdges64` (4×16-bit MRU episodic edges) + `WitnessTable<64>` / + `WitnessEntry{mailbox_ref, spo_fact_ref}` resolving the CE64 v2 W-slot. Don't + add a provenance field; **materializing the column is a candidate probe**, not + an assumption. +- **Fact store = AriGraph `TripletGraph` + SPO-G quad.** `Triplet` + G-slot + (`ContextTag::{Observation, Intervention}`) = the quad; `CounterfactualSpoG` + from `TripletGraph::intervene_on` (Pearl rung-2). Promote a CE64 to a graph + edge via `spo_bridge::promote_to_spo` past the truth gate. Cold, in + lance-graph core — **not** a SoA column. +- **Read surface = `MailboxSoaView`** (zero-copy; `edges:[CausalEdge64;N]` is the + EdgeColumn). Don't re-encode to Arrow. Node adjacency = the 16-byte `EdgeBlock` + (coarse, key-side); identity/class = `NodeGuid` / `class_id` u16 (resolves via + `lance-graph-ontology`). Three edge encodings — EdgeBlock (adjacency) / CE64 + (causal weight) / EpisodicEdges64+WitnessTable (provenance) — **never conflate**. +- **Extraction (upstream proposer) = `lance-graph-arm-discovery`** (Aerial+ + transcode, integer codebook-distance oracle → `{s,p,o,f,c}` ndjson NARS-truth + SPO candidates) + `nsm/parser.rs` PoS-FSM→SPO. No-LLM, workspace-native. +- **Vector retrieval = CAM-PQ** (`cam_pq/{ivf,storage,udf}`; codec ICC 0.9999, + `cam-pq-production-wiring-v1`). *(The exact vector-search entry symbol moved + since the agent read — verify the current entry point.)* + +BUILD (the genuine gaps — each gated on §6 P0): + +1. **Hierarchical Leiden community detection** — in-tree is example-only + (`jc/examples/splat_louvain_modularity.rs`, `splat_lpa_label_propagation.rs`); + graphrag-rs `leiden.rs` is single-level, REUSE-AS-REFERENCE. Build hierarchical + over the CE64/SPO-G graph. +2. **HippoRAG-PPR** (personalized PageRank with reset distribution) — plain + `hdr_pagerank` (`blasgraph/ops.rs`) + `ScentCsr::spmv` exist; the reset/seed + distribution + dual-signal (passage-weight) is the gap. Reference + graphrag-rs `hipporag_ppr.rs`. +3. **BM25 keyword arm** — ABSENT; small pure function. Reference graphrag-rs + `KeywordExtractor`. + +## §3. The #708 alignment — retrieval IS rung ascent + +**Do not build a bespoke retrieval-escalation ladder.** graphrag retrieval +dispatches through the **`RungElevator`** wired by **#708 (merged `8d3209c`, +D-TRI-6; `E-RUNG-ASCENT-WIRED-1`)** — now landed code, not a landing dependency. +The elevator + ladder type are contract types: +`lance_graph_contract::cognitive_shader::{RungElevator, RungLevel}` +(`cognitive_shader.rs:272` / `:157`), and the widen is +`cognitive-shader-driver::driver::rung_widened_layer_mask` +(`driver.rs:701` — `fn(base, level, req_mask: u8) -> u8`). Per #708: `on_gate()` +advances one rung per dispatch; the current rung selects the cycle's cascade +breadth via `rung_widened_layer_mask` — a UNION over the **8-bit predicate-plane +mask (CAUSES..BECOMES)**, **identity at base (zero regression), superset-monotone +above**. `RungLevel` is now canonical in `lance-graph-contract` (post-#708 dedup; +thinking-engine re-exports; `as_u8()` added). The follow-up review fix `17368ea` +locks the ordering — **advance the elevator BEFORE the sinks** — which graphrag's +`retrieve.rs` must mirror (gate the walk on the *post-advance* rung). **BLOCK +ascends, FLOW relaxes to base.** + +Map retrieval onto the Maslow pyramid (`triangle-tenants-gestalt-separation-v1` +§3a; the rung-content ladder in `persona-vs-rung-ladder.md`): + +| Rung | Pyramid level | Retrieval action | Predicate-plane mask | +|---|---|---|---| +| 0–1 (base, FLOW) | observation / gestalt | CAM-PQ vector + BM25 surface lookup | identity | +| 2 | SPO 2³ | SPO-G edge hop over `CausalEdge64` | widen (Pearl rung-1 planes) | +| 3 | CE64 NARS candidates | HippoRAG-PPR + community-scoped expansion, NARS-truth-weighted | wider CAUSES..BECOMES union | +| 3–4 | NARS candidate design / revision | community summaries (Rig oracle → compiled template, W3) | full | +| 4 (apex) | counterfactual | Pearl rung-2 intervention (`ContextTag` G-slot) | — | + +**The graph is load-bearing precisely because BLOCK ascends the elevator** +(retrieval surprise / low NARS confidence / contradiction widens the traversal). +graphrag never re-decides the level — it reads the driver's `RungLevel` and +supplies the wider graph walk. This is the anti-decorative-graph guarantee (vs +the graph-librarian-rs anti-pattern where the graph is built but never traversed). + +## §3a. The base — a meaning MEASUREMENT: tokenization, ranking (256), distribution (256²) + +The substrate is a **meaning substrate, not a meaning-agnostic one** (operator, +2026-07-17). Two **co-equal core faculties** carry it: **COCA** (distributional +semantics / syntax / pragmatics — the meaning layer) and **NARS** (truth / +reasoning). Agnosticism is scoped ONLY to **raw data** (bytes → the KV, §4a) and +**consumer modelling** (app schema); the graph is the *opposite* of agnostic +about meaning — it **measures** it. The `markov_soa` "vocabulary-agnostic" hot +loop is codebook-*parametric* for cross-domain SIMD reuse (an injected +`Fn(u16,u16)->u8`); for language/document content the injected distance IS the +core COCA meaning — correct, not a "language lens." *(Correction target: the +shipped `markov_soa.rs:16,28` "keep language out" comment reflects the older +framing; the COCA meaning codebook wants a core-reachable home, not quarantine in +the downstream `deepnsm` crate — board follow-up.)* + +**Tokenization = the L4 facet tenant.** CAM-PQ = 6 subspaces × 256 centroids +(`cam_pq/udf.rs:27,37`); DeepNSM IS that same PQ, trained from the 96D **f32** +COCA vectors *at build time* (`deepnsm/codebook.rs:3-4` — `NUM_SUBSPACES=6`, 256 +centroids) but emitting the **8-bit** code. Content → a `6×2×8bit` code = the +node's **L4 `6×(8:8)` facet tenant** (`le-contract.md:59`). +Quantizing to one centroid per subspace *is* tokenization; the token IS the +facet, no separate index. COCA ≈ 98.4 % English → the code is "as good as +agnostic" (a near-universal meaning basis, a *measurement*, not a narrowing bias). + +**Everything is 8-bit (CLAM), never f32.** The 12-byte facet is **`6×2×8bit`** — +six subspaces, each a two-byte pair — and the CLAM tree indexes those 8-bit codes; +**f32 is a build-time source only** (the codebook is trained from f32, the runtime +register is not). The same `6×(8:8)` register is **polymorphic** (classview-selected): +read as **`part_of:is_a` family identity** (L1 — the HHTL family/basin rail) OR as +the **palette256² centroid** (L4 CAM-PQ). So "6×256:256" is the is_a family +identity AND the centroid — one register, two lenses; this is *why* communities +(structural) and is_a basins (family) couple so tightly (§3b.1). + +**256 = ranking; 256² = distribution** (the le-contract distance-vs-compose split, +`:163-167`): + +| | **256 (single 8-bit) = ranking** | **256² (2×8bit pair) = distribution** | +|---|---|---| +| shape | one monotone axis | the full joint pairwise | +| accuracy | the cosine axis only — **REQUIRES the cosine-replacement** (Fisher-z: cosine→`atanh`→i8; a raw byte is not cosine) | **as accurate as content-blind f32** — the pair recovers full f32 fidelity at 8-bit (the claim P-PQ-RANK certifies) | +| read | `\|Δi8\|·(z_range/254)`, table-free | the semiring COMPOSE table (kept — "z-addition does not compose cosines", `:167`) | +| answers | "how similar / what order" | "how it relates / composes / transitions" | +| a word/doc | a scalar position | its **relational profile = its distribution** | + +**This is a MEASUREMENT, not a judgment — and that is the point.** COCA × +6×256:256 is **deterministic** (integer/table, bit-reproducible, 0 learned +params, <10 µs/sentence; an LLM is stochastic even at T=0), **agnostic** as a +*fixed, transparent, empirical* basis (vs an LLM's opaque, RLHF-shaped, +version-drifting one), and **auditable/falsifiable** — a fixed inspectable table +you can *certify* with the jc battery (ρ≥0.999). Determinism is the +**precondition** for the probe-first spine: you cannot run a reproducible +falsification against a stochastic oracle. The architecture therefore **demotes +the LLM to the escalation tail** (the "<25 %" the stack routes to an oracle only +when measurement runs out); everything in the hot loop is a certifiable +measurement. + +**The rung ascent IS ranking → distribution.** Rung 0–1 = 256-ranking; rung 2+ = +256²-distribution (SPO-G compose, PPR, community). BLOCK/surprise escalates from +"sort by similarity" to "reason over the relational field." **Exactness escape +hatch:** the bgz-tensor **index+residual ladder** (`adaptive_codec.rs`, +`:189-207`) — centroid PLACES, a Hadamard residual CORRECTS in a 3-tier LFD +split, hardest ~10 % → `Passthrough` (exact vector). + +**DeepNSM is the no-LLM tokenizer + SPO sensor — an UPSTREAM leg.** text → +COCA-4096 tokenize → PoS-FSM → 36-bit SPO → ±5 VSA context (`deepnsm/pipeline.rs`). +**Crate-layering constraint (mechanical, not semantic):** AriGraph core cannot +Cargo-depend on the `deepnsm` crate (downstream). So `community.rs`/`retrieval.rs` +consume the **in-core `crates/lance-graph/src/nsm/` copy** or receive SPO +produced upstream (osint) — the *meaning* (COCA) is core; only the *sensor +packaging* is downstream. DeepNSM's `BasinClassification` (`episodic_spo.rs:216`) +is a per-sentence basin proposer — the design reference for §3b's +community/basin cross-validation. + +## §3b. Leiden community synergies (why AriGraph, not a crate) + +Communities are the **structural partition** completing the partitions AriGraph +already carries, and they are **distributional-meaning modes, NARS-truth-weighted** +— not agnostic clustering of opaque ranks. Six synergies: + +1. **× episodic/family basins — orthogonal partitions that cross-validate.** The + basin = `part_of:is_a` rail / `EpisodicEdges64` family (`family==0` + intra-basin) — the *inherited/experiential* grouping; a community = the + *computed structural* grouping. Coincide ⇒ high-confidence structure; cross ⇒ + a discovered bridge; basin-without-community ⇒ a revision candidate. A NEW + partition beside the family rail, never a re-carving of it (measure agreement: + P-COMMUNITY-BASIN-AGREE). +2. **× the 256² distribution — Leiden clusters the distribution, not the + ranking.** Modularity runs over the compose/distribution graph (256² + palette-compose + `CausalEdge64` SPO edges), NOT the 256 rank. A community IS + a mode of the relational field — inexpressible as a scalar rank; this is why + §3a's pairwise is load-bearing. +3. **× COCA meaning / NARS truth — communities = topics, truth-weighted, no + LLM.** A word/doc's meaning IS its distribution (its 256²/4096² row); Leiden + over it = semantic communities = topics — the **no-LLM community-summary** leg + (vs graphrag-rs's LLM summaries). Edges are NARS-weighted (`community.rs` uses + `truth.confidence`; expectation-weighting is the refinement) — a community is + a *truth-weighted meaning mode*, COCA + NARS together. +4. **× the rung ascent — community IS the rung-3 layer.** 0–1 ranking → 2 SPO-hop + → **3 community-scoped expansion** (community bounds the PPR restart set; the + summary is the coarse answer) → 4 `intervene_on` apex. `detect_contradictions` + → BLOCK widens the community scope. +5. **× PPR — community-scoped personalized PageRank.** Restart-within-community = + HippoRAG passage-community scoping: cheaper (bounded) + more relevant. Community + picks the subgraph; PPR ranks within it. +6. **× the witness/KV/document layer — communities of DOCUMENTS.** Documents are + witnessed nodes (DocumentID handles, §4a). Leiden over the document-graph = + document communities = topic clusters — the "documents in this community" seam + `ogar-doc`'s reconstruct/related-docs path consumes. + +**Hierarchical Leiden ↔ the existing hierarchy.** Leiden is hierarchical; the +substrate is already hierarchical (256 = 4⁴ centroid tree; HHTL cascade; +`part_of:is_a` taxonomy). A super-community SHOULD align with a coarse HHTL tier +/ taxonomy parent — **register and measure agreement** (P-HIER-LEIDEN-HHTL), not +bolt on a foreign hierarchy; disagreement = discovered structure. **The LFD-hard +tier = community boundaries** — the ~10 % Passthrough nodes are the bridges where +assignment is ambiguous; the exactness ladder and the community geometry are the +same object seen twice. + +**Shipped (D-GR-3a):** `arigraph/community.rs` — multi-level Louvain over the +`TripletGraph` adjacency (NARS-confidence-weighted), the carrier method +`TripletGraph::communities()`, deterministic (BTreeMap-ordered moves, +sorted-entity index), 5 inline tests (two-triangle→2, clique→1, determinism, +empty-safe, weighted-cohesion). The Leiden *refinement* pass (well-connected +communities) is the next increment; PPR + community fusion into `retrieval.rs` +is D-GR-3b. + +## §4. Topology (v1.1 — expand AriGraph, no new crate) + +``` +crates/lance-graph/src/graph/arigraph/ (EXPAND — the graph owns its own structure) + community.rs # SHIPPED (D-GR-3a): TripletGraph::communities() — multi-level Louvain + # over the triplet adjacency (NARS-confidence-weighted), deterministic, + # 5 tests. The structural partition beside witness_corpus/episodic basins. + # Next: Leiden refinement pass; PPR/community fusion (D-GR-3b). + retrieval.rs # EXTEND OsintRetriever: + PPR/HippoRAG (reset-distribution atop + # blasgraph::hdr_pagerank), + community as a THIRD fusion signal + # (beside BFS + episodic). Driven by the #708 RungElevator: + # detect_contradictions → BLOCK → wider community/PPR walk. + # reuse in place: triplet_graph.rs (get_associated/intervene_on/detect_contradictions/ + # revise_with_evidence), spo_bridge.rs (promote_to_spo), witness_corpus.rs, + # episodic.rs, markov_soa.rs. NO new Edge/Provenance/Graph type. + +crates/lance-graph-contract/ (EXTEND, zero-dep) + src/doc_graph.rs # DocGraphQuery trait (impl = AriGraph's retrieval methods) — the + # read surface ogar-doc consumes. NO DocGraphView duplicate carrier. + + (BM25 lexical — NOT in AriGraph) + a tiny keyword module (or reuse an existing text index); text-index ≠ fact graph. +``` + +**Dep note (v1.1 — the feature-gate is DISSOLVED):** the community/PPR code lives +in `arigraph/` = lance-graph **core**, which already has the graph + datafusion/ +lance/arrow. There is no separate light crate to gate — the graph capabilities +live where the graph lives. The only zero-dep additions are the `DocGraphQuery` +trait (contract) and the BM25 leg. The `RungElevator` (contract, #708) is reached +from `retrieval.rs` in-core; `rung_widened_layer_mask` (still private, +`driver.rs:701`) is either made pub / moved to the contract, or replicated (§10). + +`ogar-doc` (OGAR) consumes `DocGraphQuery` **via the contract** — the correct +"feeds ogar-doc": ogar-doc is a **caller** of AriGraph's query surface (through +the contract trait), never fed data by a lance-graph crate. + +## §4a. The document-identity KV seam — witness holds a handle, never raw + +Operator threads (v1.2): a separate KV so the episodic witness doesn't carry raw +data, keyed by DocumentPath/DocumentID, related to `ogar-doc`/tesseract-rs +rendering over ClassView×WideFieldMask. Grounded: + +- **DocumentID/DocumentPath is NEW and the RIGHT key.** It doesn't exist yet (no + KV table; no `DocumentPort` in tesseract-rs). Do NOT reuse `content_sha256` + (per-acquisition DEDUP key — scan vs HTML of the same invoice differ, + `ogar-doc-ir/lib.rs:308`) or expose `kv_key` (storage-opaque). A stable + DocumentID gives the **witness ↔ node ↔ KV** triangle one handle independent of + retina and blob store — make the raw-ref's `kv_key` field CARRY the DocumentID. +- **The three-tier handle chain (no raw embedding):** (1) AriGraph **witness** = + `WitnessEntry{mailbox_ref, spo_fact_ref}` (hot path) — already **handle-only** + ✅; (2) **`document 0x080B` node value** = raw-ref `{sha256 digest, + kv_key=DocumentID, mime, counts}` — the SINGLE place DocumentID↔kv_key lives, + calcifies to Lance as SoA value bits; (3) **consumer KV** keyed by DocumentID = + the actual bytes/raster (S3 / MedCare `file_filelist` / a *passive* Lance blob + column). The render leg **late-binds** the raster only at render time. +- **The change your KV idea targets is real + localized (D-GR-6).** The hot-path + witness is already raw-free; but the **cold-path `witness_corpus::WitnessEntry` + embeds `evidence_blob: Bytes`** and **`episodic::Episode` embeds `observation: + String`** — replace those with a DocumentID handle → KV. A scoped AriGraph + change, not a rewrite. +- **Boundary + rendering:** the KV is **consumer/blob-store side, never a Lance + ingest endpoint**. OGAR assembles; lance calcifies handles; tesseract-rs stays + untouched (`doc.v1` seam). Rendering reuses the shipped `ClassView × + WideFieldMask` brick (`render_field_view` / `render_class_with_methods_wide` / + a2ui `project_node`; PDF = the unbuilt `ogar-doc reconstruct_document`), RBAC + `template ∩ role` fail-closed. **Sequencing:** the `0x080B`/`0x080A` mints + + persist/reconstruct are BLUEPRINT-ONLY, council/mint-gated — D-GR-5/6 sequence + AFTER the doc-W4 council. + +## §5. Wave sequencing (aligned to v3 W0–W6 + D-TRI) + +Neither crate is a new W-wave. Sequenced **after the W1 keystone** +(`mailbox_owner()` shipped #631; batch-writer W1b in-PR): + +- **G0 — baseline probe (FIRST, no code): P-GRAPH-LOADBEARING** (§6). Gate on + truth-architect / measurement-before-synthesis. +- **D-GR-3a — SHIPPED** — `arigraph/community.rs` (`TripletGraph::communities()`, + multi-level Louvain, deterministic, 5 tests). Landed ahead of G0 as a *pure, + reversible* capability with no write path — it computes a partition, gates + nothing. G0 still gates whether it is *wired into retrieval*. +- **D-GR-1** — `DocGraphQuery` trait in `lance-graph-contract` (impl = AriGraph's + retrieval methods). Zero SoA writes. +- **D-GR-2** — extend `arigraph/retrieval.rs` to bind **existing** CAM-PQ + SPO-G + hops onto the canonical `RungLevel`/`RungElevator` (**#708 merged `8d3209c`**; + advance-before-sinks ordering per `17368ea`). Mirrors the #708 settlement probe + (BLOCK ascends → wider walk; FLOW at base). +- **D-GR-3b** — the remaining BUILD gaps: **Leiden refinement** (well-connected + communities) on `community.rs`; **`ppr.rs` HippoRAG** (reset-distribution atop + `blasgraph::hdr_pagerank`) + **community-scoped PPR** fused into `retrieval.rs` + as a third signal beside BFS+episodic; **BM25** (small, out-of-graph). Each + gated on G0 beating vector-only. +- **D-GR-4** — community summaries via the Rig oracle → compiled template (W3 + `template-runtime`). No-LLM path preferred (DeepNSM distributional summary); + LLM only at the escalation tail. +- **D-GR-5** — wire `ogar-doc` `reconstruct_document` + "documents in this + community" to `DocGraphQuery` (cross-repo; baton-audited; AFTER doc-W4 council). +- **D-GR-6** — the witness-KV separation (§4a): replace the **cold-path** + `witness_corpus::WitnessEntry.evidence_blob` + `episodic::Episode.observation` + raw embeds with a **DocumentID handle → consumer KV**. Scoped AriGraph change; + born-stamped if it persists; AFTER doc-W4 council (DocumentID is the raw-ref key). + +Depends-on: W1b (any persisted result is born-stamped), W2 (retrieval cycles = +kanban lanes if a cycle persists), W3 (summaries = compiled templates). Composes +**atop** `oxigraph-arigraph-cognitive-shader-soa-merge-v1` (that plan builds the +unified SoA context; graphrag is the retrieval layer over it — §8). + +## §6. Falsifiers / probes (probe-first) + +- **P-GRAPH-LOADBEARING (G0, the gate).** On a document corpus, measure retrieval + quality **with vs without** graph traversal (vector-only vs vector+SPO-G+PPR). + KILL condition: if the graph does not beat vector-only on multi-hop/global + questions, do **not** build Leiden/PPR — the graph would be decorative + (graph-librarian-rs anti-pattern). Mirrors the 1BRC addressing-tax probe. +- **P-RUNG-RETRIEVAL.** Hard/contradictory query → BLOCK → elevator ascends → + wider CAUSES..BECOMES mask → higher recall; easy query → FLOW → stays at base + (identity mask, cheap). Mirrors #708's D-TRI-6 settlement probe. +- **P-PPR-MULTIHOP.** HippoRAG-PPR beats vector-only on 2+-hop questions. +- **P-COMMUNITY-GLOBAL.** Leiden community summaries beat flat chunk-concat on + global/thematic questions (the LightRAG dual-level claim). +- **P-COMMUNITY-BASIN-AGREE (new, v1.2).** Leiden communities vs the + `part_of:is_a`/`EpisodicEdges64` family basins — measure agreement + (jc::reliability). High agreement ⇒ structure confirms experience; the + disagreements are the discovered bridges / revision candidates (§3b.1). +- **P-HIER-LEIDEN-HHTL (new, v1.2).** Hierarchical Leiden super-communities vs the + coarse HHTL tiers / taxonomy parents — measure agreement; disagreement = + structure the taxonomy lacks (§3b hierarchy note). +- **P-MEASUREMENT-DETERMINISM (new, v1.2).** `community.rs` (and every base read) + is bit-reproducible: same graph → identical partition across runs/machines + (the property that makes the jc certifications *possible*, and that an + LLM-based clustering cannot offer). Asserted by the `deterministic` unit test; + the substrate-wide claim of §3a. +- **P-PQ-RANK (new, v1.2).** Spearman ρ of the `6×256:256` node-to-node distance + vs the `4096²` reference on COCA pairs (jc::reliability). ρ≥0.99 ⇒ 256 centroids + suffice for the base; else escalate hard pairs to the index+residual / + Passthrough tier (§3a). +- **jc battery** (ICC / Spearman / Cronbach) before any new lane-reading backs a + claim (v3 standing gate). **Landed #709** as `jc::reliability::{pearson, + spearman, cronbach_alpha, icc(IccForm)}` — every probe above computes its + with-vs-without delta + significance through this crate, not a hand-rolled + metric (and per `I-NOISE-FLOOR-JIRAK`, cites Jirak's weak-dependence rate for + σ-claims). + +## §7. Alignment hazards (and how this plan avoids each) + +1. **write-on-behalf / born-stamped (W1 keystone).** graphrag is read-mostly; the + only writes (persisted community membership / summaries) route the batch-writer + `cast(on_behalf = envelope.mailbox_owner(), …)` — never write-as-self (the smb + `LanceConnector::upsert` ORPHAN-WRITE is the cautionary tale). v3-mailbox-warden + gate. +2. **No-singleton.** No global `Arc>`. Community/PPR operate + over `MailboxSoA` rows + AriGraph; results are ephemeral or calcified + per-mailbox. graphrag-rs `InferenceEngine` (graph-as-config-struct) is the + explicit anti-exhibit. +3. **16-byte facet never-widen + classid canon-high.** Reuse existing task + classids (`0x080B`/`0x0807`/`0x0808`/`0x080A`, hi-u16 = concept, canon-high). A + persisted community-membership lane = a **new `ValueTenant` discriminant in the + 328 B headroom** (additive-at-end, `ENVELOPE_LAYOUT_VERSION` unchanged), + field-isolation-matrix gated by v3-envelope-auditor. `u8:u8` never widened to + u16/u24. +4. **Probe-first / wire-don't-invent.** COMPONENT-MAP + `graphrag-rs-inventory.md` + ARE the precomputed "does it exist" search; §2 leans on them. G0 gates code. +5. **CausalEdge64 duplication + cross-repo baton.** Always qualify + `causal_edge::CausalEdge64`. Any OGAR classid for the D-GR-5 seam is a + **batched** mint + sync-fuse, baton-handoff-auditor gated. +6. **Meaning-substrate, NOT meaning-agnostic (v1.2 correction).** Do NOT + reintroduce the "keep language out of the agnostic graph" framing. COCA + distributional meaning is a **core faculty** (co-equal with NARS); the + `markov_soa` hot loop is codebook-*parametric* for cross-domain reuse, and for + language content the injected COCA distance is correct, not a "language lens." + Agnosticism = raw data + consumer modelling only (§3a). Only the *crate + layering* is a real constraint: AriGraph core can't Cargo-dep the downstream + `deepnsm` crate — use the in-core `nsm/` copy. (The shipped + `markov_soa.rs:16,28` comment is a correction target, not a rule to obey.) + +## §8. Relationship to existing plans + +- **`graphrag-rs-inventory.md`** (`E-V3-GRAPHRAG-INV-1`) — the component-by-component + automataIA/graphrag-rs audit. **Mandatory pre-read**; this plan does not re-fork. +- **`oxigraph-arigraph-cognitive-shader-soa-merge-v1`** — builds the unified + SoA context (Oxigraph RDF + AriGraph episodes + SPO). graphrag is the + **retrieval layer atop it**; they compose, not compete. +- **`entropy-ladder-spo-rung-v1`** (R1 shipped) — SPO/NARS rung decomposition; + the rung mapping in §3 must reconcile with its entropy-level = fact-position. +- **`temporal-markov-and-style-classes-v1`** (ACTIVE) — governs the retrieval + substrate (episodic axis = Lance versions); graphrag rides `temporal.rs`. +- **`triangle-tenants-gestalt-separation-v1`** (D-TRI-1..6) + **#708** — the + Maslow pyramid graphrag retrieval ascends (§3). +- **`normalized-entity-holy-grail-v1`**, **`episodic-risc-spine-v1`**, + **`cam-pq-production-wiring-v1`**, **`wikidata-lazy-spine-hydration-v1`** — + adjacent; entity-resolution + episodic-addressing + codec + KG-hydration inputs. + +## §9. Board hygiene · D-ids · gate + +- D-ids: **D-GR-1..5** (this plan) — STATUS_BOARD rows on land. +- New plan → INTEGRATION_PLANS.md prepend (this commit); EPIPHANIES entry only + when a finding lands (probe result), not for the plan itself. +- **Gate:** G0 (P-GRAPH-LOADBEARING) is the first deliverable — **no Leiden/PPR + code before it is green.** D-GR-5 (OGAR seam) is mint-gated + baton-audited. +- **Open questions:** (O1) does `ogar-doc` `reconstruct_document` genuinely need a + runtime graph query, or is the "updated knowledge" re-issue a batch re-persist? + (settles whether D-GR-5 is a live query seam or a re-run). (O2) materialize the + `EpisodicWitness64` column now (helps PPR recency) or stay on + `EpisodicEdges64`+`WitnessTable`? (probe). (O3) reconcile the §3 rung mapping + with `entropy-ladder-spo-rung-v1`'s entropy levels — same ladder or two? + +## §10. Feasibility — verified against the code (`8d3209c`, #708 merged) + +Checked every reuse/build assumption against the tree. **Verdict: FEASIBLE.** +Load-bearing legs reach from **zero-dep** crates; two small caveats, both with +clean fixes; **no circular deps**. + +**Confirmed reachable (pub, verified):** +- **Rung ascent (the #708 core) — zero upstream change.** `RungElevator` is a + pub struct with **pub fields** + `const fn new(base)` + `on_gate(&mut self, + GateDecision) -> RungLevel` (pure transition), all in **zero-dep** + `lance-graph-contract::cognitive_shader` (`:272`). graphrag holds its own + elevator; retrieval-surprise → `GateDecision` → `on_gate` → `RungLevel`. +- **SPO edge + NARS truth — zero-dep** (`causal_edge::CausalEdge64`). +- **Read surface — zero-dep** (`MailboxSoaView`, contract `soa_view.rs:42`). +- **Fact-store traversal — richer than assumed (closes the §9 arigraph + question).** `TripletGraph` (arigraph) exposes, all pub: `get_associated( + entities, steps)` (`:141`, the multi-hop SPO-G walk), `find_path` (`:193`), + `intervene_on` (`:714`, Pearl-2 apex), `infer_deductions` (`:755`), + **`detect_contradictions(conf)` (`:805` — the natural BLOCK trigger)**, + `revise_with_evidence` (`:829`), `with_truth`. Plus `spo_bridge::promote_to_spo` + (`:110`). The rung loop maps 1:1: `detect_contradictions` → BLOCK → `on_gate` + ascends → `get_associated` widens the hop → `intervene_on` at the apex. +- **PPR base — pub** (`hdr_pagerank`, `blasgraph/ops.rs:275`; `ScentCsr::spmv`, + `neighborhood/sparse.rs:98`). **CAM-PQ — pub** (`cam_pq/{ivf,storage}`). + +**Caveat 1 — ~~dep-weight split~~ DISSOLVED by v1.1 (expand AriGraph).** Original +finding: the graph legs (`TripletGraph`, `spo_bridge`, `hdr_pagerank`, CAM-PQ) +live in lance-graph core (drags datafusion/lance/arrow), so a standalone crate +would need a light/heavy feature gate. **v1.1 removes the crate** — Leiden/PPR +land as `arigraph/` modules, which are already in core with the graph, so there +is nothing to gate. The only zero-dep additions are the `DocGraphQuery` contract +trait and the BM25 leg. **New design constraint (replaces the gate concern):** +`no-singleton` + `write-on-behalf` — community detection reads the graph the +mailbox owns and must NOT materialize a *global* partition singleton; a persisted +community-membership is a value-tenant lane, **born-stamped** via the batch +writer (v3-mailbox-warden gate). Confirm `TripletGraph`'s ownership model before +adding a persisting method. + +**Caveat 2 — `rung_widened_layer_mask` is private** (`driver.rs:701`, bare `fn`) +— the predicate-plane widen can't be called directly. It is a SECONDARY leg (the +rung ASCENT works without it). Fix (small): promote it `pub` / move it to the +contract beside `RungElevator` (~1-line upstream tweak, baton to the +cognitive-shader-driver owner) — recommended; or replicate the pure +`(base, level, mask) -> u8`. + +**Genuine BUILD (confirmed):** hierarchical Leiden — `jc`'s louvain is +**example-only** (nothing in `jc/src`), so `arigraph/community.rs` promotes it to +a lib over the `TripletGraph` adjacency; HippoRAG-PPR = reset-distribution atop +`hdr_pagerank`, added to `arigraph/retrieval.rs`; BM25 = a small out-of-graph fn. +All build on reachable primitives. AriGraph fit confirmed: it owns `retrieval.rs` ++ `witness_corpus.rs` + `episodic.rs` + `spo_bridge.rs` + `markov_soa.rs` today, +and has **no** community/partition type — Leiden fills the gap, does not clutter. + +**Next step (v1.2):** `arigraph/community.rs` is **DONE (D-GR-3a)** — +`TripletGraph::communities()`, multi-level Louvain, deterministic, 5 tests +(Louvain core verified standalone before the core build). Remaining, in order: +(1) `ppr.rs` HippoRAG + community-scoped PPR fused into `arigraph/retrieval.rs` +under the #708 `RungElevator` (D-GR-3b); (2) the `DocGraphQuery` contract trait +(D-GR-1); (3) the G0 `P-GRAPH-LOADBEARING` probe + P-PQ-RANK / P-COMMUNITY-BASIN-AGREE +(jc::reliability). No new crate; no upstream change for (1)-(3). diff --git a/crates/lance-graph/src/graph/arigraph/community.rs b/crates/lance-graph/src/graph/arigraph/community.rs new file mode 100644 index 00000000..527701b7 --- /dev/null +++ b/crates/lance-graph/src/graph/arigraph/community.rs @@ -0,0 +1,395 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! `community` — structural community detection over the AriGraph +//! [`TripletGraph`](super::triplet_graph::TripletGraph). +//! +//! # Where this sits (doctrine) +//! +//! A **community** is the *structural* partition of the fact graph — the +//! densely-connected regions of the relational field. It **complements**, and +//! does not duplicate, the *episodic / family* basins AriGraph already carries +//! (the `part_of:is_a` rail concept, realised by +//! [`EpisodicEdges64`](lance_graph_contract::episodic_edges) and the +//! `witness_corpus`). Two orthogonal partitions that cross-validate: +//! +//! - a community that **coincides** with a family basin = high-confidence structure; +//! - a community that **crosses** basins = a discovered bridge the episodic +//! history has not yet captured; +//! - a basin with **no** community = experientially grouped but not structurally +//! coherent (a revision candidate). +//! +//! Community detection reads the **distributional-meaning field**, not the 1-D +//! similarity ranking: modularity is optimised over the *relational* graph +//! (SPO edges, NARS-truth-weighted), so a community is a **mode of the +//! distribution** — exactly what a scalar rank cannot express. This is the +//! rung-3 layer of the ascent (observation/ranking → SPO hop → community/PPR). +//! +//! Per the workspace litmus (*method on the carrier, not a free function on its +//! state*), the entry point is [`TripletGraph::communities`], not an external +//! service over the graph. +//! +//! # Algorithm +//! +//! Multi-level **Louvain** modularity (Blondel et al. 2008): local-moving to a +//! modularity fixed point, then community aggregation, repeated — yielding a +//! hierarchy. The ΔQ gain uses the canonical `2·m²` denominator (matching +//! `jc/examples/splat_louvain_modularity.rs`). Deterministic: nodes iterate in +//! index order and candidate communities in `BTreeMap` order, so the same graph +//! always yields the same partition — a property an LLM-based clustering cannot +//! offer, and the precondition for certifying it with the jc reliability +//! battery. The Leiden *refinement* pass (guaranteeing internally-connected +//! communities) is the next increment; Louvain is the core it refines. + +use std::collections::{BTreeMap, HashMap}; + +use super::triplet_graph::TripletGraph; + +/// A hierarchical community assignment over a [`TripletGraph`]'s entities. +#[derive(Debug, Clone)] +pub struct Communities { + /// Dense node id → entity name (the inverse of the internal index). + pub entities: Vec, + /// Community label per entity at the **coarsest** (final) level. + pub labels: Vec, + /// Hierarchy: `levels[l][node]` = the node's community at level `l` + /// (level 0 = finest). `levels.last()` equals [`Self::labels`]. + pub levels: Vec>, + /// Modularity `Q` of the coarsest partition (in `[-0.5, 1.0]`). + pub modularity: f64, + /// Number of communities at the coarsest level. + pub num_communities: usize, +} + +impl Communities { + /// The community id of `entity`, if it is in the graph. + pub fn community_of(&self, entity: &str) -> Option { + self.entities + .iter() + .position(|e| e == entity) + .map(|i| self.labels[i]) + } + + /// The entity names in community `community` (coarsest level). + pub fn members(&self, community: u32) -> Vec<&str> { + self.entities + .iter() + .zip(self.labels.iter()) + .filter_map(|(e, &c)| if c == community { Some(e.as_str()) } else { None }) + .collect() + } +} + +impl TripletGraph { + /// Detect structural communities over this graph's entities via multi-level + /// Louvain modularity. Edges are the non-deleted triplets (subject↔object), + /// weighted by NARS **confidence**; self-loops (`subject == object`) are + /// dropped. Deterministic and dependency-free (reads the graph the mailbox + /// already owns; never a global partition singleton). + pub fn communities(&self) -> Communities { + let (entities, index) = self.entity_index_dense(); + let n = entities.len(); + if n == 0 { + return Communities { + entities, + labels: Vec::new(), + levels: Vec::new(), + modularity: 0.0, + num_communities: 0, + }; + } + let adj = self.build_adjacency(&index, n); + let (levels_u, labels_u, q) = detect(adj, vec![0.0; n]); + let labels: Vec = labels_u.iter().map(|&x| x as u32).collect(); + let levels: Vec> = levels_u + .iter() + .map(|lv| lv.iter().map(|&x| x as u32).collect()) + .collect(); + let num_communities = { + let mut s = labels.clone(); + s.sort_unstable(); + s.dedup(); + s.len() + }; + Communities { entities, labels, levels, modularity: q, num_communities } + } + + /// Dense entity index: a stable `Vec` (sorted for determinism) and + /// an entity → dense-id map. + fn entity_index_dense(&self) -> (Vec, HashMap) { + let mut names: Vec = self.entity_index.keys().cloned().collect(); + names.sort_unstable(); + let index: HashMap = + names.iter().enumerate().map(|(i, e)| (e.clone(), i)).collect(); + (names, index) + } + + /// Weighted undirected adjacency (both endpoints), weight = summed NARS + /// confidence over the triplets joining two entities. Deleted triplets and + /// self-loops are skipped. + fn build_adjacency( + &self, + index: &HashMap, + n: usize, + ) -> Vec> { + // Accumulate undirected weights in a per-node BTreeMap for determinism. + let mut acc: Vec> = vec![BTreeMap::new(); n]; + for t in &self.triplets { + if t.is_deleted() || t.subject == t.object { + continue; + } + let (Some(&u), Some(&v)) = (index.get(&t.subject), index.get(&t.object)) else { + continue; + }; + let w = t.truth.confidence.max(0.0) as f64; + if w <= 0.0 { + continue; + } + *acc[u].entry(v).or_insert(0.0) += w; + *acc[v].entry(u).or_insert(0.0) += w; + } + acc.into_iter().map(|m| m.into_iter().collect()).collect() + } +} + +// ── pure multi-level Louvain (std only; verified standalone) ──────────────── + +/// Weighted undirected graph: `adj[u]` = neighbours (`v != u`, weight `w`) on +/// both endpoints; `self_loop[u]` = self weight (aggregation introduces it). +struct WGraph { + adj: Vec>, + self_loop: Vec, + degree: Vec, + two_m: f64, +} + +fn build(adj: Vec>, self_loop: Vec) -> WGraph { + let n = adj.len(); + let mut degree = vec![0.0f64; n]; + for u in 0..n { + let mut d = 0.0; + for &(v, w) in &adj[u] { + if v != u { + d += w; + } + } + degree[u] = d + 2.0 * self_loop[u]; + } + let two_m: f64 = degree.iter().sum(); + WGraph { adj, self_loop, degree, two_m } +} + +/// First-appearance dense relabel — deterministic given node order. +fn relabel(label: &mut [usize]) { + let mut map: HashMap = HashMap::new(); + let mut next = 0usize; + for l in label.iter_mut() { + let nl = match map.get(l) { + Some(&x) => x, + None => { + let x = next; + next += 1; + map.insert(*l, x); + x + } + }; + *l = nl; + } +} + +/// Louvain local-moving to a modularity fixed point. Deterministic. +fn local_move(g: &WGraph) -> Vec { + let n = g.adj.len(); + let mut label: Vec = (0..n).collect(); + if g.two_m <= 0.0 { + return label; + } + let m = g.two_m / 2.0; + let two_m_sq = 2.0 * m * m; + let mut comm_deg: Vec = g.degree.clone(); + let mut improved = true; + let mut passes = 0; + while improved && passes < 100 { + improved = false; + passes += 1; + for u in 0..n { + let from = label[u]; + let k_u = g.degree[u]; + let mut ncw: BTreeMap = BTreeMap::new(); + for &(v, w) in &g.adj[u] { + if v == u { + continue; + } + *ncw.entry(label[v]).or_insert(0.0) += w; + } + let k_u_in_from = *ncw.get(&from).unwrap_or(&0.0); + comm_deg[from] -= k_u; // a_from - k_u + let gain = |to: usize, k_in: f64, comm_deg: &[f64]| -> f64 { + k_in / m - k_u * comm_deg[to] / two_m_sq + }; + let mut best = from; + let mut best_gain = gain(from, k_u_in_from, &comm_deg); + for (&c, &k_in) in &ncw { + let g_c = gain(c, k_in, &comm_deg); + if g_c > best_gain + 1e-12 { + best_gain = g_c; + best = c; + } + } + comm_deg[best] += k_u; + if best != from { + label[u] = best; + improved = true; + } + } + } + relabel(&mut label); + label +} + +/// Aggregate each community into a super-node. Preserves `two_m`. +fn aggregate(g: &WGraph, label: &[usize]) -> WGraph { + let k = label.iter().copied().max().map(|x| x + 1).unwrap_or(0); + let mut super_adj: Vec> = vec![BTreeMap::new(); k]; + let mut self_w = vec![0.0f64; k]; + for u in 0..g.adj.len() { + let cu = label[u]; + self_w[cu] += g.self_loop[u]; + for &(v, w) in &g.adj[u] { + if v == u { + continue; + } + let cv = label[v]; + if cu == cv { + self_w[cu] += w / 2.0; + } else { + *super_adj[cu].entry(cv).or_insert(0.0) += w; + } + } + } + let adj: Vec> = + super_adj.into_iter().map(|m| m.into_iter().collect()).collect(); + build(adj, self_w) +} + +/// Modularity `Q` of a partition on graph `g`. +fn modularity(g: &WGraph, label: &[usize]) -> f64 { + if g.two_m <= 0.0 { + return 0.0; + } + let k = label.iter().copied().max().map(|x| x + 1).unwrap_or(0); + let mut sigma_in = vec![0.0f64; k]; + let mut sigma_tot = vec![0.0f64; k]; + for u in 0..g.adj.len() { + let c = label[u]; + sigma_tot[c] += g.degree[u]; + sigma_in[c] += 2.0 * g.self_loop[u]; + for &(v, w) in &g.adj[u] { + if v != u && label[v] == c { + sigma_in[c] += w; + } + } + } + let two_m = g.two_m; + (0..k) + .map(|c| sigma_in[c] / two_m - (sigma_tot[c] / two_m).powi(2)) + .sum() +} + +/// Multi-level Louvain. Returns (hierarchy over ORIGINAL nodes, coarsest +/// labels, `Q` of the coarsest partition on the original graph). +fn detect( + adj0: Vec>, + self0: Vec, +) -> (Vec>, Vec, f64) { + let n0 = adj0.len(); + let g0 = build(adj0.clone(), self0.clone()); + let mut g = build(adj0, self0); + let mut levels: Vec> = Vec::new(); + let mut orig_to_super: Vec = (0..n0).collect(); + for _ in 0..64 { + let labels = local_move(&g); + let k = labels.iter().copied().max().map(|x| x + 1).unwrap_or(0); + let level: Vec = (0..n0).map(|i| labels[orig_to_super[i]]).collect(); + levels.push(level); + if k == g.adj.len() { + break; + } + for s in orig_to_super.iter_mut() { + *s = labels[*s]; + } + g = aggregate(&g, &labels); + } + let coarsest = levels.last().cloned().unwrap_or_else(|| (0..n0).collect()); + let q = modularity(&g0, &coarsest); + (levels, coarsest, q) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::graph::arigraph::triplet_graph::Triplet; + + fn tg(edges: &[(&str, &str)]) -> TripletGraph { + let mut g = TripletGraph::new(); + let ts: Vec = edges + .iter() + .enumerate() + .map(|(i, (s, o))| Triplet::new(s, o, "rel", i as u64)) + .collect(); + g.add_triplets(&ts); + g + } + + #[test] + fn two_triangles_bridge_yields_two_communities() { + // {a,b,c} triangle, {d,e,f} triangle, single a-d bridge. + let g = tg(&[ + ("a", "b"), ("b", "c"), ("c", "a"), + ("d", "e"), ("e", "f"), ("f", "d"), + ("a", "d"), + ]); + let c = g.communities(); + assert_eq!(c.num_communities, 2, "labels: {:?}", c.labels); + assert!(c.modularity > 0.3, "Q too low: {}", c.modularity); + assert_eq!(c.community_of("a"), c.community_of("b")); + assert_eq!(c.community_of("b"), c.community_of("c")); + assert_ne!(c.community_of("a"), c.community_of("d")); + assert_eq!(c.members(c.community_of("a").unwrap()).len(), 3); + } + + #[test] + fn deterministic() { + let g = tg(&[("a", "b"), ("b", "c"), ("c", "a"), ("d", "e"), ("e", "f"), ("f", "d"), ("a", "d")]); + assert_eq!(g.communities().labels, g.communities().labels); + } + + #[test] + fn clique_is_one_community() { + let g = tg(&[("a", "b"), ("a", "c"), ("a", "d"), ("b", "c"), ("b", "d"), ("c", "d")]); + assert_eq!(g.communities().num_communities, 1); + } + + #[test] + fn empty_graph_is_safe() { + let c = TripletGraph::new().communities(); + assert_eq!(c.num_communities, 0); + assert!(c.labels.is_empty()); + } + + #[test] + fn confidence_weight_keeps_strong_edges_together() { + // Strong a-b-c triangle (conf 1.0) + one weak c-d edge (conf 0.05). + let mut g = TripletGraph::new(); + let ts = vec![ + Triplet::new("a", "b", "rel", 0), + Triplet::new("b", "c", "rel", 1), + Triplet::new("c", "a", "rel", 2), + Triplet::with_truth("c", "d", "rel", crate::graph::spo::truth::TruthValue::new(1.0, 0.05), 3), + ]; + g.add_triplets(&ts); + let c = g.communities(); + assert_eq!(c.community_of("a"), c.community_of("b")); + assert_eq!(c.community_of("b"), c.community_of("c")); + } +} diff --git a/crates/lance-graph/src/graph/arigraph/mod.rs b/crates/lance-graph/src/graph/arigraph/mod.rs index 1bd94729..95220134 100644 --- a/crates/lance-graph/src/graph/arigraph/mod.rs +++ b/crates/lance-graph/src/graph/arigraph/mod.rs @@ -5,6 +5,7 @@ //! //! Transcoded from Python AriGraph — a memory architecture for LLM agents. +pub mod community; pub mod episodic; pub mod language; pub mod markov_soa; @@ -16,6 +17,7 @@ pub mod triplet_graph; pub mod witness_corpus; pub mod xai_client; +pub use community::Communities; pub use witness_corpus::{WitnessCorpus, WitnessEntry, WitnessId, WitnessIndexHashMap}; #[cfg(feature = "with-cam-pq")]