diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 4b96443c..1f4cb0a7 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,11 @@ +## 2026-07-17 — D-GR-1 + D-GR-3b + G0 shipped (3 background agents: 1 Opus filigree + 2 Sonnet grindwork; Opus orchestrator wired mod.rs + compiled centrally, one shared target) + +- **Task:** "Finish D-GR-*, Opus agents for filigree + Sonnet 5 for grindwork" — the graphrag plan's buildable/ungated deliverables. +- **Fleet:** 1 Opus → `contract/doc_graph.rs` (DocGraphQuery trait + D-GR-2 design), standalone rustc 9/9. 2 Sonnet (edit-only, no per-agent cold build) → `arigraph/ppr.rs` (PPR); `community.rs` Leiden `refine_connected` + `arigraph/bm25.rs`. Disjoint files, no write races; mod.rs wired centrally by the orchestrator. +- **Verified centrally (Opus, one shared target, PROTOC=/usr/bin/protoc):** `cargo test -p lance-graph --lib` 945 + 18 D-GR module tests green; `cargo test -p lance-graph-contract` 10 green; clippy `-D warnings` clean on all four new files (the 8 pre-existing warnings are `blasgraph/ndarray_bridge.rs` SIMD dead-code, not mine); G0 harness runs. +- **Operator sharpenings absorbed:** (a) cosine-replacement EXISTS — grep it (certified `bgz-tensor::fisher_z`/`helix`), don't build; popcount is the retired path. (b) part_of:is_a ≡ Leiden community ≡ episodic basin — one concept; P-COMMUNITY-BASIN-AGREE tests the identity. Plan §3a/§3b + EPIPHANIES `E-GRAPHRAG-DGR3B-1`. +- **Gate held:** all shipped capabilities are pure/reversible/no-write-path (land ahead of G0 like D-GR-3a #714); D-GR-2 retrieval WIRING stays gated on the G0 real-corpus verdict. Branch `claude/happy-hamilton-0azlw4` (fresh post-#714 main); PR pending. + ## 2026-07-17 — TD-PLANNER-STYLE-DEFAULT-DRIFT-1 PAID — fill the 5 planner default_modulation families from canonical (main thread, no subagents) - **Task:** pay the debt D-TSC-1b measured — the planner's 5 `_ => FieldModulation::default()` style families diverge from the canonical `UNIFIED_STYLES`≡`StyleParams` tables on resonance/fan_out/exploration. diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 4b505c48..75745ac6 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,13 @@ +## 2026-07-17 — E-GRAPHRAG-DGR3B-1 — D-GR-1 + D-GR-3b shipped (DocGraphQuery, PPR, Leiden refinement, BM25, G0 harness); two operator sharpenings — (a) the cosine-replacement distance ALREADY EXISTS (grep it, don't build it); (b) part_of:is_a category ≡ Leiden community ≡ episodic-witness basin ≡ HHTL family identity — one concept + +**Status:** SHIPPED (branch `claude/happy-hamilton-0azlw4`, fresh post-#714 main). `lance-graph` 945 lib + 18 D-GR module tests green; `lance-graph-contract` 10 green + clippy `-D warnings` exit 0; G0 harness runs. All D-GR-1/3b capabilities are pure/reversible/no-write-path — they land ahead of G0 exactly as D-GR-3a (#714) did; the D-GR-2 retrieval WIRING stays gated on the G0 real-corpus verdict. + +- **D-GR-1** `lance-graph-contract/src/doc_graph.rs` — zero-dep `DocGraphQuery` trait + `ScoredId`, rung→walk dispatch as a default method (9 tests); D-GR-2 design (OsintRetriever ↔ #708 RungElevator) embedded as its module-doc. +- **D-GR-3b** `arigraph/ppr.rs` (`personalized_pagerank`, HippoRAG, 6 tests), `community.rs` Leiden `refine_connected` (+2 tests), `arigraph/bm25.rs` (Okapi BM25, 5 tests). +- **G0** `examples/g0_graph_loadbearing.rs` — vector-only vs vector+PPR+community on a synthetic multi-hop fixture (gold `turbines`: vector-only rank 8/8 @ 0.0 vs graph rank 6 @ 0.146). SCAFFOLD, not the verdict — the real KILL/PASS needs a labeled corpus + jc::reliability. +- **Sharpening (a) — cosine-replacement EXISTS ("grep it, don't build it", operator).** The Fisher-Z `arctanh` cosine-replacement is CERTIFIED as `bgz-tensor::fisher_z::{FamilyGamma, Base17Fz}` (ρ≥0.999, 21 roles) + clean-room `helix::fisher_z` (place/residue 256-palette ladder, `crates/helix`) + `lance-graph-contract::distance`; map = `.claude/DISTANCE_METRIC_INVENTORY.md`. The stacked-6×256 distance RETIRES the `blasgraph/ndarray_bridge.rs` `hamming_*`/`popcount_*` dead-code (brute-force bit tally). Distance work = WIRING the certified primitive into the graph/PPR path (helix graduates per its KNOWLEDGE.md §Consolidation), NOT a new kernel. Process lesson: consult/grep before proposing to build; I slipped by drafting "build a kernel" prose — corrected in plan §3a. +- **Sharpening (b) — part_of:is_a ≡ community ≡ basin, one concept.** "Part of" presupposes a category to be part OF; that category IS the Leiden community = the episodic-witness basin = the HHTL family identity (same `6×(8:8)` register, L1 is_a rail). Community detection is CONSTITUTIVE of the is_a category. P-COMMUNITY-BASIN-AGREE tests an IDENTITY (agreement→1.0), not just correlation; disagreements = the bridges / revision candidates. Plan §3a/§3b.1 updated. + ## 2026-07-17 — E-THERMOMETER-ENCODE-OVERFLOW-1 — `thermometer_encode(1.0)` overflowed u8 (`1u8 << 8`) — `to_fingerprint()` panicked in debug for any full-scale modulation dim; pre-existing, surfaced by Codex on #713 **Status:** SHIPPED (`crates/lance-graph-planner/src/thinking/style.rs`, +2 tests, 220 planner lib green, clippy exit 0). A pre-existing latent bug the D-TSC-1b follow-through (E-PLANNER-STYLE-DRIFT-PAID-1) made more prevalent and Codex (P2, #713) caught. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index e412c2da..43e3d795 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,10 @@ +## 2026-07-17 — branch `claude/happy-hamilton-0azlw4` (post-#714) — D-GR-1 `contract::doc_graph::{DocGraphQuery, ScoredId}` + D-GR-3b AriGraph capabilities (PPR, Leiden refinement, BM25) + G0 harness + +### Current Contract Inventory — new entry +- `lance_graph_contract::doc_graph::{DocGraphQuery, ScoredId}` — zero-dep rung-aware document-graph read surface (D-GR-1). `DocGraphQuery`: `community_of` / `community_ids` / `community_members` / `neighbours` / `similar_by_ranking`, plus a provided `retrieve(seeds, RungLevel, top_k)` default carrying the rung→walk dispatch (0–1 ranking / 2 SPO-G hop / 3+ wider community-scoped walk). `ScoredId {id, score, depth}`. Impl target = AriGraph `OsintRetriever` (D-GR-2, gated on G0). 9 tests. Only in-crate dep is `cognitive_shader::RungLevel`. + +lance-graph core (non-contract, same PR): `graph::arigraph::{ppr::{PersonalizedPageRank, personalized_pagerank}, bm25::Bm25Index}`; `community::refine_connected` (Leiden connectivity); `examples/g0_graph_loadbearing.rs` (P-GRAPH-LOADBEARING scaffold). + # LATEST_STATE — What Just Shipped (read this FIRST) ## 2026-07-16 — branch `claude/x265-x266-plans-review-h9osnl` (v5, post-#702) — probe-wave verdicts on main diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index a7726ead..839b00c1 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,18 @@ +## graphrag-doc-retrieval-soa-integration-v1 — retrieval over AriGraph (expand-in-place, no new crate) + +Plan: `.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md` (v1.2). Pure/reversible capabilities land ahead of G0; the D-GR-2 retrieval WIRING is gated on the G0 real-corpus verdict. + +| D-id | Title | Repo | Status | Evidence | +|---|---|---|---|---| +| D-GR-1 | `DocGraphQuery` zero-dep contract trait + `ScoredId` (rung→walk dispatch) + D-GR-2 design | lance-graph | Shipped (this PR) — `doc_graph.rs`, 9 tests | plan §5 | +| D-GR-3a | `TripletGraph::communities()` multi-level Louvain, deterministic | lance-graph | Shipped (#714) | plan §3b | +| D-GR-3b | PPR (`personalized_pagerank`) + Leiden `refine_connected` + BM25 (`Bm25Index`) — pure capabilities | lance-graph | Shipped (this PR) — 13 tests | plan §3b, §5 | +| G0 | P-GRAPH-LOADBEARING harness (vector-only vs vector+PPR+community) | lance-graph | Harness shipped (this PR); real-corpus verdict OPEN | plan §5, §6 | +| D-GR-2 | Fuse CAM-PQ+SPO-G+PPR+community into `retrieval.rs` under the #708 RungElevator | lance-graph | Design done (in `doc_graph.rs` module-doc); impl GATED on G0 | plan §5 | +| D-GR-4 | Community summaries (no-LLM DeepNSM; Rig-oracle tail) | lance-graph | Deferred (W3-coupled) | plan §5 | +| D-GR-5 | `ogar-doc` reconstruct/related-docs → `DocGraphQuery` seam | lance-graph + OGAR | Deferred (mint-gated, doc-W4 council) | plan §5 | +| D-GR-6 | Witness-KV separation (DocumentID handle → consumer KV) | lance-graph | Deferred (doc-W4 council) | plan §4a, §5 | + ## triangle-tenants-gestalt-separation-v1 — triangle tenants, surface separation, chess quarantine Plan: `.claude/plans/triangle-tenants-gestalt-separation-v1.md`. Design shipped; ALL layout work mint-gated (rides the same batched mint as W2a BoardAggregates + Tasks-SoA task classid + chess classids). diff --git a/.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md b/.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md index aacf6428..0a1f7cc7 100644 --- a/.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md +++ b/.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md @@ -208,6 +208,26 @@ identity AND the centroid — one register, two lenses; this is *why* communitie | answers | "how similar / what order" | "how it relates / composes / transitions" | | a word/doc | a scalar position | its **relational profile = its distribution** | +**Distance is the stacked cosine-replacement, NOT popcount — and it ALREADY EXISTS +(operator, 2026-07-17: "grep it, don't build it").** The cosine-replacement is the +Fisher-Z `arctanh` map `z = ½(ln(1+s)−ln(1−s))`, shipped and **certified** as +`bgz-tensor::fisher_z::{FamilyGamma, Base17Fz}` (ρ≥0.999, 21 roles, the `arctanh→i8` +table), with the clean-room `helix::fisher_z::Similarity` + `helix`'s place/residue +256-palette ladder (`crates/helix/src/{fisher_z,placement,residue,distance,simd}.rs` ++ `KNOWLEDGE.md`) and the `lance-graph-contract::distance` surface; the map is +`.claude/DISTANCE_METRIC_INVENTORY.md`. A node-to-node distance is **6 of those +cosine-replacement rankings stacked** — one 256-axis per subspace, `atanh`→i8, +L1-metric-safe over the 256×256 LUT — assembled into the HHTL family identity +(`6×256`; the stacking IS the HHTL cascade / `bgz-tensor::hhtl_d`, §3b's hierarchy). +That *is* the CAM-PQ ADC, and it **retires the Hamming/popcount path**: +XOR-then-popcount (`blasgraph/ndarray_bridge.rs` `hamming_*`/`popcount_*`, +dead-code) is brute-force bit-counting — a coarse tally where the certified `atanh` +axis is cheaper and monotone-exact. So the distance work is **WIRING the existing +certified cosine-replacement into the graph/PPR distance path** (helix graduates +from clean-room per its KNOWLEDGE.md §Consolidation: re-export `bgz-tensor`'s +`FamilyGamma`), gated by the encoding-ecosystem naive-u8 floor (≥0.9980 Pearson) / +P-PQ-RANK — **not building a new kernel.** + **This is a MEASUREMENT, not a judgment — and that is the point.** COCA × 6×256:256 is **deterministic** (integer/table, bit-reproducible, 0 learned params, <10 µs/sentence; an LLM is stochastic even at T=0), **agnostic** as a @@ -250,6 +270,19 @@ already carries, and they are **distributional-meaning modes, NARS-truth-weighte a discovered bridge; basin-without-community ⇒ a revision candidate. A NEW partition beside the family rail, never a re-carving of it (measure agreement: P-COMMUNITY-BASIN-AGREE). + **Sharpening (operator, 2026-07-17) — the identity hypothesis.** "Part of" + presupposes a *category* to be part **of**, and that category IS the community + = the basin = the HHTL family identity (the same `6×(8:8)` register read as the + L1 `is_a` rail, §3a). The two partitions are *computed* differently (Leiden over + structure vs the experiential family rail) but may resolve to the **same + categories** — community detection is **constitutive** of the `is_a` category, + not a decoration on it: you detect the community/basin first, then `part_of` + points into it and membership **is** `is_a`. So P-COMMUNITY-BASIN-AGREE tests an + **identity**, not merely a correlation — community ≡ basin ⇒ agreement → 1.0; + high-but-<1.0 ⇒ correlated-but-distinct, and the disagreements are exactly the + bridges / revision candidates. Either way a finding; the identity is the + *hypothesis*, the probe is the *falsifier* (the "same concept at the same time" + claim made precise). 2. **× the 256² distribution — Leiden clusters the distribution, not the ranking.** Modularity runs over the compose/distribution graph (256² palette-compose + `CausalEdge64` SPO edges), NOT the 256 rank. A community IS @@ -286,9 +319,35 @@ same object seen twice. `TripletGraph` adjacency (NARS-confidence-weighted), the carrier method `TripletGraph::communities()`, deterministic (BTreeMap-ordered moves, sorted-entity index), 5 inline tests (two-triangle→2, clique→1, determinism, -empty-safe, weighted-cohesion). The Leiden *refinement* pass (well-connected -communities) is the next increment; PPR + community fusion into `retrieval.rs` -is D-GR-3b. +empty-safe, weighted-cohesion). + +**Shipped (D-GR-3b, this PR — all pure / reversible / no-write-path, so they +land ahead of G0 exactly as D-GR-3a did):** +- `arigraph/ppr.rs` — `TripletGraph::personalized_pagerank(seeds, damping, iters)`, + HippoRAG spread over the confidence-weighted graph, deterministic, unit-sum, + 6 tests (near-triangle-outranks-far, sum≈1, determinism, empty-safe, + unmatched-seed-fallback, seed-top-ranked). +- `community.rs` **Leiden connectivity refinement** (`refine_connected`) — splits + any internally-disconnected Louvain community into its connected components + (Leiden's guarantee), deterministic BFS over intra-community edges, +2 tests; + `labels` is now the refined coarsest partition, `levels.last()` the raw Louvain. +- `arigraph/bm25.rs` — Okapi BM25 lexical leg (`Bm25Index`, k1=1.2/b=0.75), + the rung-0/1 baseline beside CAM-PQ ranking, deterministic, 5 tests. +- **D-GR-1** `lance-graph-contract/src/doc_graph.rs` — the zero-dep `DocGraphQuery` + trait + `ScoredId`, carrying the rung→walk dispatch as a default method; 9 tests. + The D-GR-2 design (OsintRetriever ↔ #708 RungElevator) is embedded as its module-doc. +- **G0** `examples/g0_graph_loadbearing.rs` — the P-GRAPH-LOADBEARING harness: + vector-only (BM25) vs vector+PPR+community on a synthetic multi-hop fixture, + prints the with-vs-without delta (gold `turbines`: vector-only rank 8/8 @ 0.0, + graph rank 6 @ 0.146). Synthetic **scaffold**, not the verdict — the real + KILL/PASS needs a labeled corpus + jc::reliability. + +**Still gated on G0:** the **D-GR-2 wiring** — fusing CAM-PQ + SPO-G + PPR + +community into `retrieval.rs` under the RungElevator (design done, impl gated). +The cosine-replacement distance that retires popcount is a separate +P-PQ-RANK-gated **wiring** (the primitive already exists — certified +`bgz-tensor::fisher_z::{FamilyGamma, Base17Fz}` ρ≥0.999 + `helix`, §3a — route it +into the graph/PPR distance path; do not rebuild it). ## §4. Topology (v1.1 — expand AriGraph, no new crate) @@ -364,14 +423,18 @@ rendering over ClassView×WideFieldMask. Grounded: Neither crate is a new W-wave. Sequenced **after the W1 keystone** (`mailbox_owner()` shipped #631; batch-writer W1b in-PR): -- **G0 — baseline probe (FIRST, no code): P-GRAPH-LOADBEARING** (§6). Gate on - truth-architect / measurement-before-synthesis. +- **G0 — P-GRAPH-LOADBEARING — HARNESS SHIPPED** (`examples/g0_graph_loadbearing.rs`). + The with-vs-without measurement *mechanism* runs (synthetic multi-hop scaffold, + prints the delta); the real KILL/PASS **verdict** still needs a labeled corpus + + jc::reliability. Gate on truth-architect / measurement-before-synthesis — the + **D-GR-2 wiring stays blocked on the real-corpus verdict**, not on the scaffold. - **D-GR-3a — SHIPPED** — `arigraph/community.rs` (`TripletGraph::communities()`, multi-level Louvain, deterministic, 5 tests). Landed ahead of G0 as a *pure, reversible* capability with no write path — it computes a partition, gates nothing. G0 still gates whether it is *wired into retrieval*. -- **D-GR-1** — `DocGraphQuery` trait in `lance-graph-contract` (impl = AriGraph's - retrieval methods). Zero SoA writes. +- **D-GR-1 — SHIPPED** — `DocGraphQuery` trait + `ScoredId` in `lance-graph-contract` + (`doc_graph.rs`, zero-dep, 9 tests, rung→walk dispatch as a default method; the + D-GR-2 design lives in its module-doc). Zero SoA writes. - **D-GR-2** — extend `arigraph/retrieval.rs` to bind **existing** CAM-PQ + SPO-G hops onto the canonical `RungLevel`/`RungElevator` (**#708 merged `8d3209c`**; advance-before-sinks ordering per `17368ea`). Mirrors the #708 settlement probe diff --git a/crates/lance-graph-contract/src/doc_graph.rs b/crates/lance-graph-contract/src/doc_graph.rs new file mode 100644 index 00000000..6ee15c69 --- /dev/null +++ b/crates/lance-graph-contract/src/doc_graph.rs @@ -0,0 +1,536 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! `doc_graph` — the zero-dep **read surface** over the calcified +//! document / fact graph (D-GR-1). +//! +//! # Role — one query surface, two readers +//! +//! Per the graphrag integration plan (§0/§4a), "feeds both ogar-doc and +//! graphrag" is honestly a **single query/view surface** over the calcified +//! document-graph, consumed by two readers: +//! +//! - **graphrag retrieval** — the AriGraph carrier (`TripletGraph` + +//! `OsintRetriever`) *implements* this trait; and +//! - **OGAR `ogar-doc`** — `reconstruct_document` / "documents in this +//! community" *calls* this trait. +//! +//! Because `ogar-doc` reaches the graph **through this contract trait**, OGAR +//! keeps depending only on `lance-graph-contract` (never on the AriGraph impl): +//! the dependency direction stays consumer → contract, never the reverse. +//! **Neither reader ingests; both read.** +//! +//! # Litmus — a carrier method, not a service +//! +//! Every method takes `&self` and returns owned results — no `&mut self`, no +//! lifecycle, no config injection, no borrow into any graph. This is a trait an +//! AriGraph carrier *implements* (`impl DocGraphQuery for TripletGraph`), not a +//! service wrapped around the graph. It carries the **stateless** rung→walk +//! *mapping*; the **stateful** rung *elevator* (the #708 BLOCK/FLOW streak +//! machine) lives in the impl (D-GR-2, spec below), never in the contract. +//! +//! # Zero-dep +//! +//! std only. Ids are `String` (a `DocumentID` or an AriGraph entity name), +//! scores `f32`, community ids `u32`. **No lance-graph / arigraph types cross +//! this surface** — the contract is dependency-free. The one in-crate type used +//! is [`RungLevel`](crate::cognitive_shader::RungLevel) (also zero-dep, #708). +//! +//! # Usage (both readers) +//! +//! ```rust,ignore +//! use lance_graph_contract::cognitive_shader::RungLevel; +//! use lance_graph_contract::doc_graph::DocGraphQuery; +//! +//! // graphrag: rung-aware retrieval (the elevator picks the rung; see D-GR-2). +//! let seeds = vec!["acme_corp".to_string()]; +//! let hits = graph.retrieve(&seeds, RungLevel::Contextual, 20); +//! +//! // ogar-doc: "documents in this topic community" for reconstruct_document. +//! if let Some(topic) = graph.community_of("invoice_2026_07") { +//! let siblings = graph.community_members(topic); +//! } +//! ``` +//! +//! ════════════════════════════════════════════════════════════════════════ +//! # D-GR-2 design spec — binding `OsintRetriever` to the #708 `RungElevator` +//! ════════════════════════════════════════════════════════════════════════ +//! +//! **This is a SPEC. D-GR-2 code is a follow-up in +//! `crates/lance-graph/src/graph/arigraph/retrieval.rs`.** It records exactly +//! how the retriever composes *existing* methods under the #708 elevator so the +//! follow-up PR has no design latitude. +//! +//! ## Owner & elevator +//! +//! `OsintRetriever` (retrieval.rs) holds one +//! [`RungElevator`](crate::cognitive_shader::RungElevator) (#708, merged +//! `8d3209c`, `E-RUNG-ASCENT-WIRED-1`), constructed `RungElevator::new(base)` at +//! the dispatched base rung. graphrag **never re-decides the level** — it reads +//! the elevator's `RungLevel` and supplies the wider graph walk. This is the +//! anti-decorative-graph guarantee: the graph is load-bearing *because* BLOCK +//! ascends the elevator. +//! +//! **Carrier.** The full `DocGraphQuery` surface is implemented on the +//! **retriever** (`OsintRetriever`, extended to hold the graph + a CAM-PQ handle +//! + the small out-of-graph BM25 leg), NOT on `TripletGraph` alone: +//! `similar_by_ranking` needs the ranking leg `TripletGraph` does not own. The +//! graph-side methods (`community_*`, `neighbours`) delegate to `self.graph.*`. +//! +//! ## Per-cycle loop (advance-before-sinks, review fix `17368ea`) +//! +//! 1. **Compute the cycle `GateDecision`** (`crate::collapse_gate::GateDecision`): +//! - `TripletGraph::detect_contradictions(conf_threshold)` **non-empty** → +//! `GateDecision::BLOCK` — the natural BLOCK trigger (retrieval surprise / +//! conflicting facts, `triplet_graph.rs`). +//! - Low mean NARS `truth.expectation()` over the walk, or an empty / +//! low-cohesion result → also `BLOCK` (measurement ran out). +//! - Contradictions empty **and** mean expectation high **and** the result +//! is stable across the cycle → `GateDecision::FLOW_BUNDLE` (converged). +//! - else `GateDecision::HOLD`. +//! 2. **Advance FIRST:** `let rung = elevator.on_gate(gate);` — the `17368ea` +//! ordering lock: advance the elevator *before* the sinks, then gate the walk +//! on the **post-advance** rung. +//! 3. **Gate the walk on `rung`** via [`DocGraphQuery::retrieve`] (implemented +//! for the carrier), which dispatches breadth by rung — the Maslow ascent +//! (`triangle-tenants-gestalt-separation-v1` §3a): +//! +//! | Rung | Retrieval action | Existing method composed | Predicate mask | +//! |---|---|---|---| +//! | 0–1 (base, FLOW) | CAM-PQ vector + BM25 surface **ranking** | [`similar_by_ranking`](DocGraphQuery::similar_by_ranking) (impl = CAM-PQ ADC entry + the small out-of-graph BM25 leg) | identity | +//! | 2 | **SPO-G edge hop** | `TripletGraph::get_associated(seeds, 2)` → [`neighbours`](DocGraphQuery::neighbours) | widen (`elevator.causal_mask_bits()` = `0b011`, Pearl L2) | +//! | 3 | **community-scoped PPR** | `TripletGraph::communities()` (`Communities::{community_of, members}`) picks the subgraph, then the forthcoming `ppr.rs` (reset-distribution atop `blasgraph::hdr_pagerank` + `ScentCsr::spmv`) ranks within it | wider CAUSES..BECOMES union | +//! | 4 (apex) | **Pearl rung-2 intervention** | `TripletGraph::intervene_on(subject, predicate, new_object)` → `CounterfactualSpoG` (`ContextTag::Intervention` G-slot) | — | +//! +//! 4. **Revise:** after the outcome, `TripletGraph::revise_with_evidence(&obs)` +//! (NARS revision) updates truth and feeds the NEXT cycle's gate; +//! `infer_deductions()` is an optional mid-rung 2-hop signal. +//! 5. **Relax:** a FLOW streak (≥ `threshold`) relaxes `elevator.level` one rung +//! toward `base` (never below) — the walk narrows back to ranking; a sustained +//! BLOCK streak elevates → a wider walk. `RetrievalConfig.max_depth` becomes +//! **rung-derived** (rung 2 → depth 2, rung 3 → depth 3), replacing the fixed +//! `max_depth: 2`. +//! +//! ## Predicate-plane widen (SECONDARY leg) +//! +//! `elevator.causal_mask_bits()` (the P2/P3-certified S/P/O mask) feeds +//! `cognitive-shader-driver::driver::rung_widened_layer_mask(base, level, +//! req_mask) -> u8` (`driver.rs:701`, currently **private** → promote `pub` / +//! move beside `RungElevator`, or replicate the pure `(base, level, mask)->u8` +//! — §10 caveat 2). The rung ASCENT works without it; the mask only sharpens +//! which of the 8 predicate planes the walk reads. +//! +//! ## Settlement probe (P-RUNG-RETRIEVAL, §6, mirrors #708 D-TRI-6) +//! +//! Hard / contradictory query → `detect_contradictions` non-empty → BLOCK → +//! elevator ascends → wider mask + wider walk → higher recall. Easy query → +//! FLOW → stays at base (identity mask, cheap ranking). BLOCK ascends, FLOW +//! relaxes to base. + +use crate::cognitive_shader::RungLevel; + +/// A retrieval hit: an entity / document id, a relevance score, and the hop +/// depth it surfaced at. +/// +/// Fully owned (the `String` id) so the contract stays zero-dep — no borrow +/// into any graph, no lance-graph / arigraph type. +#[derive(Clone, Debug, PartialEq)] +pub struct ScoredId { + /// Entity or document id — a stable string handle (e.g. a `DocumentID` or + /// an AriGraph entity name). Never a lance-graph / arigraph type. + pub id: String, + /// Relevance score, **higher = more relevant** (a CAM-PQ similarity, a NARS + /// truth expectation, or a PPR mass, per the impl). Not normalised across + /// methods — compare only *within* one result list. + pub score: f32, + /// Hops from the nearest seed the hit surfaced at: `0` = a direct / + /// base-rank hit ([`DocGraphQuery::similar_by_ranking`]), `1..` = a + /// multi-hop graph inference. Provenance for the rung story — a deeper hit + /// is a weaker, more inferential match. + pub depth: u8, +} + +impl ScoredId { + /// A hit at an explicit hop `depth`. + pub fn new(id: impl Into, score: f32, depth: u8) -> Self { + Self { + id: id.into(), + score, + depth, + } + } + + /// A base / direct hit (`depth == 0`) — the ranking leg's shape. + pub fn base(id: impl Into, score: f32) -> Self { + Self::new(id, score, 0) + } +} + +/// A zero-dep read surface over the document / fact graph. +/// +/// Implemented by the AriGraph carrier (graphrag retrieval); consumed by OGAR +/// `ogar-doc` **through this contract** (never the impl). See the module docs +/// for the doctrine and the D-GR-2 elevator-binding spec. +/// +/// The five required methods are the retrieval **primitives**; the provided +/// [`retrieve`](DocGraphQuery::retrieve) composes them into the stateless +/// rung→walk *mapping*. Impls SHOULD override `retrieve` for the real +/// CAM-PQ / PPR path — the default is a transparent reference composition over +/// the primitives. +/// +/// The carrier is the **retriever** (`OsintRetriever`, which holds the graph + +/// the ranking leg), not `TripletGraph` alone — `similar_by_ranking` needs the +/// CAM-PQ / BM25 ranking `TripletGraph` does not own; the graph-side methods +/// delegate to `self.graph.*`. +/// +/// ```rust,ignore +/// use lance_graph_contract::doc_graph::{DocGraphQuery, ScoredId}; +/// +/// impl DocGraphQuery for OsintRetriever<'_> { +/// fn community_of(&self, id: &str) -> Option { +/// self.graph.communities().community_of(id) +/// } +/// fn neighbours(&self, seeds: &[String], hops: u8) -> Vec { +/// let set = seeds.iter().cloned().collect(); +/// self.graph.get_associated(&set, hops as usize).iter() +/// .map(|t| ScoredId::new(t.object.clone(), t.truth.expectation(), 1)) +/// .collect() +/// } +/// fn similar_by_ranking(&self, id: &str, top_k: usize) -> Vec { +/// // the CAM-PQ ADC / BM25 leg the retriever owns (not TripletGraph) +/// self.rank_similar(id, top_k) +/// } +/// // community_ids / community_members ... +/// } +/// ``` +pub trait DocGraphQuery { + /// The community id `id` belongs to (structural Louvain partition), or + /// `None` when the id is not in the graph. + fn community_of(&self, id: &str) -> Option; + + /// The distinct community ids at the coarsest partition level, ascending. + /// Lets a consumer (`ogar-doc`) enumerate topics — "for each community, + /// reconstruct" — without a separate handle. + fn community_ids(&self) -> Vec; + + /// The member ids of `community` (coarsest level). The "documents in this + /// community" seam `ogar-doc`'s related-docs / reconstruct path consumes. + fn community_members(&self, community: u32) -> Vec; + + /// Multi-hop structural neighbours reachable within `hops` from `seeds` — + /// the SPO-G walk (the impl composes `TripletGraph::get_associated`). Scored + /// by the impl (NARS truth expectation, hop proximity, …); `hops` is small + /// (1–4 typically). + fn neighbours(&self, seeds: &[String], hops: u8) -> Vec; + + /// Similar-by-ranking: the **256-ranking** leg (CAM-PQ / distributional + /// similarity) — no graph walk, the base rung. Best-first, at most `top_k`. + fn similar_by_ranking(&self, id: &str, top_k: usize) -> Vec; + + /// Rung-aware retrieve: the #708 `RungLevel` selects the walk **breadth**, + /// best-first, at most `top_k`. + /// + /// The default is the stateless rung→walk mapping (D-GR-2 minus the + /// elevator), composed from the primitives: + /// + /// - **rung 0–1** — ranking: the union of each seed's + /// [`similar_by_ranking`](DocGraphQuery::similar_by_ranking) (no walk). + /// - **rung 2** — the 2-hop [`neighbours`](DocGraphQuery::neighbours) walk + /// (SPO-G hop). + /// - **rung 3+** — community-scoped: a wider (`rung`-hop, capped) walk, + /// focused to the hits sharing a seed's community (community picks the + /// subgraph; the real PPR ranks within it — impls override for `ppr.rs`). + /// + /// Widening the underlying walk with the rung is superset-monotone; the + /// rung-3 community focus is a relevance refinement over that wider set + /// (cheaper + more relevant, HippoRAG passage-community scoping), not a + /// regression of it. + fn retrieve(&self, seeds: &[String], rung: RungLevel, top_k: usize) -> Vec { + let mut hits: Vec = match rung.as_u8() { + // Rung 0–1 (base, FLOW) — 256-ranking: union each seed's similar set. + 0..=1 => { + let mut acc: Vec = Vec::new(); + for s in seeds { + acc.extend(self.similar_by_ranking(s, top_k)); + } + acc + } + // Rung 2 — SPO-G hop: the 2-hop reachable set. + 2 => self.neighbours(seeds, 2), + // Rung 3+ — community-scoped expansion: a wider walk, then keep the + // hits that share a community with a seed (community picks the + // subgraph; the real PPR ranks within it — override for ppr.rs). + _ => { + let hops = rung.as_u8().min(6); + let wide = self.neighbours(seeds, hops); + let seed_comms: std::collections::BTreeSet = + seeds.iter().filter_map(|s| self.community_of(s)).collect(); + if seed_comms.is_empty() { + wide + } else { + wide.into_iter() + .filter(|h| { + self.community_of(&h.id) + .is_some_and(|c| seed_comms.contains(&c)) + }) + .collect() + } + } + }; + dedup_best(&mut hits); + hits.sort_by(|a, b| { + b.score + .partial_cmp(&a.score) + .unwrap_or(core::cmp::Ordering::Equal) + }); + hits.truncate(top_k); + hits + } +} + +/// Keep the highest-scoring entry per id (in place). Sorts by `(id asc, score +/// desc)` so the first of each id-run is its best, then drops the rest. +fn dedup_best(hits: &mut Vec) { + hits.sort_by(|a, b| { + a.id.cmp(&b.id).then( + b.score + .partial_cmp(&a.score) + .unwrap_or(core::cmp::Ordering::Equal), + ) + }); + hits.dedup_by(|a, b| a.id == b.id); +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::{HashMap, HashSet}; + + /// A tiny in-memory graph implementing the read surface — the "AriGraph + /// carrier" stand-in: adjacency + Louvain-style community labels + a + /// similarity (ranking) table. + struct MockDocGraph { + adj: HashMap>, + community: HashMap, + similar: HashMap>, + } + + impl DocGraphQuery for MockDocGraph { + fn community_of(&self, id: &str) -> Option { + self.community.get(id).copied() + } + + fn community_ids(&self) -> Vec { + let mut ids: Vec = self.community.values().copied().collect(); + ids.sort_unstable(); + ids.dedup(); + ids + } + + fn community_members(&self, community: u32) -> Vec { + let mut m: Vec = self + .community + .iter() + .filter_map(|(k, &c)| { + if c == community { + Some(k.clone()) + } else { + None + } + }) + .collect(); + m.sort(); + m + } + + fn neighbours(&self, seeds: &[String], hops: u8) -> Vec { + // Deterministic BFS mirror of get_associated; score = 1/(1+depth). + let mut seen: HashSet = seeds.iter().cloned().collect(); + let mut frontier: Vec = seeds.to_vec(); + let mut out: Vec = Vec::new(); + for d in 1..=hops { + let mut next: Vec = Vec::new(); + for f in &frontier { + if let Some(ns) = self.adj.get(f) { + for n in ns { + if seen.insert(n.clone()) { + out.push(ScoredId::new(n.clone(), 1.0 / (1.0 + d as f32), d)); + next.push(n.clone()); + } + } + } + } + frontier = next; + if frontier.is_empty() { + break; + } + } + out + } + + fn similar_by_ranking(&self, id: &str, top_k: usize) -> Vec { + let mut v: Vec = self + .similar + .get(id) + .map(|xs| { + xs.iter() + .map(|(o, s)| ScoredId::base(o.clone(), *s)) + .collect() + }) + .unwrap_or_default(); + v.sort_by(|a, b| { + b.score + .partial_cmp(&a.score) + .unwrap_or(core::cmp::Ordering::Equal) + }); + v.truncate(top_k); + v + } + } + + /// Two triangle communities {a,b,c}=0 and {d,e,f}=1 joined by a single a–d + /// bridge; a's similarity table lists a non-adjacent node `z` (the ranking + /// vs walk discriminator). + fn fixture() -> MockDocGraph { + let mut adj: HashMap> = HashMap::new(); + adj.insert("a".into(), vec!["b".into(), "c".into(), "d".into()]); + adj.insert("b".into(), vec!["a".into(), "c".into()]); + adj.insert("c".into(), vec!["a".into(), "b".into()]); + adj.insert("d".into(), vec!["e".into(), "f".into(), "a".into()]); + adj.insert("e".into(), vec!["d".into(), "f".into()]); + adj.insert("f".into(), vec!["d".into(), "e".into()]); + + let mut community: HashMap = HashMap::new(); + for e in ["a", "b", "c"] { + community.insert(e.into(), 0); + } + for e in ["d", "e", "f"] { + community.insert(e.into(), 1); + } + + let mut similar: HashMap> = HashMap::new(); + // `z` is similar to a but NOT a graph neighbour of a. + similar.insert( + "a".into(), + vec![("b".into(), 0.9), ("c".into(), 0.8), ("z".into(), 0.5)], + ); + similar.insert("b".into(), vec![("c".into(), 0.7), ("z".into(), 0.95)]); + + MockDocGraph { + adj, + community, + similar, + } + } + + fn ids(hits: &[ScoredId]) -> HashSet { + hits.iter().map(|h| h.id.clone()).collect() + } + + #[test] + fn scored_id_constructors() { + let d = ScoredId::new("x", 0.4, 2); + assert_eq!(d.id, "x"); + assert_eq!(d.depth, 2); + let b = ScoredId::base("y", 0.9); + assert_eq!(b.depth, 0); + } + + #[test] + fn community_surface_round_trips() { + let g = fixture(); + assert_eq!(g.community_of("a"), Some(0)); + assert_eq!(g.community_of("f"), Some(1)); + assert_eq!(g.community_of("nobody"), None); + assert_eq!(g.community_ids(), vec![0, 1]); + assert_eq!(g.community_members(0), vec!["a", "b", "c"]); + assert_eq!(g.community_members(1), vec!["d", "e", "f"]); + } + + #[test] + fn neighbours_breadth_grows_with_hops() { + let g = fixture(); + let seeds = vec!["a".to_string()]; + let one = ids(&g.neighbours(&seeds, 1)); + // 1 hop from a: b, c, d. + assert_eq!(one, ["b", "c", "d"].iter().map(|s| s.to_string()).collect()); + let two = ids(&g.neighbours(&seeds, 2)); + // 2 hops also reaches e, f (via d). + assert!(two.is_superset(&one)); + assert!(two.contains("e") && two.contains("f")); + } + + #[test] + fn similar_is_sorted_and_truncated() { + let g = fixture(); + let top = g.similar_by_ranking("a", 2); + assert_eq!(top.len(), 2); + // Best-first: b (0.9) then c (0.8), z (0.5) truncated. + assert_eq!(top[0].id, "b"); + assert_eq!(top[1].id, "c"); + assert!(top[0].score >= top[1].score); + } + + #[test] + fn retrieve_rung_base_is_ranking_not_walk() { + // Rung 0–1 uses similar_by_ranking → surfaces `z` (similar but NOT a + // graph neighbour). That `z` presence proves the base is the ranking + // leg, not the SPO-G walk. + let g = fixture(); + let seeds = vec!["a".to_string()]; + let base = ids(&g.retrieve(&seeds, RungLevel::Surface, 10)); + assert!(base.contains("z"), "base rung must use the ranking leg"); + assert!(!base.contains("e"), "base rung must NOT walk to 2-hop e"); + } + + #[test] + fn retrieve_rung2_is_two_hop_walk() { + // Rung 2 = SPO-G hop: reaches the cross-community d, e, f (the wide + // walk), and never the ranking-only `z`. + let g = fixture(); + let seeds = vec!["a".to_string()]; + let r2 = ids(&g.retrieve(&seeds, RungLevel::Contextual, 10)); + assert!(r2.contains("d") && r2.contains("e") && r2.contains("f")); + assert!(!r2.contains("z"), "the walk is not the ranking leg"); + } + + #[test] + fn retrieve_rung3_is_community_scoped() { + // Rung 3 widens the walk THEN focuses to a's community {a,b,c}: the + // cross-bridge d/e/f (community 1) are dropped. This is the community + // focus that rung 2's raw walk does not apply. + let g = fixture(); + let seeds = vec!["a".to_string()]; + let r3 = ids(&g.retrieve(&seeds, RungLevel::Analogical, 10)); + assert!(r3.contains("b") && r3.contains("c")); + assert!( + !r3.contains("d") && !r3.contains("e") && !r3.contains("f"), + "rung 3 focuses to the seed community" + ); + } + + #[test] + fn retrieve_dedups_best_score_and_truncates() { + // Seeds a and b both list `z` in their similar tables (a:0.5, b:0.95). + // The union must keep ONE z at the best score (0.95), sorted best-first. + let g = fixture(); + let seeds = vec!["a".to_string(), "b".to_string()]; + let hits = g.retrieve(&seeds, RungLevel::Surface, 10); + let zs: Vec<&ScoredId> = hits.iter().filter(|h| h.id == "z").collect(); + assert_eq!(zs.len(), 1, "z must be deduped"); + assert!((zs[0].score - 0.95).abs() < 1e-6, "keeps the best score"); + // Sorted best-first. + for w in hits.windows(2) { + assert!(w[0].score >= w[1].score); + } + // top_k respected. + let capped = g.retrieve(&seeds, RungLevel::Surface, 2); + assert!(capped.len() <= 2); + } + + #[test] + fn retrieve_empty_seeds_is_safe() { + let g = fixture(); + assert!(g.retrieve(&[], RungLevel::Contextual, 10).is_empty()); + } +} diff --git a/crates/lance-graph-contract/src/lib.rs b/crates/lance-graph-contract/src/lib.rs index 05b1de4c..be91f306 100644 --- a/crates/lance-graph-contract/src/lib.rs +++ b/crates/lance-graph-contract/src/lib.rs @@ -74,6 +74,11 @@ pub mod cycle_accumulator; /// content store. See module docs for the `dawg.{h,cpp}` byte-parity scope. pub mod dawg; pub mod distance; +/// D-GR-1 — `DocGraphQuery`, the zero-dep read surface over the calcified +/// document / fact graph. Implemented by the AriGraph carrier (graphrag +/// retrieval); consumed by OGAR `ogar-doc` through the contract. Carries the +/// D-GR-2 elevator-binding spec in its module docs. +pub mod doc_graph; /// D-V3-W6a — DDL typed-emission counting logic (`TypedForm`, /// `classify_ddl_type`, `EmissionCounts`, `count_emission`), sibling of /// [`classid_scan`]. Requested by the op-nexgen consumer session. @@ -167,6 +172,7 @@ pub use class_view::{ ClassId, ClassProjection, ClassView, FieldMask, RenderRow, ValueRow, WideFieldMask, }; pub use collapse_gate::{GateDecision, MailboxId, MergeMode}; +pub use doc_graph::{DocGraphQuery, ScoredId}; pub use episodic_edges::{EdgeRef, EpisodicEdges64}; pub use head2head::{CompetitionOutcome, Head2Head, WinnerCriterion}; pub use kanban::{ExecTarget, KanbanColumn, KanbanMove, RubiconTransitionError}; diff --git a/crates/lance-graph/examples/g0_graph_loadbearing.rs b/crates/lance-graph/examples/g0_graph_loadbearing.rs new file mode 100644 index 00000000..1cd3ef7a --- /dev/null +++ b/crates/lance-graph/examples/g0_graph_loadbearing.rs @@ -0,0 +1,190 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! G0 — the **P-GRAPH-LOADBEARING** measurement harness (the graphrag plan's gate). +//! +//! The plan (`.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md` §6, §9) +//! makes this probe the *first* deliverable and the gate on all retrieval wiring: +//! +//! > On a document corpus, measure retrieval quality **with vs without** graph +//! > traversal (vector-only vs vector+SPO-G+PPR). KILL condition: if the graph +//! > does not beat vector-only on multi-hop/global questions, do **not** wire +//! > Leiden/PPR into retrieval — the graph would be decorative. +//! +//! This harness exercises the three landed, *reversible* capabilities +//! ([`Bm25Index`], [`TripletGraph::personalized_pagerank`], +//! [`TripletGraph::communities`]) end-to-end so their composition is visible and +//! reproducible. It **prints** the with-vs-without delta; it deliberately does +//! **not** assert a PASS/FAIL, because the real gate is a measurement over a +//! *labeled multi-hop corpus with realistic distractors*, which this synthetic +//! fixture is not. Run it: +//! +//! ```text +//! cargo run -p lance-graph --example g0_graph_loadbearing +//! ``` +//! +//! ## The fixture (a controlled multi-hop instance) +//! +//! A 4-fact chain `alice → acme → beta → turbines` plus a disconnected lexical +//! distractor. The gold answer entity (`turbines`) is **lexically absent** from +//! the query yet **graph-reachable** from the lexical seed (`alice`) only by a +//! 3-hop walk. That is precisely the regime where a graph should beat a bag of +//! words — and precisely the regime the KILL condition targets. + +use std::collections::BTreeMap; + +use lance_graph::graph::arigraph::triplet_graph::{Triplet, TripletGraph}; +use lance_graph::graph::arigraph::Bm25Index; + +/// `(document text, [(subject, relation, object), ...])` — each document yields +/// the SPO facts the graph is built from, so the entity→document map is exact. +type Corpus = [( + &'static str, + &'static [(&'static str, &'static str, &'static str)], +)]; + +fn rank_of(ranked: &[(String, f64)], entity: &str) -> Option { + ranked.iter().position(|(e, _)| e == entity).map(|i| i + 1) +} + +fn main() { + // ── 1. Synthetic corpus + the fact graph each document yields ──────────── + let corpus: &Corpus = &[ + ( + "Alice founded Acme in Berlin.", + &[ + ("alice", "founded", "acme"), + ("acme", "located_in", "berlin"), + ], + ), + ( + "Acme acquired Beta Corp last year.", + &[("acme", "acquired", "beta")], + ), + ( + "Beta Corp builds wind turbines.", + &[("beta", "builds", "turbines")], + ), + ( + "Berlin is the capital of Germany.", + &[("berlin", "capital_of", "germany")], + ), + ( + "Carol paints alpine landscapes.", // disconnected distractor + &[("carol", "paints", "landscapes")], + ), + ]; + + let docs: Vec<&str> = corpus.iter().map(|(t, _)| *t).collect(); + let mut graph = TripletGraph::new(); + let mut ts = 0u64; + for (_txt, facts) in corpus { + let trips: Vec = facts + .iter() + .map(|(s, r, o)| { + let t = Triplet::new(s, o, r, ts); + ts += 1; + t + }) + .collect(); + graph.add_triplets(&trips); + } + + // entity → documents that mention it (exact, from the fact provenance). + let mut entity_docs: BTreeMap<&str, Vec> = BTreeMap::new(); + for (doc_id, (_txt, facts)) in corpus.iter().enumerate() { + for (s, _r, o) in *facts { + entity_docs.entry(s).or_default().push(doc_id); + entity_docs.entry(o).or_default().push(doc_id); + } + } + + // ── 2. The multi-hop query + the gold answer entity ────────────────────── + // "turbines" never appears in the query; it is reachable from "alice" only + // through alice→acme→beta→turbines. + let query = "the company Alice founded"; + let gold = "turbines"; + + let bm25 = Bm25Index::build(&docs); + + // ── 3a. VECTOR-ONLY (no traversal): score each entity by the best BM25 doc + // score among the documents that mention it. Pure lexical. ───────── + let mut vector_only: Vec<(String, f64)> = entity_docs + .iter() + .map(|(&e, doc_ids)| { + let best = doc_ids + .iter() + .map(|&d| bm25.score(query, d)) + .fold(0.0f64, f64::max); + (e.to_string(), best) + }) + .collect(); + vector_only.sort_by(|a, b| { + b.1.partial_cmp(&a.1) + .unwrap_or(std::cmp::Ordering::Equal) + .then_with(|| a.0.cmp(&b.0)) + }); + + // ── 3b. GRAPH-AUGMENTED: BM25 picks the top document; its entities seed a + // personalized-PageRank spread over the fact graph. ─────────────── + let top_doc = bm25.rank(query).first().map(|&(d, _)| d).unwrap_or(0); + let mut seeds: Vec<&str> = corpus[top_doc] + .1 + .iter() + .flat_map(|(s, _r, o)| [*s, *o]) + .collect(); + seeds.sort_unstable(); + seeds.dedup(); + let ppr = graph.personalized_pagerank(&seeds, 0.85, 50); + let graph_ranked: Vec<(String, f64)> = ppr + .ranked() + .into_iter() + .map(|(e, s)| (e.to_string(), s)) + .collect(); + + // ── 4. Report the with-vs-without delta ────────────────────────────────── + println!("== G0 · P-GRAPH-LOADBEARING (synthetic multi-hop scaffold) ==\n"); + println!("query : {query:?}"); + println!("seeds : {seeds:?} (entities of BM25 top document #{top_doc})"); + println!("gold : {gold:?} (lexically absent from the query; 3 hops from \"alice\")\n"); + + let print_top = |label: &str, ranked: &[(String, f64)]| { + println!("{label}"); + for (i, (e, s)) in ranked.iter().take(5).enumerate() { + println!(" {:>2}. {:<12} {:.4}", i + 1, e, s); + } + }; + print_top("vector-only (BM25 lexical, no traversal):", &vector_only); + print_top("graph-augmented (BM25 seed → PPR spread):", &graph_ranked); + + let r_vec = rank_of(&vector_only, gold); + let r_graph = rank_of(&graph_ranked, gold); + println!("\nrank of gold {gold:?}: vector-only = {r_vec:?} graph = {r_graph:?}"); + match (r_vec, r_graph) { + (Some(v), Some(g)) if g < v => println!( + " → graph surfaces the multi-hop answer {} place(s) higher (mechanism is load-bearing here).", + v - g + ), + (Some(v), Some(g)) if g == v => println!(" → no change on this fixture."), + (Some(_), Some(_)) => println!(" → vector-only ranked it higher on this fixture."), + _ => println!(" → gold missing from one ranking."), + } + + // Community focus (the "documents in this community" / D-GR-4 direction). + let comms = graph.communities(); + if let Some(c) = comms.community_of(gold) { + println!( + "\ncommunity of {gold:?}: #{c} members={:?} (Q={:.3}, {} communities)", + comms.members(c), + comms.modularity, + comms.num_communities + ); + } + + println!( + "\nNOTE: synthetic mechanism scaffold, NOT the gate verdict. The real KILL/PASS\n\ + needs a labeled multi-hop corpus with realistic distractors + jc::reliability\n\ + (Spearman/ICC) on the with-vs-without delta. D-GR-2 retrieval wiring stays\n\ + gated on that run (plan §9)." + ); +} diff --git a/crates/lance-graph/src/graph/arigraph/bm25.rs b/crates/lance-graph/src/graph/arigraph/bm25.rs new file mode 100644 index 00000000..cecded91 --- /dev/null +++ b/crates/lance-graph/src/graph/arigraph/bm25.rs @@ -0,0 +1,226 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! `bm25` — classic Okapi BM25 lexical ranking over a small in-memory +//! document corpus. +//! +//! # Where this sits (doctrine) +//! +//! BM25 is the rung-0/1 **lexical** baseline: exact-term relevance scoring, +//! computed independently of the embedding/codec stack. It complements +//! rather than competes with CAM-PQ vector ranking — the retrieval +//! capstone (D-GR-2) is expected to consume both signals side by side +//! (lexical recall for exact-term queries, CAM-PQ for semantic recall), the +//! same way [`community`](super::community) complements episodic basins: +//! two independent signals that cross-validate rather than duplicate. +//! +//! # Algorithm +//! +//! Classic Okapi BM25 (Robertson & Walker 1994). Tokenization is +//! deliberately minimal: lowercase, split on any non-alphanumeric +//! character. Per-term weight is `idf(term) * saturated_tf(term, doc)`: +//! +//! - `idf = ln(1 + (N - df + 0.5) / (df + 0.5))` — standard (non-negative) +//! Okapi idf, `N` = corpus size, `df` = document frequency of the term. +//! - `saturated_tf = tf * (k1 + 1) / (tf + k1 * (1 - b + b * dl / avgdl))` +//! — term-frequency saturation with document-length normalization. +//! - `k1 = 1.2`, `b = 0.75` — the standard defaults from the original paper +//! and most production BM25 implementations. +//! +//! Deterministic and dependency-free: per-document term frequencies are +//! accumulated via `BTreeMap` (stable iteration order) before landing in +//! the postings list, so two [`Bm25Index::build`] calls over the same +//! corpus produce identical indices, and [`Bm25Index::rank`] output is +//! stable across runs. + +use std::collections::{BTreeMap, HashMap}; + +/// BM25 free parameter: term-frequency saturation rate. +const K1: f64 = 1.2; +/// BM25 free parameter: document-length normalization strength. +const B: f64 = 0.75; + +/// A BM25-indexed corpus of small in-memory documents. +#[derive(Debug, Clone)] +pub struct Bm25Index { + /// Number of documents in the corpus. + n_docs: usize, + /// Average document length in tokens (`0.0` for an empty corpus). + avgdl: f64, + /// Token count per document, indexed by `doc_id`. + doc_len: Vec, + /// Document frequency: term → number of documents containing it. + df: HashMap, + /// Inverted index: term → `(doc_id, term_frequency)`, `doc_id` ascending. + postings: HashMap>, +} + +impl Bm25Index { + /// Build a BM25 index from `docs`. Tokenization: lowercase, split on any + /// non-alphanumeric character (Unicode-aware), empty tokens dropped. + pub fn build(docs: &[&str]) -> Self { + let n_docs = docs.len(); + let mut doc_len = Vec::with_capacity(n_docs); + let mut df: HashMap = HashMap::new(); + let mut postings: HashMap> = HashMap::new(); + + for (doc_id, doc) in docs.iter().enumerate() { + let tokens = tokenize(doc); + doc_len.push(tokens.len()); + + // BTreeMap: deterministic per-document term iteration order, so + // the postings list for every term is built in doc_id order. + let mut tf: BTreeMap = BTreeMap::new(); + for tok in tokens { + *tf.entry(tok).or_insert(0) += 1; + } + for (term, count) in tf { + *df.entry(term.clone()).or_insert(0) += 1; + postings.entry(term).or_default().push((doc_id, count)); + } + } + + let total_len: usize = doc_len.iter().sum(); + let avgdl = if n_docs == 0 { + 0.0 + } else { + total_len as f64 / n_docs as f64 + }; + + Self { + n_docs, + avgdl, + doc_len, + df, + postings, + } + } + + /// Okapi BM25 score of `doc_id` for `query`. Query terms are tokenized + /// the same way as documents and deduplicated (a repeated query term + /// does not double-count). Returns `0.0` for an out-of-range `doc_id`, + /// an empty query, or a query whose terms never occur in the corpus. + pub fn score(&self, query: &str, doc_id: usize) -> f64 { + if doc_id >= self.n_docs { + return 0.0; + } + let dl = self.doc_len[doc_id] as f64; + let mut query_terms = tokenize(query); + query_terms.sort_unstable(); + query_terms.dedup(); + + let mut total = 0.0; + for term in &query_terms { + let Some(&df) = self.df.get(term) else { + continue; + }; + let Some(postings) = self.postings.get(term) else { + continue; + }; + let tf = postings + .binary_search_by_key(&doc_id, |&(d, _)| d) + .ok() + .map(|idx| f64::from(postings[idx].1)) + .unwrap_or(0.0); + if tf == 0.0 { + continue; + } + let denom = tf + K1 * (1.0 - B + B * dl / self.avgdl); + let saturated_tf = tf * (K1 + 1.0) / denom; + total += idf(self.n_docs, df) * saturated_tf; + } + total + } + + /// All documents ranked by [`Self::score`] against `query`, descending; + /// ties (equal score, including all-zero for an empty query or a query + /// with no corpus matches) are broken by `doc_id` ascending. + pub fn rank(&self, query: &str) -> Vec<(usize, f64)> { + let mut scores: Vec<(usize, f64)> = (0..self.n_docs) + .map(|doc_id| (doc_id, self.score(query, doc_id))) + .collect(); + scores.sort_by(|a, b| { + b.1.partial_cmp(&a.1) + .unwrap_or(std::cmp::Ordering::Equal) + .then_with(|| a.0.cmp(&b.0)) + }); + scores + } +} + +/// Standard (non-negative) Okapi idf: `ln(1 + (N - df + 0.5) / (df + 0.5))`. +fn idf(n_docs: usize, df: usize) -> f64 { + (((n_docs as f64 - df as f64 + 0.5) / (df as f64 + 0.5)) + 1.0).ln() +} + +/// Tokenize: lowercase, split on non-alphanumeric, drop empty pieces. +fn tokenize(text: &str) -> Vec { + let lower = text.to_lowercase(); + lower + .split(|c: char| !c.is_alphanumeric()) + .filter(|s| !s.is_empty()) + .map(String::from) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn doc_with_query_terms_outranks_doc_without() { + let docs = ["the quick brown fox", "lorem ipsum dolor sit amet"]; + let idx = Bm25Index::build(&docs); + let ranked = idx.rank("quick fox"); + assert_eq!( + ranked[0].0, 0, + "doc 0 contains both query terms: {ranked:?}" + ); + assert!(ranked[0].1 > ranked[1].1); + assert_eq!(ranked[1].1, 0.0, "doc 1 contains neither query term"); + } + + #[test] + fn rarer_term_has_higher_idf_than_common_term() { + // 10-document corpus; a term found in 1 doc is rarer than one found + // in 8, so it must carry more idf weight. + assert!( + idf(10, 1) > idf(10, 8), + "idf(10,1)={} should exceed idf(10,8)={}", + idf(10, 1), + idf(10, 8) + ); + } + + #[test] + fn empty_query_yields_all_zero_ranking() { + let docs = ["alpha beta", "gamma delta"]; + let idx = Bm25Index::build(&docs); + let ranked = idx.rank(""); + assert_eq!(ranked.len(), 2); + assert!(ranked.iter().all(|&(_, s)| s == 0.0)); + assert_eq!( + ranked[0].0, 0, + "ties broken by doc_id ascending: {ranked:?}" + ); + assert_eq!(ranked[1].0, 1); + } + + #[test] + fn rank_is_deterministic() { + let docs = [ + "alpha beta gamma", + "beta gamma delta", + "gamma delta epsilon", + ]; + let idx = Bm25Index::build(&docs); + assert_eq!(idx.rank("beta gamma"), idx.rank("beta gamma")); + } + + #[test] + fn empty_corpus_is_safe() { + let idx = Bm25Index::build(&[]); + assert!(idx.rank("anything").is_empty()); + assert_eq!(idx.score("anything", 0), 0.0); + } +} diff --git a/crates/lance-graph/src/graph/arigraph/community.rs b/crates/lance-graph/src/graph/arigraph/community.rs index 527701b7..5df467b1 100644 --- a/crates/lance-graph/src/graph/arigraph/community.rs +++ b/crates/lance-graph/src/graph/arigraph/community.rs @@ -38,10 +38,13 @@ //! index order and candidate communities in `BTreeMap` order, so the same graph //! always yields the same partition — a property an LLM-based clustering cannot //! offer, and the precondition for certifying it with the jc reliability -//! battery. The Leiden *refinement* pass (guaranteeing internally-connected -//! communities) is the next increment; Louvain is the core it refines. +//! battery. A **Leiden connectivity refinement** pass (`refine_connected`) +//! follows: any coarsest-level community whose induced subgraph is internally +//! disconnected is split into its connected components, guaranteeing every +//! returned community is internally connected — Leiden's defining property +//! over plain Louvain. Louvain remains the core the refinement operates on. -use std::collections::{BTreeMap, HashMap}; +use std::collections::{BTreeMap, HashMap, VecDeque}; use super::triplet_graph::TripletGraph; @@ -75,17 +78,33 @@ impl Communities { self.entities .iter() .zip(self.labels.iter()) - .filter_map(|(e, &c)| if c == community { Some(e.as_str()) } else { None }) + .filter_map(|(e, &c)| { + if c == community { + Some(e.as_str()) + } else { + None + } + }) .collect() } } impl TripletGraph { /// Detect structural communities over this graph's entities via multi-level - /// Louvain modularity. Edges are the non-deleted triplets (subject↔object), - /// weighted by NARS **confidence**; self-loops (`subject == object`) are - /// dropped. Deterministic and dependency-free (reads the graph the mailbox - /// already owns; never a global partition singleton). + /// Louvain modularity, followed by a Leiden connectivity refinement pass. + /// Edges are the non-deleted triplets (subject↔object), weighted by NARS + /// **confidence**; self-loops (`subject == object`) are dropped. + /// Deterministic and dependency-free (reads the graph the mailbox already + /// owns; never a global partition singleton). + /// + /// `labels` on the returned [`Communities`] is the **connectivity-refined** + /// coarsest partition: any community whose induced subgraph came out of + /// Louvain internally disconnected is split into its connected components, + /// so every returned community is guaranteed internally connected (Leiden's + /// defining property over plain Louvain). `levels.last()` is the + /// **pre-refinement** Louvain partition — the raw coarsest level from + /// local-moving + aggregation, which may still contain an + /// internally-disconnected community. pub fn communities(&self) -> Communities { let (entities, index) = self.entity_index_dense(); let n = entities.len(); @@ -99,8 +118,9 @@ impl TripletGraph { }; } let adj = self.build_adjacency(&index, n); - let (levels_u, labels_u, q) = detect(adj, vec![0.0; n]); - let labels: Vec = labels_u.iter().map(|&x| x as u32).collect(); + let (levels_u, labels_u, q) = detect(adj.clone(), vec![0.0; n]); + let louvain_labels: Vec = labels_u.iter().map(|&x| x as u32).collect(); + let labels = refine_connected(&adj, &louvain_labels); let levels: Vec> = levels_u .iter() .map(|lv| lv.iter().map(|&x| x as u32).collect()) @@ -111,7 +131,13 @@ impl TripletGraph { s.dedup(); s.len() }; - Communities { entities, labels, levels, modularity: q, num_communities } + Communities { + entities, + labels, + levels, + modularity: q, + num_communities, + } } /// Dense entity index: a stable `Vec` (sorted for determinism) and @@ -119,19 +145,18 @@ impl TripletGraph { fn entity_index_dense(&self) -> (Vec, HashMap) { let mut names: Vec = self.entity_index.keys().cloned().collect(); names.sort_unstable(); - let index: HashMap = - names.iter().enumerate().map(|(i, e)| (e.clone(), i)).collect(); + let index: HashMap = names + .iter() + .enumerate() + .map(|(i, e)| (e.clone(), i)) + .collect(); (names, index) } /// Weighted undirected adjacency (both endpoints), weight = summed NARS /// confidence over the triplets joining two entities. Deleted triplets and /// self-loops are skipped. - fn build_adjacency( - &self, - index: &HashMap, - n: usize, - ) -> Vec> { + fn build_adjacency(&self, index: &HashMap, n: usize) -> Vec> { // Accumulate undirected weights in a per-node BTreeMap for determinism. let mut acc: Vec> = vec![BTreeMap::new(); n]; for t in &self.triplets { @@ -176,7 +201,12 @@ fn build(adj: Vec>, self_loop: Vec) -> WGraph { degree[u] = d + 2.0 * self_loop[u]; } let two_m: f64 = degree.iter().sum(); - WGraph { adj, self_loop, degree, two_m } + WGraph { + adj, + self_loop, + degree, + two_m, + } } /// First-appearance dense relabel — deterministic given node order. @@ -267,8 +297,10 @@ fn aggregate(g: &WGraph, label: &[usize]) -> WGraph { } } } - let adj: Vec> = - super_adj.into_iter().map(|m| m.into_iter().collect()).collect(); + let adj: Vec> = super_adj + .into_iter() + .map(|m| m.into_iter().collect()) + .collect(); build(adj, self_w) } @@ -298,10 +330,7 @@ fn modularity(g: &WGraph, label: &[usize]) -> f64 { /// Multi-level Louvain. Returns (hierarchy over ORIGINAL nodes, coarsest /// labels, `Q` of the coarsest partition on the original graph). -fn detect( - adj0: Vec>, - self0: Vec, -) -> (Vec>, Vec, f64) { +fn detect(adj0: Vec>, self0: Vec) -> (Vec>, Vec, f64) { let n0 = adj0.len(); let g0 = build(adj0.clone(), self0.clone()); let mut g = build(adj0, self0); @@ -325,6 +354,39 @@ fn detect( (levels, coarsest, q) } +/// Split any internally-disconnected community into its connected components +/// (the Leiden connectivity guarantee over Louvain). Deterministic: nodes are +/// visited in dense-index order and components are relabeled by first appearance. +fn refine_connected(adj: &[Vec<(usize, f64)>], labels: &[u32]) -> Vec { + let n = adj.len(); + let mut component: Vec> = vec![None; n]; + let mut next_id: u32 = 0; + for start in 0..n { + if component[start].is_some() { + continue; + } + let mut queue: VecDeque = VecDeque::new(); + queue.push_back(start); + component[start] = Some(next_id); + while let Some(u) = queue.pop_front() { + for &(v, _w) in &adj[u] { + if v == u || component[v].is_some() { + continue; + } + if labels[v] == labels[u] { + component[v] = Some(next_id); + queue.push_back(v); + } + } + } + next_id += 1; + } + component + .into_iter() + .map(|c| c.expect("every node visited exactly once")) + .collect() +} + #[cfg(test)] mod tests { use super::*; @@ -345,8 +407,12 @@ mod tests { fn two_triangles_bridge_yields_two_communities() { // {a,b,c} triangle, {d,e,f} triangle, single a-d bridge. let g = tg(&[ - ("a", "b"), ("b", "c"), ("c", "a"), - ("d", "e"), ("e", "f"), ("f", "d"), + ("a", "b"), + ("b", "c"), + ("c", "a"), + ("d", "e"), + ("e", "f"), + ("f", "d"), ("a", "d"), ]); let c = g.communities(); @@ -360,13 +426,28 @@ mod tests { #[test] fn deterministic() { - let g = tg(&[("a", "b"), ("b", "c"), ("c", "a"), ("d", "e"), ("e", "f"), ("f", "d"), ("a", "d")]); + let g = tg(&[ + ("a", "b"), + ("b", "c"), + ("c", "a"), + ("d", "e"), + ("e", "f"), + ("f", "d"), + ("a", "d"), + ]); assert_eq!(g.communities().labels, g.communities().labels); } #[test] fn clique_is_one_community() { - let g = tg(&[("a", "b"), ("a", "c"), ("a", "d"), ("b", "c"), ("b", "d"), ("c", "d")]); + let g = tg(&[ + ("a", "b"), + ("a", "c"), + ("a", "d"), + ("b", "c"), + ("b", "d"), + ("c", "d"), + ]); assert_eq!(g.communities().num_communities, 1); } @@ -385,11 +466,69 @@ mod tests { Triplet::new("a", "b", "rel", 0), Triplet::new("b", "c", "rel", 1), Triplet::new("c", "a", "rel", 2), - Triplet::with_truth("c", "d", "rel", crate::graph::spo::truth::TruthValue::new(1.0, 0.05), 3), + Triplet::with_truth( + "c", + "d", + "rel", + crate::graph::spo::truth::TruthValue::new(1.0, 0.05), + 3, + ), ]; g.add_triplets(&ts); let c = g.communities(); assert_eq!(c.community_of("a"), c.community_of("b")); assert_eq!(c.community_of("b"), c.community_of("c")); } + + #[test] + fn refine_connected_splits_disconnected_community() { + // Two disjoint edges 0-1 and 2-3, no edge between the pairs, but a + // (deliberately wrong) Louvain-style label lumps all four into + // community 0 — the case a real aggregation step can produce. + let adj: Vec> = vec![ + vec![(1, 1.0)], + vec![(0, 1.0)], + vec![(3, 1.0)], + vec![(2, 1.0)], + ]; + let labels = vec![0u32, 0, 0, 0]; + let refined = refine_connected(&adj, &labels); + assert_eq!(refined[0], refined[1], "0 and 1 share an edge: {refined:?}"); + assert_eq!(refined[2], refined[3], "2 and 3 share an edge: {refined:?}"); + assert_ne!( + refined[0], refined[2], + "0-1 and 2-3 are disconnected: {refined:?}" + ); + let mut distinct = refined.clone(); + distinct.sort_unstable(); + distinct.dedup(); + assert_eq!( + distinct.len(), + 2, + "expected exactly 2 components: {refined:?}" + ); + } + + #[test] + fn refine_connected_is_idempotent_on_connected_community() { + // A single connected triangle, already one label - refinement must + // not introduce a spurious split. + let adj: Vec> = vec![ + vec![(1, 1.0), (2, 1.0)], + vec![(0, 1.0), (2, 1.0)], + vec![(0, 1.0), (1, 1.0)], + ]; + let labels = vec![0u32, 0, 0]; + let refined = refine_connected(&adj, &labels); + let mut distinct = refined.clone(); + distinct.sort_unstable(); + distinct.dedup(); + assert_eq!( + distinct.len(), + 1, + "connected community must stay one component: {refined:?}" + ); + assert_eq!(refined[0], refined[1]); + assert_eq!(refined[1], refined[2]); + } } diff --git a/crates/lance-graph/src/graph/arigraph/mod.rs b/crates/lance-graph/src/graph/arigraph/mod.rs index 95220134..8fd9ab2c 100644 --- a/crates/lance-graph/src/graph/arigraph/mod.rs +++ b/crates/lance-graph/src/graph/arigraph/mod.rs @@ -5,11 +5,13 @@ //! //! Transcoded from Python AriGraph — a memory architecture for LLM agents. +pub mod bm25; pub mod community; pub mod episodic; pub mod language; pub mod markov_soa; pub mod orchestrator; +pub mod ppr; pub mod retrieval; pub mod sensorium; pub mod spo_bridge; @@ -17,7 +19,9 @@ pub mod triplet_graph; pub mod witness_corpus; pub mod xai_client; +pub use bm25::Bm25Index; pub use community::Communities; +pub use ppr::PersonalizedPageRank; pub use witness_corpus::{WitnessCorpus, WitnessEntry, WitnessId, WitnessIndexHashMap}; #[cfg(feature = "with-cam-pq")] diff --git a/crates/lance-graph/src/graph/arigraph/ppr.rs b/crates/lance-graph/src/graph/arigraph/ppr.rs new file mode 100644 index 00000000..fd5a1d48 --- /dev/null +++ b/crates/lance-graph/src/graph/arigraph/ppr.rs @@ -0,0 +1,368 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! `ppr` — Personalized PageRank (HippoRAG-style spread activation) over the +//! AriGraph [`TripletGraph`](super::triplet_graph::TripletGraph). +//! +//! # Where this sits (doctrine) +//! +//! [`communities`](TripletGraph::communities) partitions the fact graph +//! *structurally* — which entities cluster together. PPR answers a +//! complementary question: *given a seed set, how related is every other +//! entity to it?* Where community detection yields a discrete partition, PPR +//! yields a continuous relevance ranking over the whole graph — the rung-3 +//! multi-hop retrieval primitive HippoRAG (Gutiérrez et al. 2024, +//! "HippoRAG: Neurobiologically Inspired Long-Term Memory for LLMs") +//! popularised as a single-step alternative to iterative multi-hop RAG: +//! spread restart mass from seed entities (produced by rung-0/1 similarity +//! ranking) across the SPO fact graph and rank every entity by the mass it +//! accumulates. This is the rung-3 layer of the ascent (observation/ranking +//! → SPO hop → community/PPR), the same rung `community.rs` occupies from +//! the orthogonal, structural side. +//! +//! Per the workspace litmus (*method on the carrier, not a free function on +//! its state*), the entry point is [`TripletGraph::personalized_pagerank`], +//! never an external service over the graph. It reads the same graph the +//! mailbox already owns — no copy, no service call. +//! +//! # Algorithm +//! +//! Classic personalized PageRank via power iteration over a confidence- +//! weighted, row-normalized, undirected transition matrix (Page et al. 1999 +//! — the "random surfer" restarting at the seed set instead of uniformly). +//! Deterministic: dense sorted entity index, `BTreeMap`-ordered adjacency +//! accumulation — the identical adjacency discipline +//! [`communities`](TripletGraph::communities) uses — so the same graph and +//! seeds always yield the same scores. A property an LLM-based retriever +//! cannot offer, and the precondition for certifying it with the jc +//! reliability battery. Dangling (degree-0) nodes redistribute their mass +//! into the personalization vector each step rather than leaking it off the +//! graph, so the result is always a proper probability distribution +//! (`scores.iter().sum() ≈ 1.0`). + +use std::collections::{BTreeMap, HashMap}; + +use super::triplet_graph::TripletGraph; + +/// Result of a personalized-PageRank spread over the fact graph. +#[derive(Debug, Clone)] +pub struct PersonalizedPageRank { + /// Dense node id → entity name (parallel to [`Self::scores`]). + pub entities: Vec, + /// Restart-mass share per entity, in the same order as [`Self::entities`]. + /// Forms a probability distribution: `scores.iter().sum() ≈ 1.0`. + pub scores: Vec, +} + +impl PersonalizedPageRank { + /// The PPR score of `entity`, by exact-name lookup. `None` if `entity` + /// is not in the graph. + pub fn score_of(&self, entity: &str) -> Option { + self.entities + .iter() + .position(|e| e == entity) + .map(|i| self.scores[i]) + } + + /// All `(entity, score)` pairs, ranked descending by score; ties are + /// broken by entity name ascending, so the order is fully deterministic. + pub fn ranked(&self) -> Vec<(&str, f64)> { + let mut pairs: Vec<(&str, f64)> = self + .entities + .iter() + .map(String::as_str) + .zip(self.scores.iter().copied()) + .collect(); + pairs.sort_by(|a, b| { + b.1.partial_cmp(&a.1) + .unwrap_or(std::cmp::Ordering::Equal) + .then_with(|| a.0.cmp(b.0)) + }); + pairs + } + + /// The top `k` `(entity, score)` pairs, per [`Self::ranked`] order. + pub fn top_k(&self, k: usize) -> Vec<(&str, f64)> { + let mut top = self.ranked(); + top.truncate(k); + top + } +} + +impl TripletGraph { + /// Personalized PageRank (HippoRAG spread-activation) over the fact + /// graph. + /// + /// `seeds` are entity names that receive the restart mass — matched + /// first by exact name and, failing that, case-insensitively against + /// the entity index. `damping` is the follow-edge probability (`0.85` + /// is the canonical PageRank value; the teleport-back probability to + /// the personalization vector is `1.0 - damping`). `iters` is the + /// power-iteration step count. + /// + /// Edges are the non-deleted triplets (subject↔object), weighted by + /// NARS **confidence**; self-loops (`subject == object`) are dropped — + /// the identical adjacency discipline + /// [`communities`](Self::communities) uses. If none of `seeds` match an + /// entity in the graph, the personalization vector falls back to + /// uniform over every entity rather than producing an empty spread. + /// + /// Deterministic and dependency-free (reads the graph the mailbox + /// already owns; never a global ranking singleton). The returned scores + /// always sum to approximately `1.0`; an empty graph returns an empty + /// result rather than panicking. + pub fn personalized_pagerank( + &self, + seeds: &[&str], + damping: f64, + iters: usize, + ) -> PersonalizedPageRank { + let (entities, index) = dense_entity_index(self); + let n = entities.len(); + if n == 0 { + return PersonalizedPageRank { + entities, + scores: Vec::new(), + }; + } + + let adj = weighted_adjacency(self, &index, n); + let degree: Vec = adj + .iter() + .map(|row| row.iter().map(|&(_, w)| w).sum()) + .collect(); + let restart = personalization_vector(&entities, &index, seeds, n); + + let mut r = restart.clone(); + for _ in 0..iters { + r = power_step(&adj, °ree, &restart, &r, damping, n); + } + normalize_to_unit_sum(&mut r); + + PersonalizedPageRank { + entities, + scores: r, + } + } +} + +// ── pure PPR core (std only; verified standalone) ─────────────────────────── +// +// Free functions rather than `TripletGraph` methods: inherent-impl method +// names are unique per type across the whole crate regardless of privacy, so +// duplicating `community.rs`'s private `entity_index_dense` / +// `build_adjacency` method names here would collide. These operate on +// `TripletGraph`'s public `triplets` / `entity_index` fields directly. + +/// Dense entity index: a stable `Vec` (sorted for determinism) and +/// an entity → dense-id map — the same shape +/// [`communities`](TripletGraph::communities) builds internally. +fn dense_entity_index(g: &TripletGraph) -> (Vec, HashMap) { + let mut names: Vec = g.entity_index.keys().cloned().collect(); + names.sort_unstable(); + let index: HashMap = names + .iter() + .enumerate() + .map(|(i, e)| (e.clone(), i)) + .collect(); + (names, index) +} + +/// Weighted undirected adjacency (both endpoints), weight = summed NARS +/// confidence over the triplets joining two entities. Deleted triplets and +/// self-loops are skipped. +fn weighted_adjacency( + g: &TripletGraph, + index: &HashMap, + n: usize, +) -> Vec> { + // Accumulate undirected weights in a per-node BTreeMap for determinism. + let mut acc: Vec> = vec![BTreeMap::new(); n]; + for t in &g.triplets { + if t.is_deleted() || t.subject == t.object { + continue; + } + let (Some(&u), Some(&v)) = (index.get(&t.subject), index.get(&t.object)) else { + continue; + }; + let w = t.truth.confidence.max(0.0) as f64; + if w <= 0.0 { + continue; + } + *acc[u].entry(v).or_insert(0.0) += w; + *acc[v].entry(u).or_insert(0.0) += w; + } + acc.into_iter().map(|m| m.into_iter().collect()).collect() +} + +/// The restart / personalization vector: uniform mass over `seeds` matched +/// against `index` (exact name first, then a case-insensitive scan of +/// `entities`); uniform over every entity when nothing matches (including +/// an empty `seeds`). +fn personalization_vector( + entities: &[String], + index: &HashMap, + seeds: &[&str], + n: usize, +) -> Vec { + let mut matched: Vec = Vec::new(); + for &seed in seeds { + if let Some(&id) = index.get(seed) { + matched.push(id); + continue; + } + let seed_lower = seed.to_lowercase(); + if let Some(id) = entities.iter().position(|e| e.to_lowercase() == seed_lower) { + matched.push(id); + } + } + matched.sort_unstable(); + matched.dedup(); + + let mut p = vec![0.0f64; n]; + if matched.is_empty() { + p.fill(1.0 / n as f64); + } else { + let share = 1.0 / matched.len() as f64; + for id in matched { + p[id] = share; + } + } + p +} + +/// One power-iteration step: +/// `r' = (1 - damping)·p + damping·(Pᵀr + dangling_mass·p)`, where `P` is +/// the row-normalized transition matrix (`P(u→v) = w(u,v) / degree(u)`) and +/// dangling (degree-0) nodes redistribute their mass into `p` each step +/// instead of leaking it off the graph. +fn power_step( + adj: &[Vec<(usize, f64)>], + degree: &[f64], + p: &[f64], + r: &[f64], + damping: f64, + n: usize, +) -> Vec { + let mut r_next = vec![0.0f64; n]; + let mut dangling_mass = 0.0f64; + for u in 0..n { + if degree[u] <= 0.0 { + dangling_mass += r[u]; + continue; + } + let share = r[u] / degree[u]; + for &(v, w) in &adj[u] { + r_next[v] += share * w; + } + } + for x in 0..n { + r_next[x] = (1.0 - damping) * p[x] + damping * (r_next[x] + dangling_mass * p[x]); + } + r_next +} + +/// Rescale `r` to sum to `1.0`. A no-op on an all-zero vector, which cannot +/// occur here: `r` always carries the redistributed restart mass, and `p` +/// (the fallback when `r` sums to zero) always sums to `1.0` by +/// construction. +fn normalize_to_unit_sum(r: &mut [f64]) { + let sum: f64 = r.iter().sum(); + if sum > 0.0 { + for x in r.iter_mut() { + *x /= sum; + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::graph::arigraph::triplet_graph::Triplet; + + fn tg(edges: &[(&str, &str)]) -> TripletGraph { + let mut g = TripletGraph::new(); + let ts: Vec = edges + .iter() + .enumerate() + .map(|(i, (s, o))| Triplet::new(s, o, "rel", i as u64)) + .collect(); + g.add_triplets(&ts); + g + } + + /// {a,b,c} triangle, {d,e,f} triangle, single a-d bridge — the same + /// shape `community.rs`'s fixture uses, so the two rung-3 primitives are + /// exercised on a directly comparable graph. + fn two_triangles_bridge() -> TripletGraph { + tg(&[ + ("a", "b"), + ("b", "c"), + ("c", "a"), + ("d", "e"), + ("e", "f"), + ("f", "d"), + ("a", "d"), + ]) + } + + #[test] + fn seed_favors_its_own_triangle_over_the_far_one() { + let g = two_triangles_bridge(); + let ppr = g.personalized_pagerank(&["a"], 0.85, 50); + let near_min = ["a", "b", "c"] + .iter() + .map(|e| ppr.score_of(e).expect("entity present")) + .fold(f64::INFINITY, f64::min); + let far_max = ["d", "e", "f"] + .iter() + .map(|e| ppr.score_of(e).expect("entity present")) + .fold(f64::NEG_INFINITY, f64::max); + assert!( + near_min > far_max, + "near_min={near_min} far_max={far_max} ranked={:?}", + ppr.ranked() + ); + } + + #[test] + fn scores_sum_to_one() { + let g = two_triangles_bridge(); + let ppr = g.personalized_pagerank(&["a"], 0.85, 50); + let sum: f64 = ppr.scores.iter().sum(); + assert!((sum - 1.0).abs() < 1e-9, "sum={sum}"); + } + + #[test] + fn deterministic() { + let g = two_triangles_bridge(); + let a = g.personalized_pagerank(&["a"], 0.85, 50); + let b = g.personalized_pagerank(&["a"], 0.85, 50); + assert_eq!(a.entities, b.entities); + assert_eq!(a.scores, b.scores); + } + + #[test] + fn empty_graph_is_safe() { + let g = TripletGraph::new(); + let ppr = g.personalized_pagerank(&["anything"], 0.85, 50); + assert!(ppr.entities.is_empty()); + assert!(ppr.scores.is_empty()); + } + + #[test] + fn unmatched_seed_falls_back_to_uniform_without_panicking() { + let g = two_triangles_bridge(); + let ppr = g.personalized_pagerank(&["nonexistent"], 0.85, 50); + assert_eq!(ppr.scores.len(), 6); + let sum: f64 = ppr.scores.iter().sum(); + assert!((sum - 1.0).abs() < 1e-9, "sum={sum}"); + } + + #[test] + fn seeds_own_node_is_top_ranked() { + let g = two_triangles_bridge(); + let ppr = g.personalized_pagerank(&["a"], 0.85, 50); + assert_eq!(ppr.ranked()[0].0, "a"); + } +}