feat(arigraph): RRF fusion primitive (D-GR-2a) — the retrieval keystone#724
Conversation
Reciprocal Rank Fusion (Cormack, Clarke & Büttcher 2009): fuse N ranked lists into one ranking by Σ 1/(k+rank), k=60. Named as the D-GR-2 retrieval keystone in the representations inventory (#723) — every ranked leg exists (Bm25Index::rank, PersonalizedPageRank::ranked, CAM-PQ) but nothing fused them. - `arigraph/rrf.rs`: `reciprocal_rank_fusion(&[&[ScoredId]], k) -> Vec<ScoredId>` + `DEFAULT_RRF_K = 60`. Fuses by RANK, so the per-list scores need not be commensurable (the reason it combines BM25 f64 / PPR probability / CAM-PQ i8). Deterministic (BTreeMap id-asc + stable score-desc sort); shallowest depth wins; returns the contract `ScoredId`. - Re-exported from `arigraph/mod.rs`. Pure, reversible capability landed ahead of G0 (like Bm25Index / PersonalizedPageRank / Communities). The WIRING into OsintRetriever::retrieve stays gated on the G0 load-bearing verdict. Tests: 7 unit + 1 doctest green; clippy clean (`-p lance-graph --lib`). Board: STATUS_BOARD D-GR-2a + AGENT_LOG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_d4416c62-890b-4e5c-9725-64786b40cc27) |
|
Warning Review limit reached
Next review available in: 21 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
CI `format` check: rustfmt wants the `(0..10).map(...).collect()` in the smaller_k test wrapped across lines. No logic change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c87c04ce4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for (pos, item) in list.iter().enumerate() { | ||
| let rank = pos as f64 + 1.0; // 1-based | ||
| let entry = acc.entry(item.id.as_str()).or_insert((0.0, u8::MAX)); | ||
| entry.0 += 1.0 / (k + rank); | ||
| entry.1 = entry.1.min(item.depth); |
There was a problem hiding this comment.
When a source leg emits the same ScoredId.id more than once in a single ranked list (for example, SPO/neighbour hits can surface multiple relations to the same entity before caller-side dedup), this loop credits every occurrence with another 1/(k + rank) contribution as if it came from an independent ranking system. RRF should give each system/list at most one vote per id, using the best/first rank; otherwise duplicates from one leg can swamp consensus across the other fused legs.
Useful? React with 👍 / 👎.
Codex P2 review on #724: the accumulation loop credited every occurrence of an id within a single ranked list with another 1/(k+rank). A leg that surfaces the same entity more than once (several relations to one node, before caller-side dedup) would double-count and swamp consensus across the other fused legs. Fix: per-list dedup — each list votes once per id at its best (first) rank. Depth still folds the shallowest across all occurrences. +2 regression tests (duplicate_id_in_one_leg_votes_once, duplicate_id_still_folds_shallowest_depth). 9/9 rrf lib tests + doctest green; fmt + clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1
2 participants
What
Adds
reciprocal_rank_fusion— the retrieval keystone named in the GraphRAG representations inventory (#723, the SAP "Practical GraphRAG" reader's headline gap). Every ranked leg already exists (Bm25Index::rank,PersonalizedPageRank::ranked, CAM-PQ), but nothing fused them; the G0 harness compared vector-vs-graph by rank position instead of combining them.The primitive
arigraph/rrf.rs:Reciprocal Rank Fusion (Cormack, Clarke & Büttcher, SIGIR 2009): each id scored by
Σ_lists 1/(k + rank)(1-based). It fuses by rank position, never by the source scores — which is exactly why it combines lists whose scores are not commensurable: BM25 (tf-idff64), PPR (unit-sum probability), CAM-PQ (i8distance) share no scale, yet their rank orders fuse cleanly.BTreeMapid-ascending + stable score-descending sort).depthwins across occurrences (strongest provenance).ScoredId— the canonical cross-method result type.arigraph/mod.rs.Scope
Pure, reversible capability landed ahead of G0 — same pattern as
Bm25Index/PersonalizedPageRank/Communities. It computes a fused ranking and reads no carrier state. Wiring it intoOsintRetriever::retrieve(so the retriever actually fuses its legs) stays gated on the G0 load-bearing verdict (plan §5, STATUS_BOARD D-GR-2).Tests
cargo clippy -p lance-graph --libclean (the 8 warnings are pre-existingblasgraph/ndarray_bridge.rsSIMD dead-code).Board hygiene (same commit)
STATUS_BOARD
D-GR-2arow + AGENT_LOG entry.🤖 Generated with Claude Code
https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1
Generated by Claude Code