feat: ranking and filtering for biolink:statistical_significance_qualifier - #135
Open
SkyeAv wants to merge 4 commits into
Conversation
New shepherd_utils/statistical_significance_qualifier.py module with: - SIGNIFICANCE_BAND_SCORES / SIGNIFICANCE_ORDINAL tables (biolink#1766) - SIGNIFICANCE_SOURCE_WEIGHT (0.5, conservative, matches RTX) - get_statistical_significance(edge): qualifiers-only lookup Shared by aragorn_score, arax_rank, and the filter worker. Tests: 7/7 passing. Refs: shepherd#134
…ranker aragorn_score/worker.py (shared Aragorn + BTE ranker): - Extract qualifier via get_statistical_significance() in get_edge_values() - Add as scored property: band_score x 0.5 trust -> admittance feed - not_significant / qualifier-less edges unchanged (no penalty) Tests: 35/35 passing (4 new). Refs: shepherd#134
arax_rank/ranker.py (dict-based port of RTX ARAX_ranker.py): - Categorical bypass: qualifier looked up separately from attributes - Additive score: band_score x SIGNIFICANCE_SOURCE_WEIGHT (0.5) - Injected at method level (works even for attribute-less edges) - Mirrors RTX post-refactor: standalone trust weight, not in known_attributes_to_trust Tests: 5/5 passing. Refs: shepherd#134
…orn + BTE New filter infrastructure (shepherd#134): - shepherd_utils/shared.py: filter_edges_by_statistical_significance() ordinal edge removal, qualifier-less edges kept, result pruning - workers/filter_edges_by_statistical_significance/: super-thin worker (async, inline, mirrors filter_kgraph_orphans) - compose.yml: service registration Wired into Aragorn + BTE lookup workflows (both branches each) with minimum_significance=suggestive (removes only not_significant edges). Inert until retriever#197 ships qualifier-bearing edges. Updated 3 existing workflow-assertion tests. Tests: 511/511 passing (8 new filter tests). Refs: shepherd#134
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds ranking boosts and an edge-level significance filter for
biolink:statistical_significance_qualifier(biolink/biolink-model#1766) across all Shepherd ARAs, modeled on the equivalent ARAX implementation (RTXteam/RTX#2859). Closes #134.Shared module
shepherd_utils/statistical_significance_qualifier.py(NEW): single source of truth forSIGNIFICANCE_BAND_SCORES(0.70…0.0),SIGNIFICANCE_ORDINAL(4…0),SIGNIFICANCE_SOURCE_WEIGHT(0.5), andget_statistical_significance(edge)— a qualifiers-only lookup that strips anybiolink:prefix. Reused by all three consumers below.Ranking —
aragorn_score(Aragorn + BTE)workers/aragorn_score/worker.py: extracts the qualifier inget_edge_values()via the shared helper (the attribute loop only walksedge["attributes"]; the qualifier lives inedge["qualifiers"]). Adds it as a scored property feeding the graph-Laplacian:band_score × 0.5→ admittance-1/log(weight). Max effective weight0.35→ bounded boost.aragorn_scoreis the shared Aragorn + BTE ranker (both workflows runaragorn.score); one change updates both. There is no separate BTE ranker.not_significantand qualifier-less edges are unchanged (property_w > 0guard, also avoidslog(0)).Ranking —
arax_rank(ARAX dict port)workers/arax_rank/ranker.py: categorical-bypass injection in_calculate_edge_confidence()— the qualifier is looked up separately from the numeric attribute normalizer (the enum string would otherwise hitfloat()→ValueError). Appendsband_score × SIGNIFICANCE_SOURCE_WEIGHTtoedge_attribute_score_list. Works even for attribute-less edges (injected at method level, not nested insideif edge.get("attributes")).known_attributes_to_trust(RTX moved it out in the same refactor).Filtering —
filter_edges_by_statistical_significanceshepherd_utils/shared.py:filter_edges_by_statistical_significance(message, minimum_significance, logger)— ordinal edge removal (edges explicitly below threshold removed; qualifier-less kept), with analysis/result pruning to keep TRAPI valid. Downstreamfilter_kgraph_orphansremoves orphaned nodes.workers/filter_edges_by_statistical_significance/(NEW): super-thin async worker (inline, no process pool — mirrorsfilter_kgraph_orphans). Readsminimum_significancefrom the nestedparametersdict (the olderfilter_*_top_nworkers read flat keys and silently use defaults; this one honors the parameter).compose.yml: service registered, mirroringfilter_kgraph_orphans.Workflow wiring
workers/aragorn/worker.py+workers/bte/worker.py: filter inserted afteraragorn.score, beforesort_results_score, in all four lookup branches (two per ARA). Pathfinder branches untouched.minimum_significance=suggestive: removes only edges explicitly labelednot_significant; qualifier-less edges kept.Design
edge["qualifiers"]exclusively — the qualifieris_a: statement_qualifier, so BMT/Retriever route it there. Matches ARAX (qualifier-only as of RTX#2859; its defensiveedge.attributeslookup was removed). A one-line fallback in the shared helper covers a KP ever sending it as an attribute.score_paths(currently purely structural);edge.attributesfallback; per-ARA threshold tuning.Testing
pytest tests/unit/ -q→511 passed(24 new tests: 7 shared-helper, 4 aragorn_score, 5 arax_rank, 8 filter).