Fix a partial feature coverage#704
Draft
kmontemayor2-sc wants to merge 3 commits into
Draft
Conversation
…in collate Heterogeneous datasets may carry node/edge features on only a subset of their types. Previously BaseDistNeighborSampler._collate_fn fetched features for every sampled node/edge type, so a featureless-but-reachable type raised a KeyError inside the sampling coroutine. GLT's event loop swallows that exception, drops the batch, and never sends a message, so the loader hangs indefinitely. Guard both hetero feature-fetch loops on membership in DistFeature.local_feature (the per-type feature dict), skipping types with no feature store entry. The all2all node path is left unguarded with a comment, since GiGL never enables use_all2all. Adds regression tests (bounded by join-with-timeout, since the pre-fix bug hangs) for partial edge-feature and node-feature coverage in the standard and ABLP loaders. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Direct DistPartitioner users may register different feature/label/weight type
sets on different ranks. Because each type is partitioned via its own
cross-rank collective, divergent registration desynchronizes those
collectives and hangs or corrupts partitioning.
Add per-category normalization helpers that all_gather each rank's
{type: (trailing_shape, dtype)} schema and per-type local entity counts (via
graphlearn_torch.distributed.rpc.all_gather, since torch.distributed is not
generally initialized during partitioning), then per type in the global union:
backfill an empty (0, dim) tensor on a rank missing it with zero local rows;
raise if a rank is missing it but has nonzero local rows (unreconstructable);
and raise on trailing-shape/dtype conflicts across ranks. All raise
conditions are evaluated from the gathered data so every rank raises
symmetrically instead of one raising while others hang.
Normalization runs from partition() and from the individual
partition_node_features_and_labels() / partition_edge_index_and_edge_features()
methods (including the range partitioner override), guarded by per-category
flags so the full partition() path does not gather twice.
Also fix partition_node_features_and_labels() to partition the union of node
feature and label types, so a label-only node type is no longer dropped when
some other node type carries features.
Adds tests for count-conditional backfill, nonzero-rows and dim-conflict
raises, the node feature/label union, a zero-local-rows featured type, and a
range-partitioner backfill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kmontemayor2-sc
force-pushed
the
fix-a-partial-feature-coverage
branch
from
July 14, 2026 15:17
3d8f56c to
048014b
Compare
Collaborator
Author
|
/all_test |
Contributor
GiGL Automation@ 15:52:30UTC : 🔄 @ 17:20:44UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 15:52:30UTC : 🔄 @ 15:54:21UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 15:52:31UTC : 🔄 @ 17:02:38UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 15:52:31UTC : 🔄 @ 16:01:05UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 15:52:33UTC : 🔄 @ 15:57:39UTC : ❌ Workflow failed. |
Contributor
GiGL Automation@ 15:52:34UTC : 🔄 @ 17:19:43UTC : ❌ Workflow failed. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets a heterogeneous dataset with node/edge features on only a subset of its types train
successfully, instead of hanging on a swallowed
KeyErrorinside the sampling coroutine.Two independent changes:
1. Skip feature fetch for featureless types in the sampler collate (
base_sampler.py).BaseDistNeighborSampler._collate_fnpreviously fetched features for every sampled node/edgetype. For a heterogeneous dataset with features on only some types, a featureless-but-reachable
type raised a
KeyErrorthat GLT'sConcurrentEventLoopswallows (logging one line, notraceback) — dropping every batch so the loader blocks forever. Both heterogeneous
feature-fetch loops now skip any type absent from
DistFeature.local_feature(the per-typefeature dict). The
use_all2allnode path is left unguarded with a comment, since it is neverenabled here.
2. Normalize divergent cross-rank feature registration in the partitioner
(
dist_partitioner.py,dist_range_partitioner.py). DirectDistPartitionerusers may registerdifferent feature/label/weight type sets per rank, which desynchronizes the per-type partitioning
collectives. New helpers
all_gathereach rank's{type: (trailing_shape, dtype)}schema andper-type local entity counts (via
graphlearn_torch.distributed.rpc.all_gather, sincetorch.distributedis not generally initialized during partitioning), then per type in theglobal union: backfill an empty
(0, dim)tensor on a rank missing it with zero local rows;raise if a rank is missing it but has nonzero local rows (unreconstructable); raise on
trailing-shape/dtype conflicts. Raise conditions are evaluated from the gathered data so every
rank raises symmetrically instead of one raising while the others hang. Normalization runs from
partition()and from the individualpartition_node_features_and_labels()/partition_edge_index_and_edge_features()methods, guarded by per-category flags to avoidgathering twice. Also fixes
partition_node_features_and_labelsto partition the union of nodefeature and label types, so a label-only node type is no longer dropped when another node type
carries features.
Tests
bounded by a join-with-timeout since the pre-fix bug hangs rather than errors.
feature/label union, a zero-local-rows featured type, and a range-partitioner backfill.
make type_checkpasses; the affected unit-test files pass(
distributed_neighborloader_test.py,dist_ablp_neighborloader_test.py,distributed_partitioner_test.py).Notes / out of scope
is not present on this branch (left as a follow-up).
DistDataseton a rank that receives zero edges of a type trips a pre-existing GLTempty-topology limitation (
coo_to_csrcallsmax()on an empty tensor), and the rangepartition book cannot represent an empty per-rank edge range. Both are orthogonal to this
change; the zero-local-rows test therefore asserts on the partition output's feature keys
rather than a full dataset build.