Sharded dataset generation#73
Conversation
… sharding matches distconv spec
…layouts hash the same way
PatrickRMiles
force-pushed
the
miles30/sharded_sample_generation
branch
from
2ad7850 to
a6697cf
Compare
michaelmckinsey1
left a comment
michaelmckinsey1
left a comment
There was a problem hiding this comment.
I don't understand when unsharded "V2" datasets would be used? When would those be used? When under scale 11?
In general I think we should discuss the use of keeping around V1 and V2 support. Is it even feasible to fully migrate to V3 and deprecate?
| def _point_cloud_path(config, curr_category: int, curr_instance: int) -> str: | ||
| """Return the input point-cloud path for a fractal instance.""" | ||
|
|
||
| instances_dir = f"var{config.variance_threshold}/instances/np{config.point_num}" | ||
| return os.path.join( | ||
| str(config.fract_base_dir), | ||
| instances_dir, | ||
| f"{curr_category:06d}", | ||
| f"{curr_category:06d}_{curr_instance:04d}.npy", | ||
| ) | ||
|
|
||
|
|
||
| def _local_shape(slices): | ||
| """Return the local spatial shape described by shard slices.""" | ||
|
|
||
| return tuple(s.stop - s.start for s in slices) |
There was a problem hiding this comment.
For helpers that are invoked only once, I think we remove them and write the code inline. This PR is already adding many functions so I think the readability is decreased when over factoring.
@michaelmckinsey1 I figured leaving V2 support in would be good so we can reuse existing datasets. It does add complexity though, so I'm happy to remove it if you're cool with regenerating datasets for existing problem configs. I suppose since we already have good data up to scale 10, there won't be much opportunity to reuse existing V2 datasets anyway... |
2 participants
Summary
Adds v3 physical sharded dataset generation and loading, with shard files written to match the configured DistConv layout. This lets large samples be generated directly as per-rank physical shards instead of always writing full-volume samples, and should allow us to generate scale 11 data.
Changes
dc_num_shards/dc_shard_dimspairs reuse the same dataset cache.Validation
1x1x1,1x1x2,1x2x2, and1x1x4; all 18 pairwise comparisons passed.