refactor(pypi): reuse the same whl_library instances

docs/environment-variables.md

@@ -131,7 +131,24 @@ is not resolved.
 
 :::
 
-:::{envvar} RULES_PYTHON_REPO_DEBUG
+:::{envvar} RULES_PYTHON_WHL_LIBRARY_OPTIMIZED
+
+When `1`, the whl_library will use optimized naming and target generation:
+* Spoke repository names no longer include the Python version, allowing
+  wheel reuse across different Python versions.
+* Per-extra targets (`pkg__extra`) are generated in the spoke repository
+  instead of bundling all extras into a single target.
+* The hub repository creates explicit aliases mapping `pkg`, `pkg[]`,
+  and `pkg[extra]` to the appropriate spoke targets.
+
+Defaults to `0` if unset.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+
+:::
+
+::{envvar} RULES_PYTHON_REPO_DEBUG
 
 When `1`, repository rules will print debug information about what they're
 doing. This is mostly useful for development to debug errors.

docs/readthedocs_build.sh

@@ -1,5 +1,8 @@
 #!/usr/bin/env bash
 
+# Exercising the whl_library optimized mode to ensure it is working
+export RULES_PYTHON_WHL_LIBRARY_OPTIMIZED=1
+
 set -eou pipefail
 
 declare -a extra_env

plan.md

@@ -0,0 +1,131 @@
+# Single dep for spoke repos per `whl_name`.
+
+The goal is to have `whl_library` instances from wheels (where we pass `url` for a URL or we pass
+a `whl_file` label which points to an actual wheel) to be reused across different python versions
+and across different configurations where different extras are requested. This means that
+if the user is using the same wheel in 2 different configurations, the wheel is reused, because
+the extracted contents should not differ. This could be important for:
+* platform-specific wheels for custom platforms defined by the user. Otherwise the
+  platform-specific wheel behaviour should remain unchanged.
+* cross-platform wheels for are the main affected item here.
+
+The code should be gated by a feature flag, which can be flipped to be enabled using an
+environmental variable via 
+
+The file to modify `python/private/internal_config_repo.bzl`. Use the 
+`RULES_PYTHON_WHL_LIBRARY_OPTIMIZED=1` to enable the feature. Add it to the
+`docs/readthedocs_build.sh` file so that we are exercising the code. And document this in
+the `docs/environment-variables.md` file.
+If the flag is off, then it is equivalent to the current
+`main` branch behaviour, if it is on, then it is equivalent to turning all of the behaviour
+implemented here on.
+
+Since this is a refactor for users not using private APIs, the env variable will be switched/removed
+later together with old code.
+
+All of the code should be written in a TDD style where we add a failing test for a feature and then
+we implement the new code.
+
+## Change the naming of the spoke repositories:
+
+> The hub repositories should be named `<hub_name>_<wheel_name_suffix>.
+
+The names should be changed:
+* from `+pip+dev_pip_314_requests_py3_none_any_2a0d60c1_linux_x86_64_linux_x86_64_freethreaded`
+  to `+pip+dev_pip_requests_py3_none_any_2a0d60c1`
+* from `+pip+dev_pip_314_roman_numerals_py3_none_any_647ba99c`
+  to `+pip+dev_pip_roman_numerals_py3_none_any_647ba99c`
+* from `+pip+dev_pip_314_markupsafe_cp314_cp314t_manylinux_2_17_x86_64_fed51ac4`
+  to `+pip+dev_pip_markupsafe_cp314_cp314t_manylinux_2_17_x86_64_fed51ac4`
+
+So the naming convention after the change is:
+`<prefix>_<name>_<py_tag>_<abi_tag>_<platform_tag>_<sha256[:8]>`
+
+Where `<prefix>` is the hub repository name and the rest of the segments come from the wheel name
+itself.
+
+The sdist building `whl_library` instances should change the naming to:
+`<prefix>_<name>_<sha256[:8]>_<rules_python_target_platform>`
+
+The file to modify `python/private/pypi/whl_repo_name.bzl`
+
+## Change the `whl_library_targets`
+
+Using the `METADATA` file that is parsed from `whl_metadata` function where the `Provides-Extra` is
+retrieved from the file. Right now we generate a single target which includes all of the extras that
+are required. Instead do the following target generation:
+* `pkg` (no extras) - target with no extras, this is the main target that includes the python sources,
+  the other targets just include extra dependencies.
+* `pkg__extra` - target for each extra in the provides-extra list. If the target depends on
+  `self[another_extra]`, then explode the nodes so that the target only depends on `pkg` target +
+  extra dependencies. Use `py_library` for this. Also be smart about generating targets:
+    - If the only extra is in the env marker, but the dependency is not in the hub repo, then skip
+      generating the whole `pkg__extra` target.
+    - Propose any extra ideas here.
+
+Related files:
+* `python/private/pypi/whl_library.bzl`
+* `python/private/pypi/whl_metadata.bzl`
+* `python/private/pypi/whl_library_targets.bzl`
+
+## Change `hub_builder` and `hub_repository`
+
+Instead of passing the `requirement[extra1,extra2]` to the `whl_library`, pass it to the
+`hub_repository` via `whl_config_setting` so that the `render_pkg_aliases` is using the information to alias to the right
+extra target based on what is requested. This should retain the behaviour where different target
+platforms are allowed to target `foo[baz]` and `foo[bar]` and we select the
+`@dev_pip_spoke_repo//:pkg__baz` and `@dev_pip_spoke_repo//:pkg__bar` appropriately.
+
+We should also generate extra targets here:
+* `pkg` should point to the target with all specified extras as passed to the `hub_repository`. This is to keep backwards compatibility with how it used to be done
+  previously. This also keeps compatibility with `rules_pycross`. Add this as a comment
+  when doing this. The spoke repos should not be used directly, so this difference in
+  behaviour is OK.
+  This reuses the targets declared in the hub repository described below.
+  It is backed by a `py_library` target if there are multiple extras, or an `alias` if it
+  is only a single extra. It is the same target as described below.
+* `pkg[]` should point to the target in the spoke without extras
+* `pkg[extra]` should point to the target in the spoke with particular extras. Do this for each
+  provided extra. So for `requirements[extra1,extra2]` passed to the hub repo, 2 extra targets will
+  be created. If the extras have not been specified for certain platforms via the
+  `whl_config_setting`, then the `select` statement should raise an `error` for no match.
+  Document the mapping explicitly: hub `pkg[]` → spoke `pkg`, hub `pkg[extra]` → spoke `pkg__extra`.
+  The fact that some of the targets in the spoke repos are unreachable is intentional - this is
+  because the hub repository contents are created before the `whl_library` downloads the whl and
+  inspects the METADATA.
+  The reverse direction is not a concern because the requirements file locking results in
+  consistent file. If something like this happens, then we should provide a message to the
+  user that something went wrong and that they should create a ticket in rules_python bug
+  tracker.
+* If there is `requirement[extra1,extra2]` passed, that means that we should create a special alias
+  that includes both of the targets at once. For this use `py_library` instead of `alias` and pass
+  `:pkg[extra1]` and `:pkg[extra2]` to the `deps` of the `py_library`. This won't cause
+  a failure because for a particular platform that is requested this should work by
+  construction, because the dependency targets will be also defined for the particular platform.
+
+* If the `requirement` is passed without any extras, then hub `pkg` should alias to `pkg[]`.
+* If the target is `py_library`, then we should name it without `[]` to avoid any aspects traversing
+  `py_library` targets changing behaviour. Only alias targets can have `[]`.
+
+Related files:
+* `python/private/pypi/render_pkg_aliases.bzl`
+* `python/private/pypi/hub_builder.bzl`
+* `python/private/pypi/hub_repository.bzl`
+* `python/private/pypi/whl_config_setting.bzl` - consider extending this struct to specify which
+  extras are requested for which platform.
+
+## Modify `pip_repository`
+
+Do similar changes to the `WORKSPACE` code to ensure easier maintenance of `whl_library` so that all
+code paths to `whl_library` remain consistent.
+
+File:
+* `python/private/pypi/pip_repository.bzl`
+
+## Modify `unified_hub_repo`
+
+Pass the extras to the `unified_hub_repo` as well so that the extra targets are created. The target
+topology in the unified hub repo should correspond to the hub_repository.
+
+File:
+* `python/private/pypi/unified_hub_repo.bzl`

python/private/internal_config_repo.bzl

@@ -24,12 +24,16 @@ load(":repo_utils.bzl", "repo_utils")
 _ENABLE_DEPRECATION_WARNINGS_ENVVAR_NAME = "RULES_PYTHON_DEPRECATION_WARNINGS"
 _ENABLE_DEPRECATION_WARNINGS_DEFAULT = "0"
 
+_WHL_LIBRARY_OPTIMIZED_ENVVAR_NAME = "RULES_PYTHON_WHL_LIBRARY_OPTIMIZED"
+_WHL_LIBRARY_OPTIMIZED_DEFAULT = "0"
+
 _CONFIG_TEMPLATE = """
 config = struct(
   build_python_zip_default = {build_python_zip_default},
   supports_whl_extraction = {supports_whl_extraction},
   enable_pystar = True,
   enable_deprecation_warnings = {enable_deprecation_warnings},
+  whl_library_optimized = {whl_library_optimized},
   bazel_8_or_later = {bazel_8_or_later},
   bazel_9_or_later = {bazel_9_or_later},
   bazel_10_or_later = {bazel_10_or_later},
@@ -102,6 +106,7 @@ def _internal_config_repo_impl(rctx):
     rctx.file("rules_python_config.bzl", _CONFIG_TEMPLATE.format(
         build_python_zip_default = repo_utils.get_platforms_os_name(rctx) == "windows",
         enable_deprecation_warnings = _bool_from_environ(rctx, _ENABLE_DEPRECATION_WARNINGS_ENVVAR_NAME, _ENABLE_DEPRECATION_WARNINGS_DEFAULT),
+        whl_library_optimized = _bool_from_environ(rctx, _WHL_LIBRARY_OPTIMIZED_ENVVAR_NAME, _WHL_LIBRARY_OPTIMIZED_DEFAULT),
         builtin_py_info_symbol = builtin_py_info_symbol,
         builtin_py_runtime_info_symbol = builtin_py_runtime_info_symbol,
         supports_whl_extraction = str(supports_whl_extraction),

python/private/pypi/extension.bzl

@@ -194,13 +194,16 @@ def _configure(config, *, override = False, **kwargs):
 def build_config(
         *,
         module_ctx,
-        enable_pipstar_extract):
+        enable_pipstar_extract,
+        whl_library_optimized = False):
     """Parse 'configure' and 'default' extension tags
 
     Args:
         module_ctx: {type}`module_ctx` module context.
         enable_pipstar_extract: {type}`bool | None` a flag to also not pass Python
             interpreter to `whl_library` when possible.
+        whl_library_optimized: {type}`bool` If True, use optimized whl_library
+            naming and target generation.
 
     Returns:
         A struct with the configuration.
@@ -261,6 +264,7 @@ def build_config(
             for name, values in defaults["platforms"].items()
         },
         enable_pipstar_extract = enable_pipstar_extract,
+        whl_library_optimized = whl_library_optimized,
         toml_decode = toml.decode,
     )
 
@@ -269,6 +273,7 @@ def parse_modules(
         _fail = fail,
         simpleapi_download = simpleapi_download,
         enable_pipstar_extract = False,
+        whl_library_optimized = False,
         **kwargs):
     """Implementation of parsing the tag classes for the extension and return a struct for registering repositories.
 
@@ -277,6 +282,8 @@ def parse_modules(
         simpleapi_download: Used for testing overrides
         enable_pipstar_extract: {type}`bool` a flag to enable dropping Python dependency for
             extracting wheels.
+        whl_library_optimized: {type}`bool` If True, use optimized whl_library
+            naming and target generation.
         _fail: {type}`function` the failure function, mainly for testing.
         **kwargs: Extra arguments passed to the hub_builder.
 
@@ -314,7 +321,11 @@ You cannot use both the additive_build_content and additive_build_content_file a
                 srcs_exclude_glob = whl_mod.srcs_exclude_glob,
             )
 
-    config = build_config(module_ctx = module_ctx, enable_pipstar_extract = enable_pipstar_extract)
+    config = build_config(
+        module_ctx = module_ctx,
+        enable_pipstar_extract = enable_pipstar_extract,
+        whl_library_optimized = whl_library_optimized,
+    )
 
     # TODO @aignas 2025-06-03: Merge override API with the builder?
     _overriden_whl_set = {}
@@ -432,6 +443,7 @@ You cannot use both the additive_build_content and additive_build_content_file a
     exposed_packages = {}
     extra_aliases = {}
     whl_libraries = {}
+    hub_whl_extras = {}
     for hub in pip_hub_map.values():
         out = hub.build()
 
@@ -445,6 +457,7 @@ You cannot use both the additive_build_content and additive_build_content_file a
         extra_aliases[hub.name] = out.extra_aliases
         hub_group_map[hub.name] = out.group_map
         hub_whl_map[hub.name] = out.whl_map
+        hub_whl_extras[hub.name] = out.whl_extras
 
     return struct(
         config = config,
@@ -454,6 +467,7 @@ You cannot use both the additive_build_content and additive_build_content_file a
         facts = simpleapi_cache.get_facts(),
         hub_group_map = hub_group_map,
         hub_whl_map = hub_whl_map,
+        hub_whl_extras = hub_whl_extras,
         whl_libraries = whl_libraries,
         whl_mods = whl_mods,
         platform_config_settings = {
@@ -492,6 +506,34 @@ def _create_unified_hub_repo(mods):
                 if hub_name not in extra_aliases[qual_alias]:
                     extra_aliases[qual_alias].append(hub_name)
 
+        # Add optimized mode extras to extra_aliases for the unified hub.
+        for whl_name, extras_info in mods.hub_whl_extras.get(hub_name, {}).items():
+            norm_pkg = normalize_name(whl_name)
+            for extra_names in extras_info.values():
+                for extra in extra_names:
+                    alias_name = "%s__%s" % (norm_pkg, extra)
+                    qual_alias = "%s:%s" % (alias_name, "pkg")
+                    if qual_alias not in extra_aliases:
+                        extra_aliases[qual_alias] = []
+                    if hub_name not in extra_aliases[qual_alias]:
+                        extra_aliases[qual_alias].append(hub_name)
+
+                    # Also add whl, data, dist_info aliases for the extra
+                    for std_alias in ["whl", "data", "dist_info"]:
+                        qual = "%s:%s" % (alias_name, std_alias)
+                        if qual not in extra_aliases:
+                            extra_aliases[qual] = []
+                        if hub_name not in extra_aliases[qual]:
+                            extra_aliases[qual].append(hub_name)
+
+                # Add pkg__ alias (no extras)
+                no_extras_alias = "%s__" % norm_pkg
+                qual_alias = "%s:%s" % (no_extras_alias, "pkg")
+                if qual_alias not in extra_aliases:
+                    extra_aliases[qual_alias] = []
+                if hub_name not in extra_aliases[qual_alias]:
+                    extra_aliases[qual_alias].append(hub_name)
+
     unified_hub_repo(
         name = "pypi",
         default_hub = mods.default_hub or (hubs[0] if hubs else ""),
@@ -569,6 +611,7 @@ def _pip_impl(module_ctx):
     mods = parse_modules(
         module_ctx,
         enable_pipstar_extract = rp_config.bazel_8_or_later,
+        whl_library_optimized = rp_config.whl_library_optimized,
     )
 
     # Build all of the wheel modifications if the tag class is called.
@@ -589,6 +632,7 @@ def _pip_impl(module_ctx):
             packages = mods.exposed_packages.get(hub_name, []),
             platform_config_settings = mods.platform_config_settings.get(hub_name, {}),
             groups = mods.hub_group_map.get(hub_name),
+            whl_extras = mods.hub_whl_extras.get(hub_name, {}),
         )
 
     _create_unified_hub_repo(mods)

python/private/pypi/generate_whl_library_build_bazel.bzl

@@ -27,6 +27,8 @@ _RENDER = {
     "extras": render.list,
     "group_deps": render.list,
     "include": str,
+    "optimized": repr,
+    "provides_extra": render.list,
     "requires_dist": render.list,
     "srcs_exclude": render.list,
     "tags": render.list,