refactor(pypi): reuse the same whl_library instances#3856
Conversation
Implements the first phase of the single-dep whl_library optimization, gated by RULES_PYTHON_WHL_LIBRARY_OPTIMIZED=1. What's implemented: - Feature flag struct and env var plumbing in internal_config_repo.bzl - documented in docs/environment-variables.md - exported in docs/readthedocs_build.sh - whl_repo_name.bzl: optimized=True skips target_platforms suffix in repo names - extension.bzl: threads whl_library_optimized through config -> hub_builder - hub_builder.bzl: optimized mode creates single spoke per wheel (no per-version/platform duplication), passes optimized=True to whl_repo_name - whl_library.bzl: loads rp_config, passes provides_extra+optimized to build file generation - generate_whl_library_build_bazel.bzl: _RENDER dict updated with provides_extra/optimized fields - whl_library_targets.bzl: _whl_library_targets_from_requires_optimized generates per-extra py_library targets (pkg__extra) with marker-based selects combining deps from base and extra requirements - Tests: optimized naming tests in whl_repo_name_tests.bzl; fixed hub_builder_tests.bzl config structs to include whl_library_optimized Not yet implemented (next commits): - Hub aliases for pkg[], pkg[extra] - pip_repository.bzl WORKSPACE-mode changes - unified_hub_repo.bzl changes - Integration tests for optimized mode Part of the plan in plan.md for single-dep whl_library restructuring.
When RULES_PYTHON_WHL_LIBRARY_OPTIMIZED=1: - hub_builder.bzl: tracks per-wheel extras parsed from requirement_line, includes whl_extras in build output struct - extension.bzl: threads hub_whl_extras through parse_modules -> _pip_impl - hub_repository.bzl: accepts new whl_extras attribute, passes to alias rendering - render_pkg_aliases.bzl: generates extra alias targets for pkg[extra] and pkg[] pointing to spoke pkg__extra and pkg targets respectively The hub now creates per-extra alias directories (e.g. requests_security/BUILD.bazel) for each requested extra, plus a pkg__/ alias for no-extras access. The main pkg alias continues to work as before (select-based to the spoke). Not yet implemented (follow-up): - unified_hub_repo.bzl extras forwarding - unified_hub_setup.bzl extras handling - pip_repository.bzl WORKSPACE-mode mirror - Integration tests for optimized mode Part of plan.md for single-dep whl_library restructuring.
When RULES_PYTHON_WHL_LIBRARY_OPTIMIZED=1, the unified @pypi hub now creates extra alias targets for pkg[extra] and pkg[] access patterns, mirroring what the per-hub hub_repository does. extension.bzl: _create_unified_hub_repo iterates mods.hub_whl_extras and adds pkg__extra and pkg__ aliases to the extra_aliases dict for each hub that contains the wheel. Part of plan.md for single-dep whl_library restructuring.
There was a problem hiding this comment.
Code Review
This pull request implements an optimized mode for whl_library (gated by the RULES_PYTHON_WHL_LIBRARY_OPTIMIZED environment variable) to allow wheel reuse across different Python versions by omitting the Python version from spoke repository names, generating per-extra targets, and creating explicit aliases in the hub repository. The code review identified several critical issues with this implementation: static aliases in render_pkg_aliases.bzl break multi-version/multi-platform setups, and iterating over extras_info causes aliases to overwrite each other. Additionally, a naming mismatch in extension.bzl between the unified hub alias and the hub repository package name will result in broken aliases, and skipping pkg__extra target generation when there are no extra dependencies leads to build failures.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| def _render_extra_alias(*, name, repo, target): | ||
| return """\ | ||
| package(default_visibility = ["//visibility:public"]) | ||
|
|
||
| alias( | ||
| name = "pkg", | ||
| actual = "@{repo}//:{target}", | ||
| ) | ||
|
|
||
| alias( | ||
| name = "whl", | ||
| actual = "@{repo}//:{target}", | ||
| ) | ||
| """.format( | ||
| repo = repo, | ||
| target = target, | ||
| ) |
There was a problem hiding this comment.
The _render_extra_alias function generates a static alias pointing to a single spoke repository. However, in a multi-version or multi-platform setup, there are multiple spoke repositories. We should replace this with a multiplatform alias generator that uses select to choose the correct spoke repository based on the active configuration.
def _render_extra_alias_multiplatform(*, name, repo_mapping, target_suffix):
if type(repo_mapping) == type(""):
actual_expr = repr("@{}//:{}".format(repo_mapping, target_suffix))
load_statement = ""
else:
actual_dict = {}
for config_setting, repo_name in repo_mapping.items():
key = _repr_config_setting(config_setting)
actual_dict[key] = repr("@{}//:{}".format(repo_name, target_suffix))
if len(actual_dict) == 1 and list(actual_dict.keys())[0] == repr("//conditions:default"):
actual_expr = list(actual_dict.values())[0]
else:
sorted_pairs = sorted(actual_dict.items())
actual_expr = "select({\n"
for k, v in sorted_pairs:
actual_expr += " {}: {},\n".format(k, v)
actual_expr += " })"
needs_load = any(["whl_config_setting" in k for k in actual_dict.keys()])
load_statement = ""
if needs_load:
load_statement = 'load("@rules_python//python/private/pypi:whl_config_setting.bzl", "whl_config_setting")\n'
return """\
{load_statement}package(default_visibility = ["//visibility:public"])
alias(
name = "pkg",
actual = {actual_expr},
)
alias(
name = "whl",
actual = {actual_expr},
)
""".format(
load_statement = load_statement,
actual_expr = actual_expr,
)
| # Generate extra alias directories for optimized mode extras. | ||
| for name, extras_info in whl_extras.items(): | ||
| normalized = normalize_name(name) | ||
| for repo_name, extra_names in extras_info.items(): | ||
| for extra in extra_names: | ||
| extra_pkg = "{}_{}".format(normalized, extra) | ||
| files["{}/BUILD.bazel".format(extra_pkg)] = _render_extra_alias( | ||
| name = extra_pkg, | ||
| repo = repo_name, | ||
| target = "{}__{}".format(normalized, extra), | ||
| ) | ||
|
|
||
| # Also create pkg__ alias (no extras) | ||
| no_extras_pkg = "{}__".format(normalized) | ||
| files["{}/BUILD.bazel".format(no_extras_pkg)] = _render_extra_alias( | ||
| name = no_extras_pkg, | ||
| repo = repo_name, | ||
| target = normalized, | ||
| ) |
There was a problem hiding this comment.
In a multi-version or multi-platform setup, whl_extras contains multiple spoke repositories for the same wheel. Iterating over extras_info.items() and writing directly to files causes the alias for one spoke to overwrite the others, breaking multi-version/multi-platform support for extras. We should use the multiplatform alias generator to create a single select-based alias that correctly routes to the appropriate spoke repository.
# Generate extra alias directories for optimized mode extras.
for name, extras_info in whl_extras.items():
normalized = normalize_name(name)
pkg_aliases = aliases.get(normalized)
if not pkg_aliases:
continue
# Find all unique extras across all repos for this wheel
unique_extras = {}
for extra_names in extras_info.values():
for extra in extra_names:
unique_extras[extra] = True
for extra in unique_extras:
extra_pkg = "{}_{}".format(normalized, extra)
files["{}/BUILD.bazel".format(extra_pkg)] = _render_extra_alias_multiplatform(
name = extra_pkg,
repo_mapping = pkg_aliases,
target_suffix = "{}__{}".format(normalized, extra),
)
# Also create pkg__ alias (no extras)
no_extras_pkg = "{}__".format(normalized)
files["{}/BUILD.bazel".format(no_extras_pkg)] = _render_extra_alias_multiplatform(
name = no_extras_pkg,
repo_mapping = pkg_aliases,
target_suffix = normalized,
)
| norm_pkg = normalize_name(whl_name) | ||
| for extra_names in extras_info.values(): | ||
| for extra in extra_names: | ||
| alias_name = "%s__%s" % (norm_pkg, extra) |
There was a problem hiding this comment.
There is a naming mismatch between the unified hub alias name and the hub repository package name. The unified hub alias is generated with double underscores __ (e.g., requests__security), whereas render_pkg_aliases.bzl generates the hub repository package with a single underscore _ (e.g., requests_security). This mismatch will cause the unified hub aliases to point to non-existent packages. Update this to use a single underscore to match the hub repository package naming.
| alias_name = "%s__%s" % (norm_pkg, extra) | |
| alias_name = "%s_%s" % (norm_pkg, extra) |
| if not extra_only_deps and not extra_only_deps_select: | ||
| continue |
There was a problem hiding this comment.
Skipping the generation of the pkg__extra target when there are no extra dependencies (or when all extra dependencies are already present in the base dependencies) will cause the hub repository's alias to point to a non-existent target, leading to build failures. We should always generate the pkg__extra target, even if it only depends on ":pkg", to ensure that the hub aliases remain valid.
Summary:
opencode, continue tomorrow.Fixes #2948