Skip to content

refactor(pypi): reuse the same whl_library instances#3856

Draft
aignas wants to merge 10 commits into
bazel-contrib:mainfrom
aignas:aignas.refactor.single_dep_whl_library
Draft

refactor(pypi): reuse the same whl_library instances#3856
aignas wants to merge 10 commits into
bazel-contrib:mainfrom
aignas:aignas.refactor.single_dep_whl_library

Conversation

@aignas

@aignas aignas commented Jun 27, 2026

Copy link
Copy Markdown
Collaborator

Summary:

  • plan.md
  • feat(pypi): add whl_library_optimized mode behind env var gate
  • feat(pypi): implement hub aliases for optimized whl_library extras
  • feat(pypi): forward optimized whl extras to unified hub repo
  • TODO: up-to-this done with free opencode, continue tomorrow.

Fixes #2948

aignas added 10 commits June 27, 2026 15:43
Implements the first phase of the single-dep whl_library optimization,
gated by RULES_PYTHON_WHL_LIBRARY_OPTIMIZED=1.

What's implemented:
- Feature flag struct and env var plumbing in internal_config_repo.bzl
- documented in docs/environment-variables.md
- exported in docs/readthedocs_build.sh
- whl_repo_name.bzl: optimized=True skips target_platforms suffix in repo names
- extension.bzl: threads whl_library_optimized through config -> hub_builder
- hub_builder.bzl: optimized mode creates single spoke per wheel (no
  per-version/platform duplication), passes optimized=True to whl_repo_name
- whl_library.bzl: loads rp_config, passes provides_extra+optimized to
  build file generation
- generate_whl_library_build_bazel.bzl: _RENDER dict updated with
  provides_extra/optimized fields
- whl_library_targets.bzl: _whl_library_targets_from_requires_optimized
  generates per-extra py_library targets (pkg__extra) with marker-based
  selects combining deps from base and extra requirements
- Tests: optimized naming tests in whl_repo_name_tests.bzl; fixed
  hub_builder_tests.bzl config structs to include whl_library_optimized

Not yet implemented (next commits):
- Hub aliases for pkg[], pkg[extra]
- pip_repository.bzl WORKSPACE-mode changes
- unified_hub_repo.bzl changes
- Integration tests for optimized mode

Part of the plan in plan.md for single-dep whl_library restructuring.
When RULES_PYTHON_WHL_LIBRARY_OPTIMIZED=1:
- hub_builder.bzl: tracks per-wheel extras parsed from requirement_line,
  includes whl_extras in build output struct
- extension.bzl: threads hub_whl_extras through parse_modules -> _pip_impl
- hub_repository.bzl: accepts new whl_extras attribute, passes to alias rendering
- render_pkg_aliases.bzl: generates extra alias targets for pkg[extra] and
  pkg[] pointing to spoke pkg__extra and pkg targets respectively

The hub now creates per-extra alias directories (e.g. requests_security/BUILD.bazel)
for each requested extra, plus a pkg__/ alias for no-extras access. The main
pkg alias continues to work as before (select-based to the spoke).

Not yet implemented (follow-up):
- unified_hub_repo.bzl extras forwarding
- unified_hub_setup.bzl extras handling
- pip_repository.bzl WORKSPACE-mode mirror
- Integration tests for optimized mode

Part of plan.md for single-dep whl_library restructuring.
When RULES_PYTHON_WHL_LIBRARY_OPTIMIZED=1, the unified @pypi hub now
creates extra alias targets for pkg[extra] and pkg[] access patterns,
mirroring what the per-hub hub_repository does.

extension.bzl: _create_unified_hub_repo iterates mods.hub_whl_extras
and adds pkg__extra and pkg__ aliases to the extra_aliases dict for
each hub that contains the wheel.

Part of plan.md for single-dep whl_library restructuring.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements an optimized mode for whl_library (gated by the RULES_PYTHON_WHL_LIBRARY_OPTIMIZED environment variable) to allow wheel reuse across different Python versions by omitting the Python version from spoke repository names, generating per-extra targets, and creating explicit aliases in the hub repository. The code review identified several critical issues with this implementation: static aliases in render_pkg_aliases.bzl break multi-version/multi-platform setups, and iterating over extras_info causes aliases to overwrite each other. Additionally, a naming mismatch in extension.bzl between the unified hub alias and the hub repository package name will result in broken aliases, and skipping pkg__extra target generation when there are no extra dependencies leads to build failures.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +90 to +106
def _render_extra_alias(*, name, repo, target):
return """\
package(default_visibility = ["//visibility:public"])

alias(
name = "pkg",
actual = "@{repo}//:{target}",
)

alias(
name = "whl",
actual = "@{repo}//:{target}",
)
""".format(
repo = repo,
target = target,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The _render_extra_alias function generates a static alias pointing to a single spoke repository. However, in a multi-version or multi-platform setup, there are multiple spoke repositories. We should replace this with a multiplatform alias generator that uses select to choose the correct spoke repository based on the active configuration.

def _render_extra_alias_multiplatform(*, name, repo_mapping, target_suffix):
    if type(repo_mapping) == type(""):
        actual_expr = repr("@{}//:{}".format(repo_mapping, target_suffix))
        load_statement = ""
    else:
        actual_dict = {}
        for config_setting, repo_name in repo_mapping.items():
            key = _repr_config_setting(config_setting)
            actual_dict[key] = repr("@{}//:{}".format(repo_name, target_suffix))

        if len(actual_dict) == 1 and list(actual_dict.keys())[0] == repr("//conditions:default"):
            actual_expr = list(actual_dict.values())[0]
        else:
            sorted_pairs = sorted(actual_dict.items())
            actual_expr = "select({\n"
            for k, v in sorted_pairs:
                actual_expr += "        {}: {},\n".format(k, v)
            actual_expr += "    })"

        needs_load = any(["whl_config_setting" in k for k in actual_dict.keys()])
        load_statement = ""
        if needs_load:
            load_statement = 'load("@rules_python//python/private/pypi:whl_config_setting.bzl", "whl_config_setting")\n'

    return """\
{load_statement}package(default_visibility = ["//visibility:public"])

alias(
    name = "pkg",
    actual = {actual_expr},
)

alias(
    name = "whl",
    actual = {actual_expr},
)
""".format(
        load_statement = load_statement,
        actual_expr = actual_expr,
    )

Comment on lines +158 to +176
# Generate extra alias directories for optimized mode extras.
for name, extras_info in whl_extras.items():
normalized = normalize_name(name)
for repo_name, extra_names in extras_info.items():
for extra in extra_names:
extra_pkg = "{}_{}".format(normalized, extra)
files["{}/BUILD.bazel".format(extra_pkg)] = _render_extra_alias(
name = extra_pkg,
repo = repo_name,
target = "{}__{}".format(normalized, extra),
)

# Also create pkg__ alias (no extras)
no_extras_pkg = "{}__".format(normalized)
files["{}/BUILD.bazel".format(no_extras_pkg)] = _render_extra_alias(
name = no_extras_pkg,
repo = repo_name,
target = normalized,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

In a multi-version or multi-platform setup, whl_extras contains multiple spoke repositories for the same wheel. Iterating over extras_info.items() and writing directly to files causes the alias for one spoke to overwrite the others, breaking multi-version/multi-platform support for extras. We should use the multiplatform alias generator to create a single select-based alias that correctly routes to the appropriate spoke repository.

    # Generate extra alias directories for optimized mode extras.
    for name, extras_info in whl_extras.items():
        normalized = normalize_name(name)
        pkg_aliases = aliases.get(normalized)
        if not pkg_aliases:
            continue

        # Find all unique extras across all repos for this wheel
        unique_extras = {}
        for extra_names in extras_info.values():
            for extra in extra_names:
                unique_extras[extra] = True

        for extra in unique_extras:
            extra_pkg = "{}_{}".format(normalized, extra)
            files["{}/BUILD.bazel".format(extra_pkg)] = _render_extra_alias_multiplatform(
                name = extra_pkg,
                repo_mapping = pkg_aliases,
                target_suffix = "{}__{}".format(normalized, extra),
            )

        # Also create pkg__ alias (no extras)
        no_extras_pkg = "{}__".format(normalized)
        files["{}/BUILD.bazel".format(no_extras_pkg)] = _render_extra_alias_multiplatform(
            name = no_extras_pkg,
            repo_mapping = pkg_aliases,
            target_suffix = normalized,
        )

norm_pkg = normalize_name(whl_name)
for extra_names in extras_info.values():
for extra in extra_names:
alias_name = "%s__%s" % (norm_pkg, extra)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There is a naming mismatch between the unified hub alias name and the hub repository package name. The unified hub alias is generated with double underscores __ (e.g., requests__security), whereas render_pkg_aliases.bzl generates the hub repository package with a single underscore _ (e.g., requests_security). This mismatch will cause the unified hub aliases to point to non-existent packages. Update this to use a single underscore to match the hub repository package naming.

Suggested change
alias_name = "%s__%s" % (norm_pkg, extra)
alias_name = "%s_%s" % (norm_pkg, extra)

Comment on lines +182 to +183
if not extra_only_deps and not extra_only_deps_select:
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Skipping the generation of the pkg__extra target when there are no extra dependencies (or when all extra dependencies are already present in the base dependencies) will cause the hub repository's alias to point to a non-existent target, leading to build failures. We should always generate the pkg__extra target, even if it only depends on ":pkg", to ensure that the hub aliases remain valid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Setup fewer wheel repos in pip.parse

1 participant