Skip to content

feat(pypi): add pip.configure to select the exposed packages of a hub - #4395

Open
aignas wants to merge 14 commits into
bazel-contrib:mainfrom
aignas:aignas.feat.dep-group-selectors
Open

aignas wants to merge 14 commits into
bazel-contrib:mainfrom
aignas:aignas.feat.dep-group-selectors

Conversation

@aignas

@aignas aignas commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Before this change a hub exposed all of the packages from its
requirements files or uv.lock file, including the transitive
dependencies, and there was no way to limit them to the direct
dependencies of a project.

This adds the pip.configure tag class, which limits the packages that
a hub exposes to the ones listed in the dependency groups of a
pyproject.toml file, i.e. the [project] dependencies,
[project.optional-dependencies] and PEP 735 [dependency-groups]
tables:

pip.parse(
    hub_name = "pypi_hub",
    pyproject_toml = "//:pyproject.toml",
    requirements_lock = "//:requirements_lock.txt",
)
pip.configure(
    hub_name = "pypi_hub",
    dependency_groups = ["default", "group:dev"],
)

The other packages of the hub are still used by the wheels that need
them, but their aliases are only visible to the wheel repositories, and
they are left out of the all_requirements lists and the unified
@pypi hub. The transitive dependencies of selected groups are not
supported, the only exception is ["*", "transitive"], which exposes
all of the packages of the hub. Only the root module and the module that
created the hub can configure it, and the root module wins if both do.
The hubs that are not configured are unchanged.

This is only supported with bzlmod; the WORKSPACE code is unchanged.

Work towards #3413

@aignas

aignas commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

This has been created using AI, but I will review code manually before marking
it as ready. This may be split into smaller pieces if needed. Whilst this
implements an API similar to rules_pycross, the prompt included only the part
of the README.

@aignas

aignas commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

Ran out of free tokens on Opencode :D. You may be surprised, but I do not own a
personal subscription to an Absolute Intelligence API. It makes my life more
interesting. :D

I'll continue tomorrow/later if nobody points their AI at this.

The main idea is that pip.configure is an incremental configuration of the
hub repository. With uv.lock files this is not that may be super useful as
requirements.txt were per-platform and may require extra flags to configure.
However, I imagine that there could be usecases like:

  • different dependency groups per platform used per platform - e.g. I have an
    optional dep on GPU pytorch and that is exposed only on linux. That way we
    could even generate target_compatible_with in those cases.
  • different requirements.txt files per platform is still a usecase as long as
    we support requirements.txt format.

…oups

Read the dependencies, the optional dependencies and the PEP 735
dependency groups from the pyproject.toml and add a helper to select the
packages using dependency group specifiers.
Add the `hide_unexposed_packages` attribute to the hub repository. When set,
the aliases of the packages that are not exposed are only visible to the
wheel repositories that depend on them.
Add the `pip.configure` tag class that limits the packages that a hub
exposes to the ones listed in the dependency groups of a `pyproject.toml`
file. The other packages of the hub are still used by the wheels that need
them, but their aliases are not public.
@aignas
aignas force-pushed the aignas.feat.dep-group-selectors branch from 25deb08 to 23127e5 Compare November 7, 2026 17:08
@aignas aignas changed the title feat(pypi): support using dependency groups from pyproject.toml to hide transitive deps feat(pypi): add pip.configure to select the exposed packages of a hub Nov 7, 2026
Otherwise the unified hub would point to the hidden aliases of the hub and
fail with a visibility error instead of the missing package error.
pip.configure with ["*", "transitive"] used the packages that are
available on all platforms, which made the platform-specific packages
private. Treat it as if the hub was not configured instead, which keeps
these packages public and out of all_requirements as before.
…igure

Replace the nested comprehensions and the long inline conditions with
explicit loops and use types.is_string.
Keep a single wiring test for pip.configure in the extension tests, as the
selection is tested in hub_exposure_tests, and test that the pip.default
pyproject_toml is used. Also link the issue in the news entry.
@aignas
aignas marked this pull request as ready for review November 9, 2026 03:51
@aignas
aignas requested a review from rickeylev as a code owner November 9, 2026 03:51
@aignas

aignas commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

PTAL

@rickeylev

Copy link
Copy Markdown
Collaborator

Why a new tag class instead of additional args on pip.parse?

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this just affects the visibility of targets? Not the actual dependency groups that are made available?

Do dependency groups affect resolution? i.e., if I lock a pyproject.toml, do I get different results if i specify "use group=dev" vs not specifying a group?

Is part of the idea of having the groups controlled by visibilty, isntead of config settings, that it avoids having to re-configure the graph? That is actually pretty appealing.

@aignas

aignas commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Is part of the idea of having the groups controlled by visibilty, isntead of config settings, that it avoids having to re-configure the graph? That is actually pretty appealing.

That was my initial understanding, yes, the graph is there and we just get subsets of it. Because the uv.lock has already been generated for the universe?

@rickeylev

Copy link
Copy Markdown
Collaborator

I updated the PR description to wrap lines at 72 columns and changed hub_name in the example snippet from the reserved "pypi" name to "pypi_hub" (matching docs/pypi/download.md).

This is an agent-generated comment unreviewed by the user.

)
if configured_hubs == None:
return None
exposed_packages.update(configured_hubs)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exposed_packages.update(configured_hubs) replaces out.exposed_packages with
packages filtered from out.whl_map.keys(). Platform- or version-specific
direct dependencies then enter all_*requirements in requirements.bzl,
breaking builds on platforms or Python versions without a matching wheel.

Suggested fix: Intersect exposed_packages with configured_hubs for
all_*requirements, and pass configured_hubs separately to hub_repository so
platform-specific direct dependencies keep public aliases:

for hub_name, hub_pkgs in configured_hubs.items():
    configured_set = {pkg: None for pkg in hub_pkgs}
    exposed_packages[hub_name] = [
        pkg
        for pkg in exposed_packages[hub_name]
        if pkg in configured_set
    ]

Where: python/private/pypi/extension.bzl:542-546,
python/private/pypi/hub_repository.bzl:29-34, 87-96

This is an agent-generated comment reviewed and approved by the user.

Comment on lines +47 to +54
keys = _match(spec, groups)
if not keys:
_fail("Unknown dependency group '{}'. Available groups: {}".format(
spec,
", ".join(sorted(groups)),
))
for key in keys:
selected[key] = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifiers ("optional:*", "group:*") fail with "Unknown dependency group"
when [project.optional-dependencies] or [dependency-groups] is omitted or
empty because _match returns []. Empty wildcard matches should succeed
with zero packages, matching ("default", "*") when [project] dependencies
is omitted.

Suggested fix: Handle wildcard specifiers in select_dependency_groups before
calling _match, and have _match return str | None for a single group:

for spec in specs:
    if spec in ("*", "optional:*", "group:*"):
        prefix = "" if spec == "*" else spec[:-1]
        for key in groups:
            if key.startswith(prefix):
                selected[key] = None
        continue
    key = _match(spec, groups)
    if not key:
        _fail("Unknown dependency group '{}'. Available groups: {}".format(
            spec,
            ", ".join(sorted(groups)),
        ))
        return None
    selected[key] = None

Where: python/private/pypi/dependency_groups.bzl:72-91

This is an agent-generated comment reviewed and approved by the user.

Comment on lines +56 to +69
# A self reference like `my-project[test]` selects the optional
# dependencies, but only one level deep.
project = normalize_name(pyproject.name or "")
packages = {}
for key in selected:
for req in groups[key]:
req = requirement(req)
if req.name_ != project:
packages[req.name_] = None
continue
for extra in req.extras:
for r in groups.get("optional:" + extra, []):
packages[requirement(r).name_] = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single-level self-reference expansion drops nested extras from composite
extras like my-project[all] and adds the project name itself to packages.
The user docs state that self-references select optional dependencies without
this single-level restriction.

Suggested fix: Expand self-referenced optional groups iteratively with a
queue bounded by len(groups):

project = normalize_name(pyproject.name or "")
packages = {}
queue = list(selected)
for _ in range(len(groups)):
    if not queue:
        break
    for req in groups[queue.pop(0)]:
        req = requirement(req)
        if req.name_ != project:
            packages[req.name_] = None
            continue
        for extra in req.extras:
            opt_key = "optional:" + extra
            if opt_key in groups and opt_key not in selected:
                selected[opt_key] = None
                queue.append(opt_key)

This is an agent-generated comment reviewed and approved by the user.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a test case for this if one doesn't already exist

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented iterative self-referential extra expansion in dependency_groups.bzl and added _test_self_reference_extras in dependency_groups_tests.bzl in 28b48d5.

This is an agent-generated comment unreviewed by the user.

Comment thread docs/pypi/download.md Resolved
Comment thread python/private/pyproject_utils.bzl Resolved
Comment thread python/private/pypi/hub_exposure.bzl Resolved
Comment thread python/private/pypi/hub_repository.bzl Resolved
Align the new `pip.configure` documentation, docstrings, and Starlark
files with repository style and documentation conventions.

Add `requires-python` to the `docs/pypi/download.md` example, move the
`pip.configure` section to top-level `##` heading order, clarify the
`pyproject_toml` attribute docstrings and `{ref}` links, document helper
arguments/returns and returned `struct` fields, flatten `hub_exposure`
guards, move module-level test constants before functions, and wrap
Starlark lines and `doc` strings to 80 columns.
Restore the `pip.default.pyproject_toml` and `pip.parse.pyproject_toml`
attribute docstrings to their original text per review decision.

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: adding pip.configure tag class: SGTM

When a non-aborting `_fail` callback is injected in tests,
`select_dependency_groups` and `_expand_dependency_groups` previously
continued executing after `_fail(...)`, causing `hub_exposure` to fail
on `None` attribute access or return partial results instead of `None`.

Return `None` immediately after each `_fail(...)` call in
`dependency_groups.bzl`, guard `selection == None` in
`hub_exposure.bzl`, and assert `None` return values in error tests.
Single-level self-reference expansion dropped nested extras from
composite extras such as `my-project[all]` and added the project name
itself to `packages`, and `dependency_groups` type comments omitted list
element types.

Expand self-referenced optional groups iteratively with a bounded queue
in `dependency_groups.bzl` (with unit tests covering nested, bare, and
cyclic self-references), and specify parameterized element types in
`pyproject_utils.bzl`, `dependency_groups.bzl`, and
`render_pkg_aliases.bzl`.

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had my agent do a bunch of cleanup. The two remaining comments: I think one is fixed, but unsure about the other. I'll let you @aignas decide.

otherwise lgtm

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants