Repository navigation
Conversation
|
This has been created using AI, but I will review code manually before marking |
|
Ran out of free tokens on Opencode :D. You may be surprised, but I do not own a I'll continue tomorrow/later if nobody points their AI at this. The main idea is that
|
…oups Read the dependencies, the optional dependencies and the PEP 735 dependency groups from the pyproject.toml and add a helper to select the packages using dependency group specifiers.
Add the `hide_unexposed_packages` attribute to the hub repository. When set, the aliases of the packages that are not exposed are only visible to the wheel repositories that depend on them.
Add the `pip.configure` tag class that limits the packages that a hub exposes to the ones listed in the dependency groups of a `pyproject.toml` file. The other packages of the hub are still used by the wheels that need them, but their aliases are not public.
25deb08 to
23127e5
Compare
Otherwise the unified hub would point to the hidden aliases of the hub and fail with a visibility error instead of the missing package error.
pip.configure with ["*", "transitive"] used the packages that are available on all platforms, which made the platform-specific packages private. Treat it as if the hub was not configured instead, which keeps these packages public and out of all_requirements as before.
…igure Replace the nested comprehensions and the long inline conditions with explicit loops and use types.is_string.
Keep a single wiring test for pip.configure in the extension tests, as the selection is tested in hub_exposure_tests, and test that the pip.default pyproject_toml is used. Also link the issue in the news entry.
|
PTAL |
|
Why a new tag class instead of additional args on pip.parse? |
rickeylev
left a comment
There was a problem hiding this comment.
So this just affects the visibility of targets? Not the actual dependency groups that are made available?
Do dependency groups affect resolution? i.e., if I lock a pyproject.toml, do I get different results if i specify "use group=dev" vs not specifying a group?
Is part of the idea of having the groups controlled by visibilty, isntead of config settings, that it avoids having to re-configure the graph? That is actually pretty appealing.
That was my initial understanding, yes, the graph is there and we just get subsets of it. Because the |
|
I updated the PR description to wrap lines at 72 columns and changed This is an agent-generated comment unreviewed by the user. |
| ) | ||
| if configured_hubs == None: | ||
| return None | ||
| exposed_packages.update(configured_hubs) |
There was a problem hiding this comment.
exposed_packages.update(configured_hubs) replaces out.exposed_packages with
packages filtered from out.whl_map.keys(). Platform- or version-specific
direct dependencies then enter all_*requirements in requirements.bzl,
breaking builds on platforms or Python versions without a matching wheel.
Suggested fix: Intersect exposed_packages with configured_hubs for
all_*requirements, and pass configured_hubs separately to hub_repository so
platform-specific direct dependencies keep public aliases:
for hub_name, hub_pkgs in configured_hubs.items():
configured_set = {pkg: None for pkg in hub_pkgs}
exposed_packages[hub_name] = [
pkg
for pkg in exposed_packages[hub_name]
if pkg in configured_set
]Where: python/private/pypi/extension.bzl:542-546,
python/private/pypi/hub_repository.bzl:29-34, 87-96
This is an agent-generated comment reviewed and approved by the user.
| keys = _match(spec, groups) | ||
| if not keys: | ||
| _fail("Unknown dependency group '{}'. Available groups: {}".format( | ||
| spec, | ||
| ", ".join(sorted(groups)), | ||
| )) | ||
| for key in keys: | ||
| selected[key] = None |
There was a problem hiding this comment.
Specifiers ("optional:*", "group:*") fail with "Unknown dependency group"
when [project.optional-dependencies] or [dependency-groups] is omitted or
empty because _match returns []. Empty wildcard matches should succeed
with zero packages, matching ("default", "*") when [project] dependencies
is omitted.
Suggested fix: Handle wildcard specifiers in select_dependency_groups before
calling _match, and have _match return str | None for a single group:
for spec in specs:
if spec in ("*", "optional:*", "group:*"):
prefix = "" if spec == "*" else spec[:-1]
for key in groups:
if key.startswith(prefix):
selected[key] = None
continue
key = _match(spec, groups)
if not key:
_fail("Unknown dependency group '{}'. Available groups: {}".format(
spec,
", ".join(sorted(groups)),
))
return None
selected[key] = NoneWhere: python/private/pypi/dependency_groups.bzl:72-91
This is an agent-generated comment reviewed and approved by the user.
| # A self reference like `my-project[test]` selects the optional | ||
| # dependencies, but only one level deep. | ||
| project = normalize_name(pyproject.name or "") | ||
| packages = {} | ||
| for key in selected: | ||
| for req in groups[key]: | ||
| req = requirement(req) | ||
| if req.name_ != project: | ||
| packages[req.name_] = None | ||
| continue | ||
| for extra in req.extras: | ||
| for r in groups.get("optional:" + extra, []): | ||
| packages[requirement(r).name_] = None | ||
|
|
There was a problem hiding this comment.
Single-level self-reference expansion drops nested extras from composite
extras like my-project[all] and adds the project name itself to packages.
The user docs state that self-references select optional dependencies without
this single-level restriction.
Suggested fix: Expand self-referenced optional groups iteratively with a
queue bounded by len(groups):
project = normalize_name(pyproject.name or "")
packages = {}
queue = list(selected)
for _ in range(len(groups)):
if not queue:
break
for req in groups[queue.pop(0)]:
req = requirement(req)
if req.name_ != project:
packages[req.name_] = None
continue
for extra in req.extras:
opt_key = "optional:" + extra
if opt_key in groups and opt_key not in selected:
selected[opt_key] = None
queue.append(opt_key)This is an agent-generated comment reviewed and approved by the user.
There was a problem hiding this comment.
Add a test case for this if one doesn't already exist
There was a problem hiding this comment.
I implemented iterative self-referential extra expansion in dependency_groups.bzl and added _test_self_reference_extras in dependency_groups_tests.bzl in 28b48d5.
This is an agent-generated comment unreviewed by the user.
Align the new `pip.configure` documentation, docstrings, and Starlark
files with repository style and documentation conventions.
Add `requires-python` to the `docs/pypi/download.md` example, move the
`pip.configure` section to top-level `##` heading order, clarify the
`pyproject_toml` attribute docstrings and `{ref}` links, document helper
arguments/returns and returned `struct` fields, flatten `hub_exposure`
guards, move module-level test constants before functions, and wrap
Starlark lines and `doc` strings to 80 columns.
Restore the `pip.default.pyproject_toml` and `pip.parse.pyproject_toml` attribute docstrings to their original text per review decision.
rickeylev
left a comment
There was a problem hiding this comment.
re: adding pip.configure tag class: SGTM
When a non-aborting `_fail` callback is injected in tests, `select_dependency_groups` and `_expand_dependency_groups` previously continued executing after `_fail(...)`, causing `hub_exposure` to fail on `None` attribute access or return partial results instead of `None`. Return `None` immediately after each `_fail(...)` call in `dependency_groups.bzl`, guard `selection == None` in `hub_exposure.bzl`, and assert `None` return values in error tests.
Single-level self-reference expansion dropped nested extras from composite extras such as `my-project[all]` and added the project name itself to `packages`, and `dependency_groups` type comments omitted list element types. Expand self-referenced optional groups iteratively with a bounded queue in `dependency_groups.bzl` (with unit tests covering nested, bare, and cyclic self-references), and specify parameterized element types in `pyproject_utils.bzl`, `dependency_groups.bzl`, and `render_pkg_aliases.bzl`.
Before this change a hub exposed all of the packages from its
requirements files or
uv.lockfile, including the transitivedependencies, and there was no way to limit them to the direct
dependencies of a project.
This adds the
pip.configuretag class, which limits the packages thata hub exposes to the ones listed in the dependency groups of a
pyproject.tomlfile, i.e. the[project] dependencies,[project.optional-dependencies]and PEP 735[dependency-groups]tables:
The other packages of the hub are still used by the wheels that need
them, but their aliases are only visible to the wheel repositories, and
they are left out of the
all_requirementslists and the unified@pypihub. The transitive dependencies of selected groups are notsupported, the only exception is
["*", "transitive"], which exposesall of the packages of the hub. Only the root module and the module that
created the hub can configure it, and the root module wins if both do.
The hubs that are not configured are unchanged.
This is only supported with bzlmod; the
WORKSPACEcode is unchanged.Work towards #3413