Skip to content

feat(pypi): store dist URLs relative to their common directory in the facts - #4535

Open
solosito wants to merge 22 commits into
bazel-contrib:mainfrom
solosito:atroya/upstream-facts-url-prefix
Open

solosito wants to merge 22 commits into
bazel-contrib:mainfrom
solosito:atroya/upstream-facts-url-prefix

Conversation

@solosito

Copy link
Copy Markdown

DO NOT MERGE: awaiting #4534

Stacked on #4534. Review only the commits from feat(pypi): store dist urls relative to their common directory in the facts onward.

The MODULE.bazel.lock facts key each distribution file by its full URL. An index usually serves all files of a package from one directory, so every entry repeats the same prefix. This makes the lockfile larger.

Fact version v5 maps the longest common directory of each package's URLs to the entries. Each entry is keyed by its URL relative to that directory. The extension expands the URLs when it reads the facts and compacts them only when it writes the lockfile. The rest of the code still sees full URLs. A malformed dist fact, for example after a botched merge, counts as absent, so the extension fetches the index again.

The common directory depends on all URLs of a package. If a package goes from one file to more than one, the directory can get shorter, and every entry of that package changes in the lockfile. On PyPI, the files of a package are almost always in different hash directories, so the common directory of a package with more than one file is almost always https://files.pythonhosted.org/packages/. A version change then keeps that directory.

The extension does not read v4 facts, so the first run after the upgrade fetches the index again. If you use --lockfile_mode=error, update the lockfile once.

Example

The numbers come from the same root module as in #4534: a copy of tests/integration/bzlmod_lockfile with Python 3.13, the default platforms, and a requirements lock for the newest release of twelve packages. The baseline is #4534, so the table shows only the effect of this change.

#4534 (v4) This change (v5)
MODULE.bazel.lock 70,635 bytes 66,387 bytes
pip extension facts 34,423 bytes 30,079 bytes

The facts for requests 2.34.2:

Before
"requests@https://pypi.org/simple": {
  "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl": "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0",
  "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz": "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed"
}
After
"requests@https://pypi.org/simple": {
  "https://files.pythonhosted.org/packages/": {
    "a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl": "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0",
    "ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz": "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed"
  }
}

The facts keep each sdist and each wheel that a configured pip.default platform can select for a requested Python version. The host is not used, so all hosts write the same lockfile. The facts record the platforms in wheel_filter, and facts filtered for other platforms are not read.
Wheel tags do not encode the patch version, so a new patch version must not invalidate the facts. Update the facts of the bzlmod_lockfile integration test.
Whether a wheel is selectable depends only on its python, ABI, and platform tags. Cache the result per tag set for the whole extension run, so that select_whl runs once per tag set instead of once per wheel.
Check that a wheel that only one platform selects is kept, that yank reasons of kept wheels survive the facts round trip, and that wheels with the same tags get the same result across packages.
Update the wheel_filter entries for the current default pip platforms.
simpleapi_download rebuilt each result without the flag, so
parse_requirements warned about every wheel digest that the facts
left out on purpose.
If the filter dropped every dist of a requested version, the facts
lacked that version and every run fetched the index again. Keep the
dists of such a version, so that the facts stay complete.
A hub with a `uv_lock` does not read the index, so its Python
version only added wheels that no hub uses. It also tied the facts
of every user to the Python version of the internal
`rules_python_publish_deps` hub.
The facts now lack the wheels that no configured platform selects.
An older rules_python ignores `wheel_filter`, so it must not read
these facts as complete.
Name `hub_builder.bzl%hub_platforms` in the docstring, say that
all sdists are kept, and note the one-time fetch after the upgrade.
Keep the new lines within the 100 column limit.
… facts

Fact version v5 stores the longest common directory of each package's dist urls under the "" key, and the urls relative to it. The facts in memory keep the full urls, so only the facts written to the lockfile change.
The common directory of the dist urls now ends before any query string or fragment. A url that ends with "/" no longer becomes the common directory, so its relative key is never "" and does not overwrite the common directory.
A dist fact that is not a dict, or has no string common directory under the "" key, is now treated as absent. Before, its urls were read as relative urls.
Bazel's lockfile merge driver merges the facts of an extension shallowly, so it replaces a package's dist fact as a whole.
…tory

Store each package's dist urls as a dict keyed by their common directory instead of under a `""` key. The layout explains itself and needs no special key.
The facts_cache docstring said to persist its in-memory facts, which keep
the full URLs. facts_cache now returns the lockfile facts itself, so no
caller can persist the uncompacted shape.
Use "URL" as the rest of the file does, and wrap the long schema line.
State the real merge invariant: each group has its own common
directory, so any union of groups expands to the full URLs.
The facts move to v5, so the first run after the upgrade fetches the
index again. Tell --lockfile_mode=error users to update the lockfile.
Keep the code and docstrings within 100 columns, and use "URL" as the
rest of the file does.
Check that different schemes, or absolute and relative URLs, give the
empty common directory. Check that a dist fact with more than one
directory group, as a lockfile merge can make, expands to the full URLs.
Store the facts with fact version v5.
@solosito

Copy link
Copy Markdown
Author

I prepared this PR with help from an AI coding agent. I, the human author, review and approve every agent-drafted reply and change before it is posted. No agent replies on this PR on its own.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant