Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add environment variable to disable writing installer metadata files #8877

Merged

Conversation

adisbladis
Copy link
Contributor

@adisbladis adisbladis commented Nov 7, 2024

Summary

This change introduces the UV_NO_INSTALLER_METADATA environment variable
as a way to opt out of the extra installer metadata files that uv is creating.

This is important to achieve reproducible builds in distribution
packaging, allowing to replace usage of
installer with uv pip install.

At the time of writing these files are:

  • uv_cache.json
    Contains timestamps which are non-reproducible.
    These hashes also leak in to the RECORD file.

  • direct_url.json
    Contains the path to the installed wheel.
    While not non-reproducible it's not required for distribution packaging.

  • INSTALLER
    Again, not non-reproducible, but of no value in distribution packaging.

Test Plan

Automated test added.

@adisbladis adisbladis force-pushed the reproducible-builds-no-dist-info branch 6 times, most recently from aaa8644 to 0b53fc9 Compare November 7, 2024 05:16
@zanieb
Copy link
Member

zanieb commented Nov 7, 2024

Thanks for putting up the PR!

Note to other reviewers, we briefly discussed using an environment variable for this in Discord.

I don't have strong opinions on the name, but we might want to use UV_NO_EXTRA_DIST_INFO=1 rather than UV_EXTRA_DIST_INFO=0?

Otherwise, I'm not well suited to be the primary reviewer for this. Perhaps @konstin or @charliermarsh would be interested.

@adisbladis adisbladis force-pushed the reproducible-builds-no-dist-info branch from 0b53fc9 to 94ebfbb Compare November 7, 2024 05:33
@adisbladis
Copy link
Contributor Author

I don't have strong opinions on the name, but we might want to use UV_NO_EXTRA_DIST_INFO=1 rather than UV_EXTRA_DIST_INFO=0?

It made more sense to me when writing this to think about it the other way around, but it's indeed seems more consistent with other options to use UV_NO_EXTRA_DIST_INFO=1.

@adisbladis adisbladis force-pushed the reproducible-builds-no-dist-info branch 3 times, most recently from 7d992a0 to 64977b4 Compare November 7, 2024 05:50
@konstin
Copy link
Member

konstin commented Nov 7, 2024

What's the motivation for going through wheel installation for repackaging over re-zipping the wheel into the target format you are interested in?

@adisbladis
Copy link
Contributor Author

adisbladis commented Nov 7, 2024

What's the motivation for going through wheel installation for repackaging over re-zipping the wheel into the target format you are interested in?

Sorry, I don't understand the question? I'll explain my use case in more detail, hopefully we'll find some understanding.

The use case for this PR is to replace usage of installer with uv in distribution packaging scripts.
These scripts need to:

  • Invoke a Python build system, such as pypa/build which takes a source tree and outputs a wheel
    • Or download a pre-built wheel
  • Invoke an installer that can install said wheel into a prefix

In nixpkgs we use:

  • pypa/installer to install Python wheels.
  • pypa/build to build Python packages from source
  • In some rare cases download pre-built wheels

For distribution packaging we want reproducible builds, meaning that the outputs produced by a packaging script should be bit-for-bit identical.
Nix comes with tooling to perform these checks: nix-build ./. -A somePackage --check.

I have implemented an alternative Python build infrastructure for Nix where I use uv.
These builds end up failing reproducibility checks because of extra dist info files.

At some point in the future I'd like to replace the nixpkgs install hook with a uv implementation too. A requisite for that is that outputs are reproducible.

@konstin konstin requested a review from charliermarsh November 7, 2024 12:45
@adisbladis adisbladis force-pushed the reproducible-builds-no-dist-info branch 2 times, most recently from ca26dd0 to 8313229 Compare November 12, 2024 00:39
@adisbladis
Copy link
Contributor Author

Ping @charliermarsh

@charliermarsh
Copy link
Member

I'm not super excited to maintain this but I see the value. I think we should call this "installer metadata" rather than "extra dist-info", since the latter is just a term we made up within the wheel installer crate. How's that sound?

@adisbladis adisbladis force-pushed the reproducible-builds-no-dist-info branch 3 times, most recently from aceed99 to 3cf1701 Compare November 22, 2024 05:38
@adisbladis
Copy link
Contributor Author

I'm not super excited to maintain this but I see the value.

Thank you. I understand that the use case is a bit niche from a Python perspective.

I think we should call this "installer metadata" rather than "extra dist-info", since the latter is just a term we made up within the wheel installer crate. How's that sound?

Yep, that sounds much better!

@adisbladis adisbladis changed the title feat: add environment variable to disable writing extra dist-info files feat: add environment variable to disable writing installer metadata files Nov 22, 2024
…files

This change introduces the `UV_NO_INSTALLER_METADATA` environment variable
as a way to opt out of the extra installer metadata files that `uv` is creating.

This is important to achieve reproducible builds in distribution
packaging, allowing to replace usage of
[installer](https://pypi.org/project/installer) with `uv pip install`.

At the time of writing these files are:
- `uv_cache.json`
    Contains timestamps which are non-reproducible.
    These hashes also leak in to the `RECORD` file.

- `direct_url.json`
    Contains the path to the installed wheel.
    While not non-reproducible it's not required for distribution packaging.

- `INSTALLER`
    Again, not non-reproducible, but of no value in distribution packaging.
@adisbladis adisbladis force-pushed the reproducible-builds-no-dist-info branch from 3cf1701 to 83374b1 Compare November 27, 2024 01:16
@adisbladis
Copy link
Contributor Author

Ping again @charliermarsh

@charliermarsh
Copy link
Member

Ack, will review (and hopefully merge) soon.

@charliermarsh charliermarsh enabled auto-merge (squash) December 4, 2024 01:18
@charliermarsh charliermarsh merged commit 28d4ef3 into astral-sh:main Dec 4, 2024
64 checks passed
adisbladis added a commit to pyproject-nix/pyproject.nix that referenced this pull request Dec 4, 2024
astral-sh/uv#8877 was merged, but the feature is not released yet.
Set the env var in preparation of the next release that contains the reproducibility fix.
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Dec 9, 2024
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.5.6` -> `0.5.7` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.5.7`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#057)

[Compare Source](astral-sh/uv@0.5.6...0.5.7)

##### Enhancements

-   Ignore dynamic version in source dist ([#&#8203;9549](astral-sh/uv#9549))
-   Improve build frontend error handling ([#&#8203;9611](astral-sh/uv#9611))
-   Un-hide `uv build --no-build-logs` option ([#&#8203;9642](astral-sh/uv#9642))
-   Flag version mismatch between sdist and wheel during `uv build` ([#&#8203;9633](astral-sh/uv#9633))
-   Improve message when updater receipt is for a different uv executable ([#&#8203;9487](astral-sh/uv#9487))
-   Add environment variable to disable writing installer metadata files ([#&#8203;8877](astral-sh/uv#8877))
-   Add managed downloads for the latest CPython releases: `3.9.21`, `3.10.16`, `3.11.11`, `3.12.8`, and `3.13.1` ([#&#8203;9696](astral-sh/uv#9696))

##### Preview features

-   Build backend: Add hint on import with preview disabled ([#&#8203;9691](astral-sh/uv#9691))
-   Build backend: Add direct builds to the resolver and installer ([#&#8203;9621](astral-sh/uv#9621))
-   Build backend: Add integration test for scripts ([#&#8203;9635](astral-sh/uv#9635))
-   Build backend: Add template to `uv init` ([#&#8203;9661](astral-sh/uv#9661))
-   Build backend: Add `--list` option ([#&#8203;9610](astral-sh/uv#9610))

##### Bug fixes

-   Create missing parent directories for output file of `uv export` / `uv pip compile` ([#&#8203;9648](astral-sh/uv#9648))
-   Fix missing display of non-freethreaded Python 3.13 in `python list` ([#&#8203;9669](astral-sh/uv#9669))
-   Implement `Ord` and `PartialOrd` without origin for `Requirement` ([#&#8203;9624](astral-sh/uv#9624))
-   Include more sources to avoid lowest bound warning ([#&#8203;9644](astral-sh/uv#9644))
-   Respect build tag priority in `uv.lock` ([#&#8203;9677](astral-sh/uv#9677))

##### Documentation

-   Add `build-essentials` note to build failures doc ([#&#8203;9641](astral-sh/uv#9641))
-   Add entry-point for distroless image in GitLab documentation ([#&#8203;9093](astral-sh/uv#9093))
-   Add documentation for `uv python pin` without a `REQUEST` argument ([#&#8203;9631](astral-sh/uv#9631))
-   Add a link to `uv python pin` reference docs ([#&#8203;9630](astral-sh/uv#9630))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants