Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI cache pruning corrupts cache for other Python versions #8929

Closed
nijel opened this issue Nov 8, 2024 · 2 comments · Fixed by #8955
Closed

CI cache pruning corrupts cache for other Python versions #8929

nijel opened this issue Nov 8, 2024 · 2 comments · Fixed by #8955
Assignees
@charliermarsh
Labels
bug Something isn't working cache Caching of packages and metadata

Comments

@nijel
Copy link
Contributor

nijel commented Nov 8, 2024

We use multiple Python versions in one of our CI pipelines, but the uv cache is generated only for one of the Python versions. On subsequent runs when cache is used, the package build fails with:

Resolved 1 package in 107ms
error: Failed to prepare distributions
  Caused by: Failed to download and build `pycairo==1.27.0`
  Caused by: Attempted to re-extract the source distribution for `pycairo==1.27.0`, but the hashes didn't match. Run `uv cache clean` to clear the cache.

Clearing the cache indeed fixes the issue.

The issue does not exist without running uv cache prune --ci.

Reproduced with uv 0.4.29 and 0.5.0.

Reproducer:

#!/bin/sh

export UV_CACHE_DIR="$PWD/cache"

rm -rf "$UV_CACHE_DIR" .venv .venv-2 pyproject.toml uv.lock

uv init --name uv-cache-issue
uv add --python 3.13 "pycairo"

uv cache prune --ci

rm -rf .venv .venv-2

uv venv --python python3.11 .venv-2
. .venv-2/bin/activate
uv pip install "pycairo"
nijel added a commit to WeblateOrg/weblate that referenced this issue Nov 8, 2024
@nijel
fix(ci): use separate UV CI cache for migrations
4387fbb 
This job uses multiple Python versions and thus can be different from
the standard test cache.

This avoids astral-sh/uv#8929
@charliermarsh charliermarsh added bug Something isn't working cache Caching of packages and metadata labels Nov 8, 2024
@charliermarsh
Copy link
Member

Sounds like a bug though not obvious why this would happen.

@charliermarsh
Copy link
Member

I think the proximate problem is that uv add generates hashes by default but uv pip install does not.

@charliermarsh charliermarsh self-assigned this Nov 8, 2024
@charliermarsh charliermarsh mentioned this issue Nov 8, 2024
Merged
konstin pushed a commit that referenced this issue Nov 10, 2024
@charliermarsh @konstin
Compute superset of existing and required hashes when healing cache (#…
fa79d6e 
…8955)

## Summary

The basic issue here is that `uv add` will compute and store a hash for
each package. But if you later run `uv pip install` _after_ `uv cache
prune --ci`, we need to re-download the source distribution. After
re-downloading, we compare the hashes before and after. But `uv pip
install` doesn't compute any hashes by default. So the hashes "differ"
and we error.

Instead, we need to compute a superset of the already-existing and
newly-requested hashes when performing this re-download. (In practice,
this will always be SHA-256.)

Closes #8929.

## Test Plan

```shell
export UV_CACHE_DIR="$PWD/cache"

rm -rf "$UV_CACHE_DIR" .venv .venv-2 pyproject.toml uv.lock

echo $(uv --version)

uv init --name uv-cache-issue
cargo run add --python 3.13 "pycairo"

uv cache prune --ci

rm -rf .venv .venv-2

uv venv --python python3.11 .venv-2
. .venv-2/bin/activate
cargo run pip install "pycairo"
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@charliermarsh charliermarsh

Labels
bug Something isn't working cache Caching of packages and metadata
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Compute superset of existing and required hashes when healing cache astral-sh/uv
2 participants
@nijel @charliermarsh