Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use uv to manage and generate dependencies #4232

Merged
merged 6 commits into from
Dec 5, 2024
Merged

Use uv to manage and generate dependencies #4232

merged 6 commits into from
Dec 5, 2024

Conversation

theskumar
Copy link
Member

@theskumar theskumar commented Nov 28, 2024

  • Use frozen dependencies with hashes to ensure reproducible dependency installation in a secure way.
  • Uses uv and pre-commit to automatically generate the locked requirements files from pyproject.toml.
  • While the translation dependencies can be installed directly with uv, uv doesn’t seem to be able to add --find-link=… to the exported requirements/translate.txt, for that reason the translation dependencies are left as they are and won’t be automatically exported via pre-commit.
  • So whenever uv lock —upgrade is run, it will freeze the latest patch versions.
  • The production dependencies are specified with a ~ which means >= at the last digit of requirements.
  • The dev dependencies have >= to make them easy to update, if something breaks in dev either fix it or go back to the old frozen dependencies.
  • Moved generated dependencies in requirements/ folder.
  • Use --no-deps where applicable while installing these generated requirements as the sub-dependencies are already resolved.

Test Steps

  • Ensure the local and production builds are able to build.

@theskumar theskumar requested review from frjo and wes-otf November 28, 2024 08:04
@theskumar
Copy link
Member Author

theskumar commented Nov 28, 2024

@wes-otf I don’t have much context into the why we needed to add the explicit linux+cpu dependency, but if we would have to generate frozen requirements for translate as well, then doesn't it make sense to document the additional argument needed for pip? So instead of

pip install -r requirements-translate.txt

The documentation would say on Linux, use this instead.

pip install -r requirements-translate.txt --find-links https://download.pytorch.org/whl/torch_stable.html

@theskumar
Copy link
Member Author

theskumar commented Nov 28, 2024

Now that we have converted these into pyproject dependencies, running uv tree -d 1 --outdated lists these outdated packages with major or minor releases. These require manual bumping. All packages are up to date with security fixes.

Resolved 230 packages in 6.79s
hypha v5.19.0
├── babel v2.14.0 (latest: v2.16.0)
├── boto3 v1.34.162 (latest: v1.35.71)
├── celery v5.3.6 (latest: v5.4.0)
├── dj-database-url v2.1.0 (latest: v2.3.0)
├── django v4.2.16 (latest: v5.1.3)
├── django-anymail v10.3 (latest: v12.0)
├── django-countries v7.5.1 (latest: v7.6.1)
├── django-file-form v3.6.0 (latest: v3.8.0)
├── django-filter v23.5 (latest: v24.3)
├── django-fsm v2.8.2 (latest: v3.0.0)
├── django-hijack v3.4.5 (latest: v3.7.0)
├── django-select2 v8.1.2 (latest: v8.2.1)
├── django-two-factor-auth v1.16.0 (latest: v1.17.0)
├── drf-nested-routers v0.93.5 (latest: v0.94.1)
├── environs v11.0.0 (latest: v11.2.1)
├── gunicorn v22.0.0 (latest: v23.0.0)
├── heroicons v2.6.0 (latest: v2.9.0)
├── lark v1.1.9 (latest: v1.2.2)
├── more-itertools v10.2.0 (latest: v10.5.0)
├── phonenumberslite v8.13.50
├── pillow v10.3.0 (latest: v11.0.0)
├── psycopg[binary] v3.1.20 (latest: v3.2.3)
├── pwned-passwords-django v2.1 (latest: v5.1.1)
├── qrcode v7.4.2 (latest: v8.0)
├── reportlab v4.0.9 (latest: v4.2.5)
├── scout-apm v3.1.0 (latest: v3.2.1)
├── sentry-sdk v2.8.0 (latest: v2.19.0)
├── tablib v3.5.0 (latest: v3.7.0)
├── wagtail v5.2.7 (latest: v6.3.1)
├── whitenoise v6.6.0 (latest: v6.8.2)
├── xhtml2pdf v0.2.15 (latest: v0.2.16)
├── xmltodict v0.13.0 (latest: v0.14.2)

@frjo frjo added Status: Needs dev testing 🧑‍💻 Tasks that should be tested by the dev team Status: Needs testing Tickets that need testing/qa Type: Maintenance Type: Minor Minor change, used in release drafter labels Dec 3, 2024
@frjo
Copy link
Member

frjo commented Dec 3, 2024

The deploy to Hypha Test on Heroku went without issues.

@wes-otf
Copy link
Contributor

wes-otf commented Dec 3, 2024

I don’t have much context into the why we needed to add the explicit linux+cpu dependency, but if we would have to generate frozen requirements for translate as well, then doesn't it make sense to document the additional argument needed for pip?

@theskumar so this was specifically for the heroku deploy - I'm fine tossing whatever and making those options a branch-specific thing. whatever you think is best works for me!

@theskumar
Copy link
Member Author

I believe using the requirements/translate.txt in the un-genereated form is fine for now.

Updated django v4.2.16 -> v4.2.17
Updated httpx v0.27.2 -> v0.28.0
Updated mkdocs-material v9.5.46 -> v9.5.47
Updated nh3 v0.2.18 -> v0.2.19
Updated phonenumberslite v8.13.50 -> v8.13.51
Updated pillow-heif v0.20.0 -> v0.21.0
Updated protobuf v5.29.0 -> v5.29.1
Updated pytest v8.3.3 -> v8.3.4
Updated ruff v0.8.0 -> v0.8.1
Updated six v1.16.0 -> v1.17.0
@theskumar
Copy link
Member Author

theskumar commented Dec 5, 2024

Merged main and

  • Updated django v4.2.16 -> v4.2.17
  • Updated httpx v0.27.2 -> v0.28.0
  • Updated mkdocs-material v9.5.46 -> v9.5.47
  • Updated nh3 v0.2.18 -> v0.2.19
  • Updated phonenumberslite v8.13.50 -> v8.13.51
  • Updated pillow-heif v0.20.0 -> v0.21.0
  • Updated protobuf v5.29.0 -> v5.29.1
  • Updated pytest v8.3.3 -> v8.3.4
  • Updated ruff v0.8.0 -> v0.8.1
  • Updated six v1.16.0 -> v1.17.0

@frjo frjo removed Status: Needs testing Tickets that need testing/qa Status: Needs dev testing 🧑‍💻 Tasks that should be tested by the dev team labels Dec 5, 2024
@frjo frjo merged commit d62fca0 into main Dec 5, 2024
7 checks passed
@theskumar theskumar deleted the uv branch December 5, 2024 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Maintenance Type: Minor Minor change, used in release drafter
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants