Configuring multiple indexes with uv publish

[cthoyt]

This issue is related to #7676.

I would like to switch from twine to uv publish, but in my workflow, I first upload to Test PyPI then later to vanilla PyPI. This means that I need to have configuration for the separate username and token for each index.

Currently, the only way that I saw to get the username and token into uv publish is by an environment variable or passing separately as flags (ref Oct. 7th). This isn't so convenient to set and unset to different values during a single workflow, or require me to shuttle text through flags in a potentially messy/insecure way. Further, there are probably people who have private indexes they also want to quickly switch between vanilla PyPI, test PyPI, and their private indexes.

One solution from the PyPA was to keep track of different index-servers in the .pypirc file (ref). There have been reservations about relying on text-based configurations, but currently as it is, it's not comfortable to switch to uv publish (even though I want to!)

[cthoyt]

The solution to #8448 is awfully close to what’s needed here - it looks like there’s already a uv solution for specifying indexes now, so it would just require having some more dynamic env var names for the publish credentials in the same way as the index credentials

[konstin]

Can you use:

      - name: Publish to Test PyPI
        env:
          UV_PUBLISH_TOKEN: ${{ secrets.TEST_PYPI_API_TOKEN }}
        run: |
          uv publish --publish-url https://test.pypi.org/legacy/
      - name: Publish to PyPI
        env:
          UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
        run: |
          uv publish --publish-url https://upload.pypi.org/legacy/

or could you use keyring authentication, where credentials are tagged by URL?

[cthoyt]

hi @konstin, thanks for checking in on this. If I were using something like GitHub Actions for publishing, then this is a great solution, so thanks for writing that up. Typo note for the second publish-url (shouldn't have test.pypi, should just be pypi)

Unfortunately, I am not publishing from the GitHub actions workflow, but rather from my local machine. I get that there's a push against people doing it, but that's my workflow. I'm using twine in a tox configuration like this (abbreviated):

[testenv:release]
skip_install = true
deps =
    twine >= 1.5.0
    uv
commands =
    uv build --sdist --wheel --no-build-isolation
    twine upload --skip-existing dist/*

And I have a second environment that's basically the same but passes the --repository testpypi flag to twine:

[testenv:testrelease]
skip_install = true
deps =
    twine >= 1.5.0
    uv
commands =
    uv build --sdist --wheel --no-build-isolation
    twine upload --skip-existing --repository testpypi dist/*

If your perceptive is, uv shouldn't support this local workflow then I also understand. I think it would be a bummer though if I had to do some hacky things to get the credentials out of my .pypirc file to get it into the right environment variables for uv, though

Please let me know if there's more information you'd like from my side

[cthoyt]

This might be a wild idea to throw out there, but I was reading up on https://docs.astral.sh/uv/configuration/indexes/, which for now is for installing packages from certain indexes, but ultimately this is a way of defining a specific name and URL for an upload URL. Maybe this is a place there the publish URL could also be stored, then you could have in your pyproject.toml:

[[tool.uv.index]]
# Optional name for the index.
name = "testpypi"
# Required URL for the index.
url = "https://test.pypi.org/"
# Optional URL for uploading to the index
publish_url = "https://test.pypi.org/legacy/"

then, uv could get tricky and say, let's look for an environment variable like UV_PUBLISH_<index.name>_TOKEN which in this case would be UV_PUBLISH_TESTPYPI_TOKEN. This would follow the same idea like for providing credentials, which is written up lower on the same guide at https://docs.astral.sh/uv/configuration/indexes/#providing-credentials

It's not clear to me right now if the index username/password should be the same as the publish token and if those could be shared in the same environment variable as written up in the existing docs

edit: looks like something like this was also proposed before #8531 (comment)

[konstin]

While publishing from a local machine isn't the recommended workflow (less reproducibility, no auditability compared to CI runs, no trusted publishing, etc.), it is a supported workflow: We understand that not every project and setup is worth doing the extra work for publishing from CI.

Can you use the keyring integration? This stores the credentials in the system keyring instead of a plain text file. When using keyring, it store the token with the publish URL, so when you call publish with testpypi or pypi, it picks the right credentials.

[cthoyt]

I'm not familiar with keyring but if what you're saying is that you can use the publish_url as a "key" and then it associate the token with it, I think this is an excellent alternative. In this case, I would only suggest that we should write up a tutorial to show how using this is a solution for this multiple-indexes use case. Additionally, having a part specifically on migrating away from .pypirc would be great! I am happy to contribute this once I figure out how keyring works myself :)

[konstin]

Let me test run the keyring instructions here before I add them to the docs.

keyring is a PyPI package that is used to interact with the system keyring. The system keyring stores your passwords securely, depending on operating system and platform they are encrypted or even in a hardware enclave. The keyring package allows plugins to interface with external services, some alternative indexes only provide their credentials only through keyring plugins (e.g. keyrings.google-artifactregistry-auth or artifacts-keyring). For local publishing to e.g. PyPI, we only need the plain keyring package.

To use the keyring, install it as tool first:

uv tool install keyring

Enter the token you got on PyPI in the interactive prompt, while replacing PACKAGE_NAME with the name of your package. The package name is an unfortunate workaround because we're using the same username but different tokens per package, but keyring stores one password per URL/username combination.

keyring set https://test.pypi.org/legacy/?PACKAGE_NAME __token__
keyring set https://upload.pypi.org/legacy/?PACKAGE_NAME __token__

Then you can publish with:

uv publish --username __token__ --keyring-provider subprocess --publish-url https://test.pypi.org/legacy/?PACKAGE_NAME
uv publish --username __token__ --keyring-provider subprocess --publish-url https://upload.pypi.org/legacy/?PACKAGE_NAME

[cthoyt]

One bit of confusion here is why the package name has anything to do with authentication. Does this mean I have to configure keyring over and over for every package I want to publish?

I tried setting it for a package I wrote https://github.com/cthoyt/class-resolver with:

$ keyring set "https://test.pypi.org/legacy/?class-resolver" __token__

and then running the following:

$ uv publish --username __token__ --keyring-provider subprocess --publish-url https://test.pypi.org/legacy/
warning: `uv publish` is experimental and may change without warning
Publishing 10 files https://test.pypi.org/legacy/
Uploading class_resolver-0.5.2-py3-none-any.whl (28.1KiB)
error: Failed to publish `dist/class_resolver-0.5.2-py3-none-any.whl` to https://test.pypi.org/legacy/
  Caused by: Permission denied (status code 403 Forbidden): 403 Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for more information.

I tried both ways with keyring for class_resolver and class-resolver, that didn't change it.

uv version:

$ uv --version
uv 0.4.29 (85f9a0d0e 2024-10-30)

[konstin]

A limitation of the keyring is that it stores one password per URL/username key, so we have to modify the URL to allow different tokens for different packages. I've updated the example to include the correct URL.

We should also warn when using the keyring but not getting credentials from it.

[cthoyt]

Ahh, I see what's going on. It's also possible to use a URL that doesn't have the parameter set for the package if you just have a token used for everything. The updated you made in your comment before where you added the ?PACKAGE made this more clear - so for a potential tutorial, it makes sense to note that there's a generic solution (which may be insecure/less desirable) and a package-by-package solution too.

Confirming that the following worked

$ keyring set https://test.pypi.org/legacy/ __token__
Password for '__token__' in 'https://test.pypi.org/legacy/': 

$ uv publish --username __token__ --keyring-provider subprocess --publish-url https://test.pypi.org/legacy/
warning: `uv publish` is experimental and may change without warning
Publishing 10 files https://test.pypi.org/legacy/
Uploading class_resolver-0.5.2-py3-none-any.whl (28.1KiB)
Uploading class_resolver-0.5.2.dev0-py3-none-any.whl (27.8KiB)
Uploading class_resolver-0.5.2.dev0.tar.gz (41.3KiB)
Uploading class_resolver-0.5.2.tar.gz (41.8KiB)
Uploading class_resolver-0.5.3-py3-none-any.whl (28.5KiB)
Uploading class_resolver-0.5.3.tar.gz (42.3KiB)
Uploading class_resolver-0.5.4-py3-none-any.whl (28.6KiB)
Uploading class_resolver-0.5.4.tar.gz (42.4KiB)
Uploading class_resolver-0.5.5.dev0-py3-none-any.whl (28.7KiB)
Uploading class_resolver-0.5.5.dev0.tar.gz (42.5KiB)

See https://test.pypi.org/project/class_resolver/