uv exports incomplete requirements if using external packages

[Mateuscvieira]

This issue is linked to #7494 but doesn't have the same solution, or any solution I have found.

The problem

I can start my uv project with commands:

uv init
uv add "ceic-api-client<=2.7" --extra-index-url https://downloads.ceicdata.com/python

Then appending the extra index to the pyproject file like so:

[tool.uv]
extra-index-url = ["https://downloads.ceicdata.com/python"]

This creates an environment where I can freely add, remove, run, sync and lock. So, running uv export, I would expect a valid requirements file, but I get the following (using the no-hashes flag for legibility):

ceic-api-client==2.6.1.148
certifi==2024.8.30
python-dateutil==2.9.0.post0
six==1.16.0
urllib3==2.2.3

This, for obvious reasons, will not build.

Suggestions

Maybe this is unwarranted advice, but from what I understand from src/commands/project/export.rs, the export command reads from the lockfile, not pyproject. Well, as mentioned in #7494, the lockfile is in fact correct when it comes to extra indexes, so it wouldn't require adding full support for adding indexes in add to fix this issue with export (I think).

Versions

Running uv 0.4.11 in Windows 10

[zanieb]

Can you clarify what the expected behavior is? We should write the index URL into the exported file?

[Mateuscvieira]

Yes. The requirements file should have the --extra-index-url flag if one wishes to use pip install

[zanieb]

I don't think we should do this by default, but maybe we could include --emit-index-urls. I'm pretty hesitant, generally, because our index behavior doesn't match pip and I'm not sure we will be able to generate something that always works.

[Mateuscvieira]

Shouldn't export always export environments that work, though? As it stands, even using uv pip install, it's broken:

uv export > ../another_directory/requirements.txt
cd ../another_directory
uv venv .venv
.venv\Scripts\activate
uv pip install -r requirements.txt

Output:

  × No solution found when resolving dependencies:
  ╰─▶ Because ceic-api-client was not found in the package registry and you require ceic-api-client==2.6.1.148, we can conclude that your requirements are unsatisfiable.

The export help file currently specifies it as "Export the project's lockfile to an alternate format". IMO it's to be expected that the output should be as close a match of the lockfile as possible. The output I showed isn't, as it won't even generate a working environment using any form of index behavior, be it uv or pip, while the lockfile does.

[zanieb]

I don't think passing settings inside the requirements.txt file (which is not a standardized format, it's just a thing that pip does) is best practice and we will generally avoid it where we can. Additionally, the requirements.txt format doesn't allow specifying an index for specific packages, so even if we were to start emitting index URLs now, once we support #171 via #7481 we won't be able to generate a file that "works".

I understand this isn't ideal and we may be able to make it better somehow, but it's not a simple problem.

[Mateuscvieira]

I understand the concern. Well, keeping the behavior as is, this maybe warrants a warning of some kind, either when the engine detects a package only available in a non-default index or simply when there are extra indexes in the lockfile, so this behavior is more explicit.
Also, maybe we could make uv export --extra-index-url prepend the extra index into the requirements file, maybe with a warning about unsafe behavior. This makes the user explicitly opt in, and also uses a flag that is (apparently) currently not used.

[charliermarsh]

uv pip compile also does not export these by default; the behavior is intentional. Whether we want to support --emit-index-url is an open question though.

[Mateuscvieira]

uv pip compile also does not export these by default; the behavior is intentional

I understand, but both uv export and uv pip compile have --extra-index-url flags which don't actually add the extra index to the output, and neither of them provide feedback about using the extra indexes in resolution but not in output. As is, the functionality is pretty confusing for us using private packages

[generatedone]

Hello @Mateuscvieira,

The following commands do the job for me.

uv venv
uv pip install --index https://downloads.ceicdata.com/python ceic_api_client

Please let me know if that is what you need.

In addition to that, I have couple of more questions to raise here:

1. Can you/sb compare it (kind of benchmark testing) with the regular installation way - using the pip install command?
2. Which are the overall benefits of using the uv package manager instead the pip?
3. Are there any new/enhanced features and functionalities that it provides (from the set of custom commands/options) that outperform their pip equivalent or are of greater convenience?
4. Does the uv manager allows any particular benefit when utilizing the ceic_api_client compared to pip?
5. Why have you even considered uv as an option as it is not described in any documentations of the package?
6. Are there any other tools (data manipulation/visualization, etc.) that you are using with that specific package? If yes - please list and explain each of them.

Apologies for the long list of questions but the discussed topic is very interesting.
I would appreciate if you manage to reply to some (if not all) of them.

[neggert]

I'll chime in supporting this. pip-compile deals with this by making it optional via the --emit-index-url flag. I'd love it if uv export supported something similar.

For my use case, I'm trying to export a requirements.txt file that I can pass to a remote Apache Ray cluster. Running uv on the other end to do this the "right" way isn't currently an option, but just adding in the --extra-index-url flags would solve the problem for 99% of cases.

My current workaround is to just manually edit the uv export output to add the index URLs, but that's not really a great solution.

[ties]

uv pip compile also does not export these by default; the behavior is intentional. Whether we want to support --emit-index-url is an open question though.

I would also really appreciate support for this feature. I want to generate requirements.txt from uv to replace the current requirements file and this blocks it. I am using a workaround in a pre-commit hook (gist).

Either just --emit-index-url or --emit-index-url --index-strategy unsafe-best-match (because of pip's effective resolution behaviour) would be perfect.

[tony-garcia]

+1 to adding --emit-index-url to uv export