Unable to prompt user for credentials for private pypi repo

[tmct]

Description

Hi - I was wondering if you could help work out the best way to get this functionality working again please. I'm happy to write a PR to fix this if we can agree on a good solution.

Issue

I wish to install a package from a password-protected private pypi repo, from my Windows machine, and for pip install to prompt me for my username and password to achieve this - this used to work.

The command I believe is correct for this, which on 21.0.1 successfully prompts me for a username and password, is:

pip install --extra-index-url https://username:[email protected]/api/pypi/pypi-this-particular-repo/simple my-desired-package

n.b. these are the literal strings "username" and "password" rather than real credentials.

In all versions I have tried since then, and on main, no password is requested and I instead proceed to the "Could not find a version that satisfies..." error.

Analysis of network/auth.py

My understanding of the prompting code is that it gets triggered by handle_401, the 401 response having been triggered by the initial request, as it was attempted with Basic Auth but with incorrect credentials. In 21.0.1, I get to _prompt_for_password, but in recent versions it never gets there, because the additional call to _get_new_credentials added in 886275d returns the literal strings "username" and "password" once more, and so skips the prompt check.

One's first thought might be to try to remove "username:password@" from the index URL, in the hope this is no longer needed. Alas, under this input, pip install's behaviour is to not wrap the initial request in Basic Auth at all, and so handle_401 is never reached (in the case of our particular of private repos at least.)

It is of course possible that I am formatting this index URL incorrectly or otherwise mistaken - please let me know if so! But otherwise I would like to work out how to fix this behaviour, without breaking anyone else's scenarios.

The intent of 886275d seems clear enough: to move the keyring call later, after 401s only, for performance. But it seems to have had the side effect of making the deliberately-incorrect, literal username and password be re-tried in that scenario, instead of prompting for something. I can't see this being desired, or indeed this ever succeeding.

There are two calls to _get_url_and_credentials, the "happy path" one and the "post-401" one, distinguished by each having exactly one of allow_netrc and allow_keyring True - this feels like sensible re-use of code, given the common URL parsing at the start. There is a section of the method commented, "# If an index URL was found, try its embedded credentials", which can return, and does so on the path to reproducing this bug for me. I suggest that disallowing the return here in the "post-401" case may be a sufficient change to make the right fix here.

Firstly - do you think that this suggested change would be right to make? Below I will briefly consider how this might be implemented.

Options for altering _get_new_credentials

The smallest change that implements my suggestion would be to add another argument to _get_new_credentials, e.g. allow_url, whose Falseness would stop the # If an index URL was found, try its embedded credentials section from returning. I have implemented this on a branch: main...tmct:tmct/private-repo-prompt-1

However, we may wish to refactor _get_new_credentials and its calls - with this change it would effectively become two distinct functions, _get_new_credentials_from_url_netrc and _get_new_credentials_from_keyring - maybe moving explicitly to this pattern would make things more clear? (With common code in a third method, say)

Other

--index-url exhibits similar behaviour, and would be similarly affected by any change

Thanks

Expected behavior

See above - user should be prompted for username and password, as before. These should then be used successfully to authenticate.

pip version

21.0.1 < version <= current HEAD of main

Python version

3.8

OS

Windows 10

How to Reproduce

Run pip install command in Description above, pointed at a password-protected private pypi repo

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

One's first thought might be to try to remove "username:password@" from the index URL, in the hope this is no longer needed. Alas, under this input, pip install's behaviour is to not wrap the initial request in Basic Auth at all, and so handle_401 is never reached (in the case of our particular of private repos at least.)

Something is amiss here. Why does the handle_401 hook not called if the request is not done with basic auth? I’d think it should.

[piotr-dobrogost]

You might be interested in checking out the behaviour with the pending PR related to basic http auth in #10904.
I'm not saying what you raised here is not a genuine bug.

[potiuk]

I indeed believe #10904 should fix the problem indeed. @tmct - maybe you can check if it solves your problem ?