Add CLI options for enhancing requests with HTTP headers

[amancevice]

Adressess issue #8042. Originally proposed in PR #8030 but more discussion was requested. This PR is a new attempt to add the barebones functionality requested without compromising on security.

This change adds a cURL-like -H / --header option to the CLI to allow users to enhance requests with custom HTTP headers.

To prevent pip from sending sensitive headers to unintended hosts, this option is ignored when extra index URLs are provided. The CLI will emit a warning to the user and any provided headers will be ignored if pip is configured to use multiple index URLs.

Example:

pip install \
  --index-url http://pypi.index/simple/ \
  --trusted-host pypi.index \
  -H 'X-Spam: ~*~ SPAM ~*~' \
  requests

[wisechengyi]

I am unable to stamp this, but verified this works great. Thanks!

[amancevice]

Any thoughts on this? @uranusjr @pradyunsg @NoahGorny ?

[NoahGorny]

I really like the idea, but I am only a simple commenter and can not approve this PR in any way :(

[uranusjr]

The logic looks good to me 🙂 A couple of notes on implementation.

[amancevice]

Thank you for your comments, @uranusjr ! I've made the requested changes; hope they are to your satisfaction.

[amancevice]

Any updates on this? Happy to discuss

[amancevice]

Hi, just pinging to inquire about where this PR stands.

[sbidoul]

I am personally not entirely comfortable with this, mostly for security reasons.

To prevent pip from sending sensitive headers to unintended hosts, this option is ignored when extra index URLs are provided.
The CLI will emit a warning to the user and any provided headers will be ignored if pip is configured to use multiple index URLs.

pip can do HTTP requests to many other hosts than indexes (--find-links, --requirements, and PEP 508 direct URLs come to mind).

So I tend to think this would require a per-host configuration. Or per index if we restrict this to indexes only, but that is probably too restrictive? I'm not sure how to do that elegantly (without going all the way to specifying a fully pluggable http session mechanism).

[pfmoore]

without going all the way to specifying a fully pluggable http session mechanism

... and frankly, a fully pluggable mechanism would enable solutions to many other open issues as well (Kerberos and NTLM support, integration with platform native networking). I don't want to derail a quick win by expanding the scope to something unachievable, but I do think it's time that we had a serious look at pushing the complexity and limited flexibility of pip's current network handling options out to a pluggable mechanism of some sort.

[amancevice]

I'm definitely sensitive to concerns about security, but by the same token (no pun intended) it would also be great to see pip support something more advanced than basic auth in the near term.

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

[connorlwilkes]

This is really useful for me. I am surprised that it doesn't currently exist within pip. Does anyone have any workarounds in the meantime?

What is the latest on this? Is this going to be merged or is it stuck? @amancevice Thanks

[amancevice]

Some security concerns were raised and because of that I think the devs want to take this feature in a different direction.

My personal feeling is that this is a case of "letting perfect be the enemy of good" but the security concerns are valid.

I'm happy to revise this PR if the devs decide they would like to move forward on it, but my guess is this is a dead end.

[pfmoore]

I'd say you should ignore the digression about plugins, but look at how to address the point that pip doesn't just make HTTP requests to indexes. So if we don't want to expose sensitive information to (say) the host serving a direct URL, how should we do that? Modelling the interface on curl may be a mistake, as curl only does a single request each run, whereas pip does a lot of requests as part of an install. Per-host extra headers seems more secure/reliable.

It's also worth noting that we haven't had many requests for a feature like this, so risking a security hole for something that may be of limited benefit is something we have to take seriously.

[amancevice]

thanks for the reply @pfmoore — I originally opened #8042 to open up the conversation as to what the interface could look like but it wasn't getting much traction (ie, I got antsy waiting for feedback 🙂) and opened this as a sort of minimum-viable implementation.

One of the proposals I put forward in #8042 was providing a command-line arg that accepted JSON in the form:

'{ "pypi.private.com": { "Authorization": "Bearer …", … } }'

And then pip would enhance any requests to pypi.private.com with the provided headers. Do you think this would be an acceptable second pass at this feature?

[pfmoore]

-1 on needing to enter JSON on the command line.

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

[pradyunsg]

Closing since this is fairly out of date and bitrotten.

Let's continue the discussion in #8042, since it's unclear if adding this functionality to pip would make sense. :)