pip tries to install setuptools 45.0.0 on Python 2

[yoctozepto]

This is pip version of pypa/virtualenv#1493 and pypa/setuptools#1963
Please look there for more background.

I know Python 2 is EOL and killable so feel free to ignore but please notice this means pip does not behave as advertised.

Environment

• pip version: 19.3.1
• Python version: 2.7.17
• OS: Ubuntu 18.04

Description
setuptools 45.0.0 tries to install on Python 2 but it is not meant to be installed on Python < 3.5

Expected behavior
pip installs setuptools<45
per setuptools docs:

1458: Drop support for Python 2. Setuptools now requires Python 3.5 or later. Install setuptools using pip >=9 or pin to Setuptools <45 to maintain 2.7 support.

How to Reproduce

1. pip install setuptools

Output

http://paste.openstack.org/show/788302/

[uranusjr]

pip does it because setuptools 45.0.0 declares itself to be Python 2 compatible. Regardless of the actual situation (I have little knowledge about), pip is doing what it is supposed to do, and any fix should happen on setuptools.

Suggest to close this as wrong project.

[pganssle]

@uranusjr According to PyPI, setuptools 45.0.0 requires Python >=3.5. If the latest version of pip is installing version 45.0.0 on 2.7, then it's a bug in pip.

That said, I can't reproduce it. pip install -U setuptools on Python 2.7 gives me 44.0.0.

[uranusjr]

@pganssle Hmm interesting. I did not do a throughout check (should have, sorry) and was assuming based on the wheel being declared py2.py3. Maybe there is a subtle packaging problem in there? (But pip should do the right thing anyway since requires-python should be enough to block installation.)

[chrahunt]

setuptools 45.0.0 correctly indicates that it is only compatible with Python >= 3.5 (see here, which is included in 45.0.0). On upload to PyPI, this value is propagated to data-requires-python attribute of the file links in the simple API that pip normally uses to find applicable setuptools packages here (as described in PEP 503). The sdist and wheel for setuptools 45.0.0 seem to have the correct value: data-requires-python=">=3.5".

In your log it looks like you are using two mirrors. Here are the setuptools pages that pip would look at:

1. http://mirror.regionone.limestone.openstack.org/pypi/simple/setuptools/
2. http://mirror.regionone.limestone.openstack.org/wheel/ubuntu-18.04-x86_64/setuptools/

Mirror 1 looks like it is correctly populating the data-requires-python attribute, but 2 is not. Since pip can't tell that the setuptools 45.0.0 from 2 does not support Python 2, it downloads it and fails. This is currently the expected behavior. It could be improved, but only as part of #988.

[ianw]

In your log it looks like you are using two mirrors. Here are the setuptools pages that pip would look at:
http://mirror.regionone.limestone.openstack.org/pypi/simple/setuptools/
http://mirror.regionone.limestone.openstack.org/wheel/ubuntu-18.04-x86_64/setuptools/

This is correct; we pre-build wheels and setup extra-index-url for our CI hosts to speed up their runs (see https://review.opendev.org/#/c/702166/1/roles/configure-mirrors/templates/etc/pip.conf.j2 ... we may have to turn this off if we can't find an expedient solution). We have not considered PEP503; we build the wheels and publish them just via apache auto-indexing.

Given a directory full of .whl files, how would we determine the data-requires-python values to populate? We could write a script to create an index.html and populate this for the wheels, if there's a way to actually determine this metadata from the .whl ...

[chrahunt]

Here are the specific standards-compliant steps to get the value from a wheel:

1. Open the wheel (which is a zip) and get the list of contained files (e.g. zipfile.ZipFile(p).namelist())
2. Find the single file with path matching ^[^/]+\.dist-info/METADATA$.
3. Extract the file and decode as utf-8 (Python 3) or do not decode (Python 2).
4. Parse the text with email.message_from_string
5. In the parsed message, get the value of the single "Requires-Python" header, if any. If not present then you should put no data-requires-python value.

If these don't work then pip wouldn't've been able to install the wheel anyway.

Make sure to encode the value as described in PEP 503.

[yoctozepto]

Thanks, Christopher. @chrahunt
In fact I suspected the mirrors but forgot to check the wheel one. Thanks for finding this out.

[chrahunt]

Any time. I will close this now, but please let us know if you have any other concerns. Thanks!