BUILD: Unable to compile on Windows 10 with Python 3.10

[TimeHorse]

When downloading the tar ball and executing “py setup.py develop” most of the build succeeds until you ge to the step where it tries to build pandas._libs.indexing, because indexing.c is not in the _libs directory, the build fails with the following error:

Exception: Cython-generated file ‘pandas_lib/indexing.c’ not found.
Cython is required to compile pandas from a development branch.
Please install Cython or download a release package of pandas.

For the record, there is no official release of Pandas for Python 3.10 on Windows or Macintosh, only Linux.

cython is installed:

py -c “import cython; print(cython.version)”
0.29.24

[lithomas1]

Attempting to install pandas 1.3.4 on a fresh conda python 3.10 env works for me.
These are my logs

(py310) C:\Users\liwende>pip install pandas
Collecting pandas
  Downloading pandas-1.3.4.tar.gz (4.7 MB)
     |████████████████████████████████| 4.7 MB 3.2 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy>=1.21.0
  Downloading numpy-1.21.3-cp310-cp310-win_amd64.whl (14.0 MB)
     |████████████████████████████████| 14.0 MB 6.4 MB/s
Collecting python-dateutil>=2.7.3
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2017.3
  Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: pandas
  Building wheel for pandas (PEP 517) ... done
  Created wheel for pandas: filename=pandas-1.3.4-cp310-cp310-win_amd64.whl size=10048963 sha256=dfe756adc45b06fe509ef939f9a2786ad502be0a1841075c001bc6240f69ffdb
  Stored in directory: c:\users\liwende\appdata\local\pip\cache\wheels\f5\c7\c7\67b16ccaa45a66d1b9a188c3e3fc8238de49d6aa5e1420d2b6
Successfully built pandas
Installing collected packages: six, pytz, python-dateutil, numpy, pandas
Successfully installed numpy-1.21.3 pandas-1.3.4 python-dateutil-2.8.2 pytz-2021.3 six-1.16.0

Are you trying to compile a development version of pandas?

[TimeHorse]

GitHub is blocked at my office so I have to install from source. I am writing this from my phone.

[simonjayhawkins]

I've milestoned 1.3.5 for now. However we will probably be able to create Windows wheels for 1.3.4 (and upload to PyPI) now that numpy 3.10 wheels for windows are available (numpy-1.21.3-cp310-cp310-win_amd64.whl) @lithomas1 ?

[TimeHorse]

I already had that version of numpy; I reinstalled it but I still get the same file not found error.

[lithomas1]

Why are you using python setup.py develop instead of doing pip install pandas? Is there a problem with accessing pypi.org from your office?

If you still want to try to install from source, try cleaning your installation folder with python setup.py clean, followed by python setup.py install. The c files are auto-generated by Cython, so I assume something went wrong there.

[lithomas1]

I've milestoned 1.3.5 for now. However we will probably be able to create Windows wheels for 1.3.4 (and upload to PyPI) now that numpy 3.10 wheels for windows are available (numpy-1.21.3-cp310-cp310-win_amd64.whl) @lithomas1 ?

I have a PR for macOS and Windows wheels ready on the MacPython repo. (Merge first MacPython/pandas-wheels#159, then MacPython/pandas-wheels#156, the second one might need a rebase). I think it would be better to just upload everything to 1.3.5. Installs using pip seem to work for me at least, and there were some changes to the macOS wheels(they became universal2.) so just lower risk to wait.

[jorisvandenbossche]

@lithomas1 what's the risk exactly to still upload the Python 3.10 wheels for 1.3.4, now they can be built? (it's the first time wheels for 3.10 would become available, so it's not there were changes compared to the previous version of pandas for py3.10?)

[lithomas1]

Hmm.. I might be coming around on this one. I forgot that a lot of people don't have C Compilers/Cython on their system, and it might be difficult to compile from source for them.

Do we have a way of changing the release notes after a release? I don't want to do this silently, since bad wheels make it much more difficult to install a version of pandas. Also, there are no more x86_64 wheels on macOS, so the size of the package is gonna be 2x for people with Intel Macs.

[lithomas1]

Tagging this as with "admin" label since someone with permissions needs to upload to pypi. Also pinging @pandas-dev/pandas-core to see if there's any objections.

[jreback]

@simonjayhawkins if you can upload would be great.

[bashtage]

Title has wrong version of Python. Presumably 3.10.1.

[hugovk]

Python 3.10.0 is the latest.

https://www.python.org/downloads/release/python-3100/

[simonjayhawkins]

@simonjayhawkins if you can upload would be great.

No further comments here, so will upload shortly.

[simonjayhawkins]

No further comments here, so will upload shortly.

done.

[jorisvandenbossche]

Thanks Simon!

[lithomas1]

It turns out we don't have thin wheels for x86_64 macOS users for Python 3.10, because of a multibuild limitation. So someone with Intel Mac will install the universal2 wheel which is ~2x bigger. Recently, multibuild allowed building the thin x86_64 macos wheels for Python 3.10.

I have opened MacPython/pandas-wheels#163 accordingly. @simonjayhawkins Can you merge that PR? (the failures can be ignored cuz they were fixed on changes that were backported but only in 1.3.5 not 1.3.4)

I am not sure whether it is too close to 1,3,5 to upload the wheels, though. I'll leave that decision to the rest of @pandas-dev/pandas-core

[simonjayhawkins]

I am not sure whether it is too close to 1,3,5 to upload the wheels, though. I'll leave that decision to the rest of @pandas-dev/pandas-core

Once 1.3.5 wheels are up, the 1.3.4 wheel will probably only be downloaded if specifically requested. Probably best to leave as is although I've not gone through the process of changing build number for PyPI release so not sure what is required.

[simonjayhawkins]

I have opened MacPython/pandas-wheels#163 accordingly. @simonjayhawkins Can you merge that PR? (the failures can be ignored cuz they were fixed on changes that were backported but only in 1.3.5 not 1.3.4)

done.

We will have an additional wheel pandas-1.3.4-cp310-cp310-macosx_10_9_x86_64.whl which could be uploaded? and the pandas-1.3.4-cp310-cp310-macosx_10_9_universal2 (and others) will now be re-built with a later version of multibuild so cannot be uploaded to PyPI unless the build number is changed?

[lithomas1]

Yeah. The config for the other wheels should have remained the same. The only question is whether to upload pandas-1.3.4-cp310-cp310-macosx_10_9_x86_64.whl.

[simonjayhawkins]

no further comments here so let's close and leave 1.3.4 as is.