NumPy 2 is not supported

[leofang]

We hit this issue locally. Took us a while to figure out a strategy to untangle this issue from the other #3... 😅

Using the Python 3.10 CI log from conda-forge/numpy-allocator-feedstock#9, which uses NumPy 2 at build time

2024-10-07T01:39:12.5131564Z ## Package Plan ##
2024-10-07T01:39:12.5131783Z 
2024-10-07T01:39:12.5133208Z   environment location: /home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
2024-10-07T01:39:12.5133609Z 
2024-10-07T01:39:12.5134974Z 
2024-10-07T01:39:12.5135426Z The following NEW packages will be INSTALLED:
2024-10-07T01:39:12.5142863Z 
...
2024-10-07T01:39:12.5163127Z     numpy:             2.0.2-py311h71ddf71_0      conda-forge
...

we see the following compile-time warnings:

2024-10-07T01:39:35.6192406Z   building 'numpy_allocator' extension
2024-10-07T01:39:35.6201840Z   creating build/temp.linux-x86_64-cpython-311
2024-10-07T01:39:35.6374655Z   /home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_build_env/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include -fPIC -O2 -isystem /home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/work=/usr/local/src/conda/numpy-allocator-1.2.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place=/usr/local/src/conda-prefix -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include -fPIC -I/home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.11/site-packages/numpy/_core/include -I/home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include/python3.11 -c numpy_allocator.c -o build/temp.linux-x86_64-cpython-311/numpy_allocator.o
2024-10-07T01:39:35.7566917Z   numpy_allocator.c: In function 'handles':
2024-10-07T01:39:35.7572001Z   numpy_allocator.c:417:51: warning: implicit declaration of function 'PyArray_HANDLER'; did you mean 'PyArray_SHAPE'? [-Wimplicit-function-declaration]
2024-10-07T01:39:35.7574545Z     417 |                         PyObject *array_handler = PyArray_HANDLER((PyArrayObject *) array);
2024-10-07T01:39:35.7581257Z         |                                                   ^~~~~~~~~~~~~~~
2024-10-07T01:39:35.7594984Z         |                                                   PyArray_SHAPE
2024-10-07T01:39:35.7598005Z   numpy_allocator.c:417:51: warning: initialization of 'PyObject *' {aka 'struct _object *'} from 'int' makes pointer from integer without a cast [-Wint-conversion]
2024-10-07T01:39:35.7653414Z   numpy_allocator.c: In function '__exit__':
2024-10-07T01:39:35.7655017Z   numpy_allocator.c:474:33: warning: implicit declaration of function 'PyDataMem_SetHandler'; did you mean 'PyDataMem_DefaultHandler'? [-Wimplicit-function-declaration]
2024-10-07T01:39:35.7656847Z     474 |         PyObject *old_handler = PyDataMem_SetHandler(new_handler);
2024-10-07T01:39:35.7657592Z         |                                 ^~~~~~~~~~~~~~~~~~~~
2024-10-07T01:39:35.7659682Z         |                                 PyDataMem_DefaultHandler
2024-10-07T01:39:35.7660783Z   numpy_allocator.c:474:33: warning: initialization of 'PyObject *' {aka 'struct _object *'} from 'int' makes pointer from integer without a cast [-Wint-conversion]
2024-10-07T01:39:35.7662321Z   numpy_allocator.c: In function '__enter__':
2024-10-07T01:39:35.7663633Z   numpy_allocator.c:505:33: warning: initialization of 'PyObject *' {aka 'struct _object *'} from 'int' makes pointer from integer without a cast [-Wint-conversion]
2024-10-07T01:39:35.7671137Z     505 |         PyObject *old_handler = PyDataMem_SetHandler(new_handler);
2024-10-07T01:39:35.7679441Z         |                                 ^~~~~~~~~~~~~~~~~~~~
2024-10-07T01:39:35.7687281Z   numpy_allocator.c: In function 'set_handler':
2024-10-07T01:39:35.7719293Z   numpy_allocator.c:628:24: warning: returning 'int' from a function with return type 'PyObject *' {aka 'struct _object *'} makes pointer from integer without a cast [-Wint-conversion]
2024-10-07T01:39:35.7720481Z     628 |                 return PyDataMem_SetHandler(NULL);
2024-10-07T01:39:35.7722175Z         |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~
2024-10-07T01:39:35.7754551Z   numpy_allocator.c:630:24: warning: returning 'int' from a function with return type 'PyObject *' {aka 'struct _object *'} makes pointer from integer without a cast [-Wint-conversion]
2024-10-07T01:39:35.7755729Z     630 |                 return PyDataMem_SetHandler(handler);
2024-10-07T01:39:35.7773779Z         |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-10-07T01:39:35.7789118Z   numpy_allocator.c: In function 'get_handler':
2024-10-07T01:39:35.7794616Z   numpy_allocator.c:648:59: warning: initialization of 'PyObject *' {aka 'struct _object *'} from 'int' makes pointer from integer without a cast [-Wint-conversion]
2024-10-07T01:39:35.7806070Z     648 |                                 PyObject *array_handler = PyArray_HANDLER((PyArrayObject *) array);
2024-10-07T01:39:35.7816807Z         |                                                           ^~~~~~~~~~~~~~~
2024-10-07T01:39:35.7831997Z   numpy_allocator.c:663:24: warning: implicit declaration of function 'PyDataMem_GetHandler'; did you mean 'PyDataMem_DefaultHandler'? [-Wimplicit-function-declaration]
2024-10-07T01:39:35.7842678Z     663 |                 return PyDataMem_GetHandler();
2024-10-07T01:39:35.7855122Z         |                        ^~~~~~~~~~~~~~~~~~~~
2024-10-07T01:39:35.7866332Z         |                        PyDataMem_DefaultHandler
2024-10-07T01:39:35.7880791Z   numpy_allocator.c:663:24: warning: returning 'int' from a function with return type 'PyObject *' {aka 'struct _object *'} makes pointer from integer without a cast [-Wint-conversion]
2024-10-07T01:39:35.7884034Z     663 |                 return PyDataMem_GetHandler();
2024-10-07T01:39:35.7894052Z         |                        ^~~~~~~~~~~~~~~~~~~~~~

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1047781&view=logs&j=4f922444-fdfe-5dcf-b824-02f86439ef14&t=937c195f-508d-5135-dc9f-d4b5730df0f7&l=844
and the import test fails

2024-10-07T01:39:52.7336944Z import: 'numpy_allocator'
2024-10-07T01:39:52.7340790Z Traceback (most recent call last):
2024-10-07T01:39:52.7341770Z   File "/home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/test_tmp/run_test.py", line 2, in <module>
2024-10-07T01:39:52.7342443Z     import numpy_allocator
2024-10-07T01:39:52.7343762Z ImportError: /home/conda/feedstock_root/build_artifacts/numpy-allocator_1728265022062/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/numpy_allocator.cpython-311-x86_64-linux-gnu.so: undefined symbol: PyDataMem_SetHandler
2024-10-07T01:39:53.5108629Z WARNING: Tests failed for numpy-allocator-1.2.0-py311h18e1886_2.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1047781&view=logs&j=4f922444-fdfe-5dcf-b824-02f86439ef14&t=937c195f-508d-5135-dc9f-d4b5730df0f7&l=1009

We can reproduce the missing symbol import error locally.

This is a blocker for CuPy to adopt support for using system memory on Grace Hopper (cupy/cupy#8442). Would be nice to know how much work is needed so that we can discuss, coordinate, and maybe help too 🙂

cc: @emcastillo @seberg @mmccarty @rongou

[eliaskoromilas]

This issue first appears in NumPy v1.25.0.
conda-forge/numpy-allocator-feedstock#8 (comment)

[eliaskoromilas]

I think it might be related to this commit:
numpy/numpy@3a81135#diff-d4fc38ad3fd8b43b6569ee82dbe5ba189619e0bc4898e7ab7f723d76a839f32fL370-R371

[eliaskoromilas]

As it turns out:

NumPy 1.22 NPY_FEATURE_VERSION == 15
NumPy 1.23 NPY_FEATURE_VERSION == 16
NumPy 1.24 NPY_FEATURE_VERSION == 16
NumPy 1.25 NPY_FEATURE_VERSION == 13 (< 15)
NumPy 1.26 NPY_FEATURE_VERSION == 13 (< 15)
NumPy 2.0 NPY_FEATURE_VERSION == 14 (< 15)
NumPy 2.1 NPY_FEATURE_VERSION == 14 (< 15)

which breaks the following check:

#if NPY_FEATURE_VERSION >= NPY_1_22_API_VERSION
...
#endif

[eliaskoromilas]

I was missing #define NPY_TARGET_VERSION NPY_1_22_API_VERSION. I will push the fix shortly.

The following commit changed the default NPY_FEATURE_VERSION:
numpy/numpy@6309cf2

[leofang]

Appreciate a ton for the speedy fix and release, @eliaskoromilas!