NumPy 2.0 support

[jakirkham]

Feature description

NumPy 2.0 is coming out soon ( numpy/numpy#24300 ). NumPy 2.0.0rc1 packages for conda & wheels came out 3 weeks back ( numpy/numpy#24300 (comment) )

To prepare for NumPy 2.0, it might be worthwhile to start testing against NumPy 2 in CI

NumPy has put out a migration guide. More details are in the release notes

If GDAL make use of NumPy's C API (and produces wheels that use it), having a release of GDAL with wheels built against NumPy 2.0.0rc1 would be helpful to ensure NumPy 1 & 2 compatible wheels (as wheels built against NumPy 1 won't be compatible with NumPy 2). More details in this NumPy 2 ABI doc

Also as NumPy is tracking ecosystem support for NumPy 2.0, it would be helpful to share GDAL current support status in issue (with any plans): numpy/numpy#26191

Additional context

No response

[rouault]

To prepare for NumPy 2.0, it might be worthwhile to start testing against NumPy 2 in CI

I gave a try a few weeks ago locally, and everything went fine without requiring any change (GDAL usage of NumPy is rather primitive). We have some CI targets targetting rolling distributons (Alpine Linux, CondaForge), so at some point NumPy 2 will come into CI by default. That said, we might try to hack an existing CI configuration to test against numpy 2.

Having a release of GDAL with wheels built against NumPy 2.0.0rc1

The GDAL project itself doesn't deliver binary wheels on pypi. GDAL and its complex set of dependencies are not appropriate for pip (cf #4352, #3060), due to potential clashes with other packages that might use the same native dependencies of GDAL. conda-forge builds are our current answer to that need.

Also as NumPy is tracking ecosystem support for NumPy 2.0, it would be helpful to share GDAL current support status in issue (with any plans): numpy/numpy#26191

we'll do that

[rouault]

I gave a try a few weeks ago locally, and everything went fine without requiring any change

Actually I should have written any change in the library itself. A few adjustments were made in the test suite per #9044

[rouault]

Any Python packaging champion here (perhaps @pl-kevinwurster @rcoup) ?
While I manage to build GDAL against numpy 2 if it is already installed on the system or a virtualenv, using regular cmake && make process, or manually running python setup.py, when I try to do "pip install gdal-3.X.0.tar.gz" from the sdist artifact, the new pyproject.toml file gets used, and it has a requires = [ "oldest-supported-numpy" ], which draws numpy 1.X . So it looks like the sdist artifact must make a choice between numpy 1.X and 2.0, which kind of makes sense. I presume it would be safer for the upcoming GDAL 3.9.0 to remain on oldest-supported-numpy ? and perhaps switch to "numpy>=2.0" once it has made its way a bit more. Or maybe there's a way at build time that people can ask pip install to use numpy 2... ? Or we would have to deliver 2 different sdist for numpy 1 and numpy 2, like gdal-numpy1 and gdal-numpy2, but I would not want to go into that forest of complication...

[jakirkham]

Hopefully this context is helpful

If GDAL builds against a Python version and NumPy 2.0 (using the RC1 would suffice), then the packages built would support the oldest NumPy 1.x released for that Python version and NumPy 2.x

Ralf shared an example in this comment: conda-forge/conda-forge.github.io#1997 (comment)

IOW building against NumPy 2 in all builds would make oldest-supported-numpy obsolete. This is also covered in oldest-supported-numpy's deprecation notice

[rouault]

then the packages built would support the oldest NumPy 1.x released for that Python version and NumPy 2.x

how is that possible? Doesn't numpy 2 changes the Numpy C ABI ?

I don't think we can require numpy 2 because current Linux distributions don't ship it yet, and they wouldn't want to depend on a pip package not present on their system.

[jakirkham]

Yes this is covered in the "NumPy 2.0-specific advice" section of "For downstream packages authors" doc:

NumPy 2.0 is an ABI-breaking release, however it does contain support for building wheels that work on both 2.0 and 1.xx releases. It’s important to understand that:

• When you build wheels for your package using a NumPy 1.xx version at build time, those will not work with NumPy 2.0.
• When you build wheels for your package using a NumPy 2.x version at build time, those will work with NumPy 1.xx.

The first time the NumPy ABI for 2.0 is guaranteed to be stable will be the release of the first release candidate for 2.0 (i.e., 2.0.0rc1).

Here are more details on specific NumPy C API changes

[rouault]

how is that possible? Doesn't numpy 2 changes the Numpy C ABI ?

hum according to https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-abi-handling, "When you build wheels for your package using a NumPy 2.x version at build time, those will work with NumPy 1.xx" ... Looks like magic to me, but if they say so...

@sebastic I'd be interested by your insights regarding Linux packaging. Have you seen packages that start requiring numpy 2 and how that has been handled ? That said if we bump the build requirement only in pyproject.toml to be numpy>=2.0.0rc1, I don't think that would affect the Debian build as it probably doesn't use that file if using just a make && make install process to build the Python bindings.

[sebastic]

@sebastic I'd be interested by your insights regarding Linux packaging. Have you seen packages that start requiring numpy 2 and how that has been handled ? That said if we bump the build requirement only in pyproject.toml to be numpy>=2.0.0rc1, I don't think that would affect the Debian build as it probably doesn't use that file if using just a make && make install process to build the Python bindings.

Some of the PyTroll related projects started requiring NumPy 2: pykdtree, pyresample, geotiepoints, and trollimage for example.

Their Debian packages were patched to drop the version requirement as the numpy package in Debian unstable is still at 1.26.4. The pyproject plugin for the pybuild buildsystem is used to build Python packages using the build module instead of running setup.py <clean|config|build|test|install> which is deprecated, and the plugin uses the version information from pyproject.toml for the package dependencies.

The gdal Debian package lets CMake build the Python modules, but would use pybuild-plugin-pyproject when the Python module is split out of the GDAL source tree like we already do for fiona and rasterio.

[rgommers]

That said if we bump the build requirement only in pyproject.toml to be numpy>=2.0.0rc1, I don't think that would affect the Debian build as it probably doesn't use that file

This is correct, and applies to all distros. Even if they use pip in their own build process, they always run it with --no-build-isolation to avoid pulling in anything from PyPI. Hence a build requirement of numpy>=2.0.0rc1 doesn't affect them, and they can continue to build with 1.2x just fine.

[rouault]

One issue I'm thinking about is that for GDAL 3.9 we still support Python 3.8, but numpy 2.0 only supports >= 3.9.

[rgommers]

You can use Python version specific metadata like this:

    "oldest-supported-numpy; python_version=='3.8'",
    "numpy >=2.0.0rc1; python_version>='3.9'",

That is reliable, and for example h5py already did a release with that: https://github.com/h5py/h5py/blob/c8972f1a93b9b55620e617a615d3a912a71bff05/pyproject.toml#L4-L5

[rouault]

ok, I've queued in #9761 a PR updating pyproject.toml along the lines of #9751 (comment) . This should hopefully be safe for backport to the 3.9 branch. We'll probably discuss that during the tomorrow GDAL maintainer monthly meeting

[pl-kevinwurster]

@rouault I'm a bit swamped and can't fully dig into this at the moment, but it seems like your plan is sound.

[jakirkham]

Thanks all! 🙏

Am very appreciative of all the engagement around this issue 😊

Wonder if there is a low stakes way of getting Even's work out there for testing. Is there an RC/dev or nightly release process for GDAL? Maybe this would provide a way to test out the changes a bit more and build confidence around them?

[rouault]

The PR passing shows that at least "distribution-like" building workflows this doesn't seem to break things. For "pip install" workflows, we'll mostly know after the release where it will land. If that goes into the 3.9 branch, we have at least a 3.9.0rc1, and possibly a 3.9.0beta2, but we traditionally don't issue pip packages for the non-final versions (partly because our rcs are really candidate final versions, and so the pip package version has no rc in it, and we can't upload twice a package with the same version)

[jakirkham]

Ofc that makes sense. Thanks for the insight into the release process! 🙏

[rouault]

Decision from today's GDAL maintainer meetings. We'll include the change in pyproject.toml in the 3.9.0beta2 release to give it a bit more chance to be tested before 3.9.0RC1.