[numpy] add new port #37409

Neumann-A · 2024-03-13T08:03:24Z

Good to also have for this:

[blas/lapack] force vendor #24327
A Fortran compiler
More knowledge how cross builds of python work.
Fix arm64-windows cross builds
x86-windows cython not linking python lib

[python] add module build/install infra

Neumann-A · 2024-03-15T07:10:51Z

Traceback (most recent call last):
  File "/mnt/vcpkg-ci/b/numpy/import_test.py", line 4, in <module>
    from numpy import *
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/__init__.py", line 149, in <module>
    from . import lib
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/lib/__init__.py", line 23, in <module>
    from . import index_tricks
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/lib/index_tricks.py", line 12, in <module>
    import numpy.matrixlib as matrixlib
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/matrixlib/__init__.py", line 4, in <module>
    from . import defmatrix
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/matrixlib/defmatrix.py", line 12, in <module>
    from numpy.linalg import matrix_power
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/linalg/__init__.py", line 73, in <module>
    from . import linalg
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 35, in <module>
    from numpy.linalg import _umath_linalg
ImportError: /mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/linalg/_umath_linalg.cpython-311-x86_64-linux-gnu.so: undefined symbol: _gfortran_concat_string

Seems like the test needs to know about implicit link directories to setup LD_LIBRARY_PATH?

m-kuhn · 2024-03-15T07:27:05Z

Just exportring LD_LIBRARY_PATH=${CURRENT_PACKAGES_DIR}/lib won't help here?

Neumann-A · 2024-03-15T07:36:06Z

Just exportring LD_LIBRARY_PATH=${CURRENT_PACKAGES_DIR}/lib won't help here?

Nope. gfortran should be installed by the system on linux.

m-kuhn · 2024-03-15T07:40:11Z

How is this supposed to be discovered at runtime (after installation)?

Neumann-A · 2024-03-15T12:53:45Z

How is this supposed to be discovered at runtime (after installation)?

Was a missing -lgfortran in lapack. Don't know why numpy compiles with -Wl,--allow-shlib-undefined

m-kuhn · 2024-04-03T07:01:35Z

Do I understand this correctly that this is currently blocked by static builds not having a python executable linked to cython?
Should there be a helper function vcpkg_acquire_python_with_modules(cython ...) that can be invoked by consumers (like numpy here) to create a python executable with specific additional modules linked?

Neumann-A · 2024-04-04T15:12:40Z

Do I understand this correctly that this is currently blocked by static builds not having a python executable linked to cython?

Don't know if this is the problem. For me the problem is that I don't know how to create static python extensions which can be embedded into an application since that is what would be required for downstream users. (->https://www.kitware.com/static-python-and-numpy-with-paraview/)

Should there be a helper function vcpkg_acquire_python_with_modules(cython ...) that can be invoked by consumers (like numpy here) to create a python executable with specific additional modules linked?

Would be nice and is probably required if downstream tries to invoke python somehow with the modules for the build

m-kuhn · 2024-04-04T16:07:03Z

Or this is listed as not supported static builds until this is sorted

# Conflicts: # ports/lapack-reference/portfile.cmake # ports/lapack-reference/vcpkg.json

…to add_numpy

Neumann-A · 2024-04-22T05:27:09Z

@m-kuhn: Ideas how to solve the linux regression?
ImportError: /mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/linalg/_umath_linalg.cpython-311-x86_64-linux-gnu.so: undefined symbol: _gfortran_concat_string
Somehow need to tell python that since lapack-reference is static it needs to load gfortran itself.

m-kuhn · 2024-04-22T05:51:25Z

Do I understand this correctly: Numpy is built dynamic while lapack-reference (which contains _gfortran_concat_string) is static and python links lapack-reference?

If yes, I guess numpy needs to link lapack-reference too which might well result in ODR violations as lapack-reference is present in two places. Or lapack-reference and its dependency tree need to be built dynamic as well which should work.

Neumann-A · 2024-04-22T07:32:01Z

@m-kuhn: No the problem is that the so does not know to load gfortran:

$ readelf -d _umath_linalg.cpython-311-x86_64-linux-gnu.so


Dynamic section at offset 0x3c6c80 contains 26 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x1b000

Neumann-A · 2024-04-22T07:36:46Z

Ah wait that is just the same issue as before. I thought I already solved that one but the merge probably killed it.

Neumann-A · 2024-04-22T12:36:35Z

@autoantwort:
Could you investigate the arm64-osx failure here?
For me it seems like the build is using the wrong (system) python for some reason since the file list clearly shows that the missing module should be present in the vcpkg installed python.

autoantwort · 2024-04-22T12:53:55Z

[75/301] /Users/leanderSchulten/git_projekte/vcpkg/buildtrees/numpy/src/1d0eb3c352-64e3143c32.clean/numpy/_build_utils/tempita.py ../src/1d0eb3c352-64e3143c32.clean/numpy/random/_bounded_integers.pxd.in -o numpy/random/_bounded_integers.pxd
FAILED: numpy/random/_bounded_integers.pxd 
/Users/leanderSchulten/git_projekte/vcpkg/buildtrees/numpy/src/1d0eb3c352-64e3143c32.clean/numpy/_build_utils/tempita.py ../src/1d0eb3c352-64e3143c32.clean/numpy/random/_bounded_integers.pxd.in -o numpy/random/_bounded_integers.pxd
/opt/homebrew/opt/[email protected]/bin/python3.12
Traceback (most recent call last):
  File "/Users/leanderSchulten/git_projekte/vcpkg/buildtrees/numpy/src/1d0eb3c352-64e3143c32.clean/numpy/_build_utils/tempita.py", line 7, in <module>
    from Cython import Tempita as tempita
ModuleNotFoundError: No module named 'Cython'

And first line of buildtrees/numpy/src/1d0eb3c352-64e3143c32.clean/numpy/_build_utils/tempita.py

#!/usr/bin/env python3

ports/blas/portfile.cmake

ports/cblas/portfile.cmake

ports/cblas/vcpkg.json

ports/lapack/portfile.cmake

…dd_numpy

# Conflicts: # ports/lapack-reference/portfile.cmake # ports/lapack-reference/vcpkg.json

aminya · 2024-05-21T18:26:17Z

This PR can possibly fix this issue too:
#34565

@Neumann-A Anything blocking you to make this ready? I might be able to help

Neumann-A · 2024-05-21T18:30:07Z

This PR can possibly fix this issue too:
#34565

Figuring out cross-compiling with pyhton. Crossenv only works for linux and not for windows.

# Conflicts: # ports/lapack-reference/vcpkg.json

# Conflicts: # ports/vcpkg-tool-meson/meson.template.in # ports/vcpkg-tool-meson/vcpkg.json # ports/vcpkg-tool-meson/vcpkg_configure_meson.cmake

dg0yt

These comment were unsent for a while.

dg0yt · 2024-04-22T11:00:41Z

ports/lapack-reference/portfile.cmake

+  list(REVERSE implicit_c_libs)
+  list(REMOVE_DUPLICATES implicit_c_libs)
+  list(REVERSE implicit_c_libs)


Suggested change

list(REVERSE implicit_c_libs)

list(REMOVE_DUPLICATES implicit_c_libs)

list(REVERSE implicit_c_libs)

Just items to be removed from another list.

dg0yt · 2024-11-27T06:13:32Z

ports/lapack-reference/portfile.cmake

+  list(REVERSE implicit_c_dirs)
+  list(REMOVE_DUPLICATES implicit_c_dirs)
+  list(REVERSE implicit_c_dirs)
+  string(REGEX MATCH "CMAKE_Fortran_IMPLICIT_LINK_DIRECTORIES:([^\n]+)" implicit_fortran_dirs "${config_log}")
+  set(implicit_fortran_dirs "${CMAKE_MATCH_1}")
+  list(REVERSE implicit_fortran_dirs)
+  list(REMOVE_DUPLICATES implicit_fortran_dirs)
+  list(REVERSE implicit_fortran_dirs)


For link dirs, reversal isn't really needed.
And for the c dirs, deduplication is redundant.

Neumann-A added 4 commits March 13, 2024 08:42

[numpy] Add port

70d0566

[python] add module build/install infra

Merge remote-tracking branch 'upstream/master' into add_numpy

cf6befb

opencv use numpy from vcpkg

ff1ff2e

use sitecustomize.py in python3 for dll search paths

515adbb

LilyWangLL assigned JonLiu1993 Mar 13, 2024

revert boost-python change

ad8374d

JonLiu1993 added the category:new-port The issue is requesting a new library to be added; consider making a PR! label Mar 13, 2024

format-manifest try fixing errors

f88edba

Neumann-A mentioned this pull request Mar 14, 2024

Systematically test ports open-vcpkg/python-registry#6

Open

Fix numpy linux

5b0069e

Neumann-A added 3 commits April 21, 2024 23:37

Don't test stuff which does not work due to static python

582c6e5

Merge remote-tracking branch 'upstream/master' into add_numpy

f9cf981

# Conflicts: # ports/lapack-reference/portfile.cmake # ports/lapack-reference/vcpkg.json

Merge remote-tracking branch 'origin/openimageio-no-static-python' in…

c219a84

…to add_numpy

Neumann-A added 3 commits April 22, 2024 10:17

add implicit fortran libs to pc file

8f68a37

print path to python executable

74b72f4

fix arm64-windows

f1d2f50

Neumann-A added 2 commits April 24, 2024 14:47

back to accelerate

3c1d1b5

enable ci

7c2727d