Review content of crossenv to respect build vs cross.

[th0ma7]

Description

Review content of crossenv to respect build vs cross.

• spksrc.requirement.mk: Unified method to process requirements files
• Migrate pure python wheels to build using crossenv/build instead of native
• Manage distinct installation of crossenv/build and crossenv/cross
• Duplicate log output under crossenv|wheel-<arch>-<tcversion>.log
• Upgrade to python 3.12.9
• Upgrade to python 3.13.2

Closes #6430

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Bug fix
• New Package
• Package update
• Includes small framework changes
• This change requires a documentation update (e.g. Wiki)

[hgy59]

@th0ma7 I was hoping this would fix numpy for python312, but it doen't (still getting ../meson.build:37:2: ERROR: Problem encountered: NumPy requires Cython >= 0.29.34)

[th0ma7]

@hgy59 thnx for testing but it's not ready yet, I've only laid-down the foundation to have a reusable requirement mk file. I still have to:

1. Update the crossenv mk file to use the new requirement mk file
2. convert crossenv definitions to use cross|build
3. Migrate environment PATH to use build only (and stop relying on native/python3* maturin)

Only once that is complete I should be ready to start looking further into numpy in particular.

[th0ma7]

@hgy59 still a bit of testing todo but I should be iches away from being ready to look in particular at the numpy issue.

I'll let things run but changes done so far in this PR to the PATH and most importantly the crossenv cross vs build contents should now address many of the remaining issues found 🤞

[mreid-tt]

hey @th0ma7, looking forward to this PR. Looking at the build errors it seems that based on this python-greenlet/greenlet#392 (comment), they are caused by:

That's because you're trying to install greenlet < 3.1, which doesn't support Python 3.13. You must use greenlet >= 3.1.x.

As such, I believe if you comment out this line your build should succeed:

spksrc/spk/python313-wheels/src/requirements-crossenv-greenlet-v3.txt

Line 9 in 17ece54

greenlet==3.0.3

[th0ma7]

I believe it is now complete and ready for testing. Only remaining item (unless github-action tells me otherwise): numpy.

[th0ma7]

Suggested plan of actions:

1. Merge this pr
2. Update pythons (updates got out this week)
3. Fix numpy

Thoughts?

[hgy59]

@th0ma7 I have updated the comment in python-wheels packages related to the include of spksrc.python.mk.

Please feel free to revert it, if it is not correct...

[hgy59]

@th0ma7 one thing I am missing

when building single wheels with make WHEEL="{name==version}" wheel-{arch}-{tcversion} it would be handy to have the build logs collected in a file.

I guess we only need to add something like | tee --append build-wheel-$*.log in the make wheel command of the wheel-% target in spksrc.wheel.mk.

[hgy59]

2. Update pythons (updates got out this week)

Python 3.13.2 is already released.

EDIT:
and Python 3.12.9 too

[th0ma7]

when building single wheels with make WHEEL="{name==version}" wheel-{arch}-{tcversion} it would be handy to have the build logs collected in a file.

I guess we only need to add something like | tee --append build-wheel-$*.log in the make wheel command of the wheel-% target in spksrc.wheel.mk.

Yeah, I thought of that too. By default it gest collected when invoking a full standard build but not for neither crossenv-* or wheel-*. I can look into that. I wonder, would it make sense to have a per-wheel log in a sub-directory? or having a single wheel-arch-version.log and similarly a crossenv-arch-version.log created? maybe even only when invoking them directly (not sure i can do that)?

That I'd add to this PR.

[mreid-tt]

I believe it is now complete and ready for testing. Only remaining item (unless github-action tells me otherwise): numpy.

Great work, @th0ma7! You might consider re-enabling llfuse based on the fix I mentioned here. I'm not sure if there are any other commented-out items (besides numpy) that could be re-enabled, but it’s worth a check.

[hgy59]

@th0ma7 with my manylinux patch above, all wheels for x86_64 are correct and successfully installed.

with the latest version, the wheels are now lacking the platform (linux) in the tag and the name

bluetooth_data_tools-1.20.0-cp312-cp312-x86_64.whl
cached_ipaddress-0.8.0-cp312-cp312-x86_64.whl
convertertools-0.6.1-cp312-cp312-x86_64.whl
dbus_fast-2.24.3-cp312-cp312-x86_64.whl
fnv_hash_fast-1.0.2-cp312-cp312-x86_64.whl
habluetooth-3.7.0-cp312-cp312-x86_64.whl
kasa_crypt-0.5.0-cp312-cp312-x86_64.whl
ulid_transform-1.0.2-cp312-cp312-x86_64.whl
zeroconf-0.136.2-cp312-cp312-x86_64.whl

ERROR: bluetooth_data_tools-1.20.0-cp312-cp312-x86_64.whl is not a supported wheel on this platform.

my working solution in spksrc.crossenv.mk (at runtime tested with x64 only)

# map archs to WHEEL_ARCH
WHEEL_ARCH =
ifeq ($(findstring $(ARCH), $(x64_ARCHS)),$(ARCH))
WHEEL_ARCH = x86_64
else ifeq ($(findstring $(ARCH), $(i686_ARCHS)),$(ARCH))
WHEEL_ARCH = i686
else ifeq ($(findstring $(ARCH), $(ARMv7_ARCHS) $(ARMv7L_ARCHS)),$(ARCH))
WHEEL_ARCH = armv7l
else ifeq ($(findstring $(ARCH), $(ARMv8_ARCHS)),$(ARCH))
WHEEL_ARCH = aarch64
else ifeq ($(findstring $(ARCH), $(PPC_ARCHS)),$(ARCH))
WHEEL_ARCH = ppc
endif

ifneq ($(WHEEL_ARCH),)
MANYLINUX = --manylinux manylinux2014_$(WHEEL_ARCH)
endif

I don't have another idea how to fix the build of wheels that use poetry-core build-system...

[th0ma7]

Out of curiosity, if you rename the files to have a linux tag does that solve the issue? Also in the wheel log there is now a more verbose message related to the final renaming, maybe that's where the issue is?

Away from my build system atm.

[th0ma7]

@hgy59 I was finally able to reproduce your issue. I thought this was general but no, it is specific to poetry baased wheel builds.

From my reading (and chatgpt'ing), other than manylinux option there could be worth testing :

1. declaring CROSSENV_TARGET_PLATROFM="linux" when creating the crossenv
2. adding --target=<arch-triplet> at crossenv creation (such as x86_64-linux-gnu)
3. adding --platform linux_<arch> --only-binary=:all: to the pip wheel call (although you may have tested that already?)
4. passing --plat-name linux_<arch>

As maylinux could have other side-effects, and there's no way of identifying poetry dependent wheels, an alternative would be interesting.

EDIT: Adding --machine <arch-triplet> worked, at least with x86_64:

===>  _PYTHON_HOST_PLATFORM="x86_64-pc-linux-gnu" PATH=/home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/bin:/home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/cross/bin:/home/spksrc/wheel-update4/spksrc/distrib/cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games::/home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/build/bin: CMAKE_TOOLCHAIN_FILE=/home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/tc_vars.cmake MESON_CROSS_FILE=/home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/tc_vars.meson /home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/bin/cross-python -m pip wheel --disable-pip-version-check --no-binary :all: --find-links /home/spksrc/wheel-update4/spksrc/distrib/pip --cache-dir /home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/pip --no-deps --wheel-dir /home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/wheelhouse --no-index --no-build-isolation zeroconf==0.136.2
Looking in links: /home/spksrc/wheel-update4/spksrc/distrib/pip
Processing /home/spksrc/wheel-update4/spksrc/distrib/pip/zeroconf-0.136.2.tar.gz
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: zeroconf
  Building wheel for zeroconf (pyproject.toml): started
  Building wheel for zeroconf (pyproject.toml): finished with status 'done'
  Created wheel for zeroconf: filename=zeroconf-0.136.2-cp312-cp312-x86_64_linux_gnu.whl size=2521007 sha256=2b97b11c40561f41c171ab454df4ab8d2956f4a9fa09f6b86eeea86f9ea0a944
  Stored in directory: /home/spksrc/wheel-update4/spksrc/spk/python312-wheels/work-x64-7.1/pip/wheels/09/30/3b/6f73304f484b833a3c2ec4df8ec0e3ca52ad6817577ee92b9e
Successfully built zeroconf
===>  Installing wheel [zeroconf], version [0.136.2], type [crossenv]
===>  Adding zeroconf==0.136.2 to wheelhouse/requirements-crossenv.txt
===>  Copying requirements.txt to wheelhouse
===>  Copying to wheelhouse: zeroconf-0.136.2-cp312-cp312-x86_64_linux_gnu.whl -> share/wheelhouse/zeroconf-0.136.2-cp312-cp312-linux_x86_64.whl

[mreid-tt]

Hopefully, we can complete successful tests and get this merged, allowing us to move on to the numpy issue.

[hgy59]

I was finally able to reproduce your issue. I thought this was general but no, it is specific to poetry baased wheel builds.

❤️

with this, my homeassistant 2025.1.4 for x64 works.

there are other wheels (beside numpy) that fail to build and the installer will take those from the index.
(some, like grpcio, where already moved to requirements-pure.txt for this reason)

av==13.1.0
awscrt==0.23.5
deebot_client==11.0.0
grpcio==1.67.1
# isal fails to cross compile...
isal==1.7.1

numpy==2.2.0

# pandas: maturin error
pandas==2.2.3

unfortunately I can test homeassistant with py312 on x64-71 only.
my armv7 is DSM 7.1 and is waiting for #6337 to be fixed.
my aarch64 is still on DSM 6 and HA requires now DSM 7.

[hgy59]

I expected to name the new log files

• build-crossenv-*.log
• build-wheel-*.log

without the build- prefix, I guess some clean* targets need an update.

but this can be done later...

[th0ma7]

I tried that first but the ordering wasn't optimal. Using build-<arch>-<tcversion>-crossenv.log and build-<arch>-<tcversion>-wheel.log would make things orderred when listing log files.

I suggest either using proposal above or trying current for a little while and reconvene later?

[hgy59]

your proposal sounds good, but as I said: no priority (I am longing for the merge)...

[hgy59]

LGTM

---

Unfortunately, the build log is (in my opinion) too detailed.
we could remove the logs like
===> Processing wheel for {packagename}
and
WARNING: Skipping download - pure python packaging disabled
and
WARNING: Skipping building pure Python wheel [{name}], version [{version}], type [pure] - pure python packaging disabled

---

And when skipping the build of wheels (due to existing *_done files) there are also a lot of obsolete logs.

===>  make ARCH=x64 TCVERSION=7.1 REQUIREMENT="/spksrc/spk/homeassistant_2025/src/requirements-crossenv2.txt" REQUIREMENT_GOAL="wheel" requirement
===>  Processing wheel for homeassistant
===>  Processing requirement file [/spksrc/spk/homeassistant_2025/src/requirements-crossenv2.txt:aioesphomeapi==28.0.0]
===>  make ARCH=x64 TCVERSION=7.1 REQUIREMENT="aioesphomeapi==28.0.0" WHEEL_TYPE="crossenv" REQUIREMENT_GOAL="wheel" requirement
===>  Processing wheel for homeassistant
===>  Processing requirement [aioesphomeapi==28.0.0]
===>  make ARCH="x64" TCVERSION="7.1" WHEEL_NAME="aioesphomeapi" WHEEL_VERSION="28.0.0" WHEEL_TYPE="crossenv" WHEEL_URL="" wheel
make[5]: Nothing to be done for 'wheel'.

it is not only the amount of log output, it is mainly the time it takes to evaluate for each nothing to be done.

---

This has lower priority and must not delay the merge of this PR.

[th0ma7]

@hgy59 and @mreid-tt I noted two things missing:

1. in order to entirely skip pure-python files when variable not declared. This will avoid passing thru the entire file and skipping each and every single one of them, thus reducing log output and expediting execution. Although note that subsequent WARNING msg helps when forcing pure:<wheel> which otherwise isn't manage from the pure requirement file.
2. to close the loop on the --machine flag needed for poetry based wheels, I want to test using arch declaration without the gnu flag which would avoid the need for renaming when copying to the target/share/wheelhouse.

[hgy59]

1. in order to entirely skip pure-python files when variable not declared. This will avoid passing thru the entire file and skipping each and every single one of them, thus reducing log output and expediting execution. Although note that subsequent WARNING msg helps when forcing pure:<wheel> which otherwise isn't manage from the pure requirement file.

It sounds good for me to entirly skip the pure-python files. We could then copy the requirements-pure file to the wheelhouse folder as is. To ensure all pure requirements exist, a developer should once build with included pure-wheels for validation (or maybe we can add a pure-requirements check without downloading the wheels?).

[th0ma7]

I'll look into the skipping of pure python requirement into a subsequent PR. This one has been undergoing long enough.

[mreid-tt]

llfuse fails to build for python313:

      src/llfuse.c: In function ‘__Pyx_PyInt_As_gid_t’:
      src/llfuse.c:60451:23: error: too few arguments to function ‘_PyLong_AsByteArray’
                       ret = _PyLong_AsByteArray((PyLongObject *)v,
                             ^~~~~~~~~~~~~~~~~~~
      In file included from /spksrc/spk/python313/work-x64-7.1/install/var/packages/python313/target/include/python3.13/longobject.h:107,
                       from /spksrc/spk/python313/work-x64-7.1/install/var/packages/python313/target/include/python3.13/Python.h:81,
                       from src/llfuse.c:16:
      /spksrc/spk/python313/work-x64-7.1/install/var/packages/python313/target/include/python3.13/cpython/longobject.h:111:17: note: declared here
       PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,
                       ^~~~~~~~~~~~~~~~~~~

When we revisit this, we may need to apply a patch to lfuse-1.5.1/src/llfuse.c. Looking at the source code at python3.13/cpython/longobject.h, we see the following definition:

PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,
    unsigned char* bytes, size_t n,
    int little_endian, int is_signed, int with_exceptions);

As such, where we have this function call we would need to patch it to send an extra int value. Based on a similar fix in mkleehammer/pyodbc#1361, we could just add a , 1 to the function calls.

[mreid-tt]

hey @th0ma7, I believe this commit has broken our build process for non-Python packages. I was recently working with #6448 and saw the following failures:

===>  Processing wheel for dotnet-runtime
make[2]: *** [../../mk/spksrc.wheel-install.mk:72: install_python_wheel] Error 1
make[2]: Leaving directory '/github/workspace/spk/dotnet-runtime'
make[1]: *** [../../mk/spksrc.supported.mk:74: build-arch-x64-7.1] Error 1
make[1]: Leaving directory '/github/workspace/spk/dotnet-runtime'
make: Leaving directory '/github/workspace/spk/dotnet-runtime'
make: Entering directory '/github/workspace/spk/dotnet-runtime'
rm -fr work work-* build-*.log publish-*.log status-*.log
make: Leaving directory '/github/workspace/spk/dotnet-runtime'

Don't know if this is an easy fix but any reason it's trying to execute install_python_wheel at all?