pkgs/sage-conf_relocatable

[mkoeppe]

From sagemath/sage#31396, squashed.

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

So far, a wheel has been built for XCode python 3.8 on macOS 10.15. The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in pypi/support#985) has kindly been granted by the PyPI team.

The wheel declares dependencies (install-requires) to all Python packages in the Sage distribution that have extension modules. The dependencies are specific using @ to URLs on
https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2, where I have uploaded the binary wheels. Unfortunately, by PyPI policy, such @ references are not allowed for packages on PyPI.
Hence, the pip invocation needs to use a URL.

(See also https://twitter.com/mkoeppe_math/status/1378860285537054723)

Instructions for testing:

1. Make sure that the bin directory of the user installation scheme is in PATH. For example, on macOS:

$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin

2. Install the wheel in the user scheme:

$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl

3. Run the installed sage.

$ sage

Procedure for building the sage_conf wheel and compatible wheels:

For building wheels on macOS Catalina that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

or

tox -e local-macos-10.15-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

This creates (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

• src/pkgs/sage_conf-relocatable/dist/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
• src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

Build for macOS Big Sur (requires a machine running Big Sur or Monterey) (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

tox -e local-macos-11.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for macOS Monterey (requires a machine running Monterey)

tox -e local-macos-12.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for Linux (does not work yet):

$ TARGETS_PRE=Makefile tox -e docker-manylinux-2_24-minimal-python3.9 -- sage-wheels

Follow-up steps:

• also build a sagemath-standard wheel
• build wheels for manylinux using tox -e docker-manylinux....
  • wheel is much too big: 2.3GB
  • auditwheel complains: RuntimeError: Invalid binary wheel, found the following shared library/libraries in purelib folder:; so we should replace our current abuse of package data (install_data) by platlib (install_lib). pypa/build on Linux - extension / binary modules in purelib pypa/packaging-problems#542 (comment): subclass build_ext instead of build_py
• disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible -- for example by storing a tar file sage_root.tar (with symlinks) instead of including the sage root as a directory - https://docs.python.org/3/library/tarfile.html#tarfile.open
• modularized relocatable wheel version of sage_conf sagemath/sage#31602: split out sage_root from sage_conf and use install-requires in the sage_conf sdist/wheel with @ links to GH releases to the sage_root wheel and the built non-any wheels.

Same as sagemath/sage#36366