Add all buildable packages to osx-arm64

[jonashaag]

This adds all packages to osx-arm64 that I was able to build on my M1 machine without modifications.

Checklist

• Used a personal fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jaimergp]

Oh wow, thanks for the effort! I wonder though if this amount of packages can functionally DoS our bot?

[beckermr]

The bot will do fine, but we cannot merge this.

Moving recipes over to the M1 is a burden on our maintainers and our cross-compiling architecture. Even if it builds natively, things can fail when cross-compiling.

Please add only what you need.

[jonashaag]

How about I attempt to cross-compile all of those packages and keep only the ones that cross-compile without modification?

[xhochy]

How about I attempt to cross-compile all of those packages and keep only the ones that cross-compile without modification?

That would be better but still quite some burden on the maintainers and quite informative. I would expect many of them to fail. Keeping this list down to things you really need is much better (even better if you also step in as the maintainer of some of these packages).

[wolfv]

Just FYI before attempting to cross compile -- there is some "magic" somewhere in the bot that modifies the build.sh scripts slightly (e.g. injecting ${CMAKE_ARGS} for cmake, or autotools reconf and things like this. Also cross_python sometimes is automatically added to teh host dependencies.
Unfortunately I can't tell you exactly where that source is and how you could use it.

[jonashaag]

My motivation for this was that some M1 users (most, actually) don't know how to build Conda packages for M1. Not even simply adding the packages to this repo. What would be a way forward to help those people, in a way that doesn't put too much burden on the Conda-forge maintainers?

[xhochy]

Just FYI before attempting to cross compile -- there is some "magic" somewhere in the bot that modifies the build.sh scripts slightly (e.g. injecting ${CMAKE_ARGS} for cmake, or autotools reconf and things like this. Also cross_python sometimes is automatically added to teh host dependencies. Unfortunately I can't tell you exactly where that source is and how you could use it.

Most of them are in https://github.com/regro/cf-scripts/blob/449368112dd90c4155934b526475aa9065647b86/conda_forge_tick/migrators/cross_compile.py#L1

[xhochy]

My motivation for this was that some M1 users (most, actually) don't know how to build Conda packages for M1. Not even simply adding the packages to this repo. What would be a way forward to help those people, in a way that doesn't put too much burden on the Conda-forge maintainers?

1. Definitely try out whether cross-compilation works.
2. Do it by popularity, do it in small chunks.
3. First check the graph in https://conda-forge.org/status/#armosxaddition The packages in that graph are ones that were requested but don't build yet. These actually provide the highest value.

[jonashaag]

List of packages that can be cross-compiled, top 100 by number of downloads:

pandoc
enum34
backports.weakref
pathlib
quaternion
billiard
kombu
gensim
pycurl
compas
bitarray
newrelic
luigi
trio_asyncio
profilehooks
humanfriendly
ptvsd
nipype
wrf-python
pyomo
mrjob
fs
jpype1
xlwings
python-chromedriver-binary
cmd2
iteration_utilities
python-crfsuite
implicit
mysql-connector-python
jcc
pigz
pre_commit
elasticsearch-dsl
clickhouse-driver
apscheduler
aliyun-python-sdk-core
cherrypy
wordcloud
libsass
mpl_sample_data
falcon
promise
sagemaker-python-sdk
backports.os
hpc05
frozendict
doconce
cx_oracle
python-annoy
cchardet
thriftpy2
flask-restplus
inflect
mdanalysis
flask-restalchemy
kuyruk
oslo.config
fastrlock
blessings
pygram11
docker-compose
skein
sanic
adaptive
dpath
filebutler
pyvisa
sasl
plumpy
python-lzo
rebound
ciso8601
py-lz4framed
compas_fab
radical.pilot
pycares
schema-salad
azure-storage
naima
brian2
jupyter_dashboards
mongoengine
validictory
aws-parallelcluster
pyomo.extras
conda-wrappers
pymssql
anypytools
cis
wsproto
biotite
mercurial
pyahocorasick
sos-notebook
resonance
pytmatrix
py-ubjson
pyflann
zodbpickle

There are definitely some false-positives in there, eg. Pandoc, enum34 (archived).

[beckermr]

Thanks for this. If you actually use any of those packages, please submit them in a new PR.

[ngam]

Damn... it would'a been fun to see how merging this PR would'a unfolded 😆

@beckermr what's the endgame for these migrations usually? Does the bot have an expiration date where it goes all out or does keep going drip drip drip drip drip ad infinitum?

[beckermr]

For a migration like this we’ll just let it go forever. If we ever get enough CI support, we’ll likely build packages natively and turn on OS X arm by default. It’ll be at least a year for that or more is my guess.

[ngam]

For a migration like this we’ll just let it go forever. If we ever get enough CI support, we’ll likely build packages natively and turn on OS X arm by default. It’ll be at least a year for that or more is my guess.

A matter of time, we will definitely get CI support

I do wonder if the conda-forge community should petition Apple directly for some support, maybe they'd be so generous to give us something like this https://developer.apple.com/xcode-cloud/ (no way in hell would Apple do such a thing in general lol, but maybe if we fundamentally tweak how we build osx stuff...)

[manics]

Successfully building and publishing an osx-arm64 package doesn't mean it's correct, as I've unfortunately just discovered 😢.
The zeroc-ice osx-arm64 migration PR built successfully and I merged it, but the arm64 package contained x86_64 libraries:
conda-forge/zeroc-ice-feedstock#20 (comment)

This is due to the upstream package setup.py containing

if platform == 'darwin':
    if not 'ARCHFLAGS' in os.environ:
        os.environ['ARCHFLAGS'] = '-arch x86_64'

I'm testing a fix for that here conda-forge/zeroc-ice-feedstock#25

[BastianZim]

Yes, unfortunately the osx-arm64 packages are only built but not tested right now: https://conda-forge.org/docs/maintainer/conda_forge_yml.html#test

[psobolewskiPhD]

As a stopgap, could an osx_x86 host be able to use lipo -info to check any compiled components that they match the expected architecture?

[beckermr]

cc @conda-forge/core @isuruf Should we write this into the conda build checks?