Multiple issues to resolve for the 2022.4.0 release

[consideRatio]

As I've failed to resolve this myself, I've aimed to summarize known issues we have to fix leading to a release here in conda-forge. PRs to resolve things in this feedstock very much appreciated.

Goal definition

• Ensure we are multiple maintainers of this feedstock ([ci skip] Add maintainers #22).
• Feedstock's CI to go green for dask-gateway and dask-gateway-server version 2022.4.0 on PyPI.
• Build base and extras packages:
  • base packages
    • dask-gateway
      Published as py3-none-any on PyPI
    • dask-gateway-server
      Published as py3-none-[...] on PyPI where [...] is the four variations of combining the choices of linux/osx and amd64/arm64.
  • extras packages (related to extras_require in respective setup.py file)
    • dask-gateway-kerberos
    • dask-gateway-server-[jobqueue|local|slurm|yarn]
      Note that getting these packages built may not be essential and a possible compromise.

Notes

Python 3.10 / Py 38+ / No CPython / Included golang binary

Right now we publish one version per Python version (but not for Python 3.10). Preferably we would stop doing variations based on Python version as there is no need. But, we still need variations for dask-gateway-server which bundles a Golang executable binary that is to be made available for linux-amd64 / linux-arm64 / osx-amd64 / osx-arm64.

Arm64 support

In PyPI, we have with the 2022.4.0 release created wheels for dask-gateway-server built for Linux and Mac, both for amd64 and arm64. Previously, we didn't have wheels for arm64. I consider it an optional success if we manage to package for that as well, but I think osx-arm64 requires cross-compiling which may make the configuration need to adjust significantly.

Cross compiling for dask-gateway-server's golang binary can be done with GOOS and GOARCH, see https://github.com/dask/dask-gateway/blob/d0cb34d5a21d50998d157f7ede432eb9f0726d9b/dask-gateway-server/setup.py#L140-L156.

Decision about the extras packages

Would we be willing to stop providing variations for these? I think for dask-gateway we could simply declare build.noarch=python and we would be fine for all architectures, but by adjusting the dependencies based on windows or not, we may not be able to as seen in dask-gateway-kerberos. Similarly for dask-gateway-server, should we really provide all kinds of variations based on extras_require? I've failed to find documentation on this practice or if there are options on mapping extras_require to a conda-forge feedstock.

[AlbertDeFusco]

Something I had to do is write a conda_build_config.yaml file in the recipe/ directory to build on my Mac M1. Here's the file I use. This may help with the linux-aarch64 build issues.

MACOSX_DEPLOYMENT_TARGET:  # [osx]
  - 10.12                  # [osx]
macos_min_version:         # [osx]
  - 10.12                  # [osx]
go_compiler:
  - go-nocgo
cgo_compiler:
  - go-cgo
target_goos:
  - linux                      # [linux]
  - darwin                     # [osx]
  - windows                    # [win]
target_goarch:
  - 386                        # [x86]
  - amd64                      # [x86_64]
  - arm64                      # [arm64]
target_goexe:
  -                            # [unix]
  - .exe                       # [win]
target_gobin:
  - '${PREFIX}/bin/'           # [unix]
  - '%PREFIX%\bin\'            # [win]

[consideRatio]

Nice, i've tries to find a reference for this file a long time without success! Is this parsed by conda-build or conda-smithy? Do you know if its documented somewhere? How did you realize you could write such keywords etc?

[AlbertDeFusco]

The file is referenced here https://docs.conda.io/projects/conda-build/en/latest/resources/variants.html#creating-conda-build-variant-config-files.

I found a template in another feedstock and inferred the correct arch names. I don't remember which one, though.

[consideRatio]

Thank you so much @AlbertDeFusco for this input!!! It helps a lot!

@TomAugspurger I'm still struggling with what determines what restricts the conda-forge packages to certain platforms etc, but have now managed to produce 4 different outputs for the dask-gateway-server package that should be provided for linux-64 / linux-aarch64 / osx-64 / osx-arm64.

See https://github.com/conda-forge/dask-gateway-feedstock/pull/26/files#diff-ff61408cdc05bc9667deeadb55e4aaceb1371972076b6bf6934f9008920f2bd2R14-R40 for notes about this.

[consideRatio]

"dask-gateway-server-2022.4.0-py310ha8f183a_0.tar.bz2": {
      "build": "py310ha8f183a_0",
      "build_number": 0,
      "depends": [
        "aiohttp",
        "colorlog",
        "cryptography",
        "python >=3.10,<3.11.0a0",
        "python_abi 3.10.* *_cp310",
        "traitlets"
      ],

I'd very much like to not see python and python_abi pinned like that, but I can't get it to not pin it without using build.noarch=python. If someone knows on how to make a package be platform dependent, without being coupled to Python, that would be amazing.