support for musl environment

[dsp]

This is an experimental pull request aimed to support environments.

Problem

Linux and in rare cases windows binaries can be built with different environments. At the moment rye assumes we will be always using a gnu environment on linux. This is not true for distributions like Alpine, which use musl instead of glibc. Hence, rye currently does not support Alpine Linux.

Approach

This PR introduces the notion of an environment to our internal handling of python versions and python version requests. It extends the version matching to include environments, and modifies our download link creators for uv and cypthon builds to include the environment. Lastly, it changes how we link libcurl to make builds for Alpine linux working.

Once the PR is merged, we can build and use rye on alpine.

Notable, there are some caveats to the current implementation. I decided to do this in multiple PRs to not overload the PR with more.

Status of implementation

• We introduce the notion of a "environment", which can either be unset (None) or set to a string.
• We changed rye-devtools to include the environment for all uv and cpython builds.
• We modified match_versions to correctly resolve toolchain.
• We define the default_toolchain based on the target_env, so based on the target triple rye was build for. This is deliberate, as we have no easy way to find the target_env at runtime (e.g. a system that provides both gnu and musl, is completely valid on Linux).
• We add a cargo feature to select the features of the curl crate we are using, to allow linking to system openssl on alpine.
• Added tests to bootstrap on different environments (gnu and alpine). We ran manual tests to compile and run on Alpine, see: https://asciinema.org/a/REzOhL77l95VsqaBSuOxf9pH5

Caveats

Missing CI integration

There is currently not CI integration for Alpine linux or one that links against musl on Ubuntu. However, we did manually test it.

Python Storage Paths

We currently still save python into [RYE_HOME]/py/[email protected]. This can lead to problems where a user would have a both a musl and a gnu version. It is something that we should support, particularly when we allow users to more easily select their environments within a project. However it is out of scope of the current PR.

User selected environments

Users cannot easily select a python version for a specific environment. For example rye pin cpython-linux-musl doesn't work. we should make this work in the future.

[mitsuhiko]

The script that generates this out is this one here: https://github.com/mitsuhiko/rye/blob/main/rye/find-downloads.py

I would like to avoid the term "toolchain" here as this term is already used to refer to the python installations itself. I'm not sure what a good term is, but toolchain probably not.

[rdbo]

Rust uses the term target_env in Conditional Compilation (https://doc.rust-lang.org/reference/conditional-compilation.html#target_env) to disambiguate between targets. So I guess "env" or "environment" could be candidates for a different term (even though it is not exactly the same thing, since the target_env can be empty).

[mitsuhiko]

This appears to be a common structure for target triples: https://docs.rs/target-lexicon/latest/target_lexicon/struct.Triple.html

pub struct Triple {
    pub architecture: Architecture,
    pub vendor: Vendor,
    pub operating_system: OperatingSystem,
    pub environment: Environment,
    pub binary_format: BinaryFormat,
}

What this refers to is in fact called environment.

[dsp]

I'll change it to environment.

[dsp]

rebased

[dsp]

rebased again

[mitsuhiko]

Playing with this still. Pretty determined to merge but I need to figure out how this actually plays out in practice still.

[mitsuhiko]

I'm temporarily holding on until I have the test system in place.

Refs #687

[dsp]

I am adding tests for this. There is a problem that currently makes using Indygreg's python3 musl packages unusable for rye:

1. rye uses python3 -mvenv --upgrade-deps to bootstrap.
2. This will call ensurepip, which bootstraps pip from a bundled wheel and calls pip.
3. pip fails when trying to detect libc version. It asks for the libc version to set it as a user-agent for any request it makes. This uses ctypes.CDLL, which fails for indygregs python3 with "Dynamic loading not supported".

There are patches for this in indygregs repo, but they don't affected the bundled wheel which ensurepip uses during bootstrapping.

Standard alpine musl python allows dynamic loading and hence is not affected.

Putting this on hold until it's resolved upstream, or someone has a good idea how to go about it.

[mitsuhiko]

@dsp it would be interesting to know if uv has the same issue. Because one option would be to do away with pip to bootstrap but manually fetch a matching uv release and bootstrap via that one.

[dsp]

Okay, I hope this is finally ready. I updated the description. Manually tested on Alpine Linux, but not tested on a GNU system with musl installed.

[mitsuhiko]

I would like to push out the current main branch of rye as a release, and then schedule this to go into the release after. The current changes are mostly bugfixes.

[dsp]

@mitsuhiko Thank you! Looking forward to having it in! I'll rebase it one more time today.

Note that likely the merge will fail if you update uv releases or python releases in between. You would have to regenerate them.

[tuananh]

Thanks @dsp for working on this

[tuananh]

olps, this one doesnt make it in 0.33.0 release :(

[dsp]

If someone wants to pick it up, rebase and check why the test is failing, please go ahead. I still think this is valuable but won't have time to work on it. Since I am personally not running Alpine, it's not too relevant for me (although a bit sad since I poured quite a bit of time into it)

[dsp]

I also assume we have it wait for @mitsuhiko to be back. I think it's quite a big and scary feature, so I totally see why it is not merged currently while @charliermarsh is helping out with more straight forward PRs.

[dsp]

Since there is no more traction since a while I am going to close it out.

[bluss]

uv seems to have support for musl environment, so I hope it will carry the torch.

[charliermarsh]

In theory we could always use the uv musl builds which would simplify some things. (We already use them always for ARM Linux.)

Do we also need to change the deployment pipeline to ship a musl build of Rye?

[dsp]

I think there are a few pieces to the puzzle if i remember correctly:

1. Python musl support:
   1. Support for selecting musl builds
   2. Automatic detectin of musl vs gnu environment
2. UV support for musl
   1. Support for selecting uv builds that are musl compatible (or just buil uv direclty into rye)
3. Rye musl support:
   1. I punted on this one, and left it to Alpine & Co to support this for now. Likely needs disabling auto-updates on musl.

I think the PR dealt with basically 1 and 2, but didn't care about Rye support for musl just yet. My position was a bit like: musl rare enough that we should just disable auto-update and have it