Use maturin as build-backend and update Release PyPi GitHub workflow

[yvan-sraka]

Closes #105
Closes #75

We would want to close #176 too

[torymur]

Agree with the steps, especially this:

merging jobs wasn't the best approach

And we need to take into consideration, that currently for macos builds there is inconsistency #168, I will fix this using macos-latest: #169
But this reopens the issue to have builds both for macos-14 & macos-15: #75

Maybe having jobs per each os with strategy matrix per python version and target:

  release:
    name: Release to PyPI
    runs-on: ubuntu-latest
    needs:
      - linux
      - windows
      - macos-14
      - macos-15
      - linux-cross

[yvan-sraka]

 __________________ ERROR collecting tests/test_json_schema.py __________________
ImportError while importing test module '/home/runner/work/outlines-core/outlines-core/tests/test_json_schema.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/_pytest/python.py:493: in importtestmodule
    mod = import_path(
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/_pytest/pathlib.py:549: in import_path
    mod = _import_module_using_spec(
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/_pytest/pathlib.py:715: in _import_module_using_spec
    spec.loader.exec_module(mod)  # type: ignore[union-attr]
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:184: in exec_module
    exec(co, module.__dict__)
tests/test_json_schema.py:5: in <module>
    from outlines_core.json_schema import build_regex_from_schema
E   ModuleNotFoundError: No module named 'outlines_core.json_schema'       

[yvan-sraka]

I gave a try to % maturin generate-ci > .github/workflows/release_pypi.yaml, WDYT?

[yvan-sraka]

Run python -m coverage combine
Combined data file .coverage.8cc0c411e24d7bac9c3565f8296dcf91
No source for code: '/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/outlines_core/__init__.py'.
Error: Process completed with exit code 1.

[torymur]

To address questions:

1. generated release file could be a good starting point (maybe, if it's easier), but it can't be used as it is, we need to do match exactly what we currently have and then iterate if needed
2. scopes seem okay, tests are failing, I think you might need to adjust the testing workflow to do maturin develop before running them

[torymur]

Fortunately, PR was quickly merged in tokenizers, so we're opting out from openssl in favor of rustls, which drastically simplifies absolutely everything for us 🎉

Some notes:

• adding dtolnay rust setup with swatinem action to hit the cache
• pinning linux to ubuntu-22.04 to avoid glibc versioning issues
• error: #error "ARM assembler must define __ARM_ARCH" for aarch64 manylinux needs to be pinned, from comment seems manylinux_2_28 with GCC 7.5.0 is working ok (and it does): https://github.com/rust-cross/manylinux-cross#manylinux_2_28
• ubuntu wheels builds are running fast (under 10 mins), but especially when hitting cache it's under 3 mins (for aarch64 too): https://github.com/dottxt-ai/outlines-core/actions/runs/13269377037/job/37044932298?pr=165

All is green now: https://github.com/dottxt-ai/outlines-core/actions/runs/13269673921/job/37045906346?pr=165

A couple more things to check, but we're close here.