Web / Add Support for "Pyodide Packages"

[rmartin16]

What is the problem or limitation you are having?

Pyodide maintains a collection of popular Python packages that are compatible with the pyscript runtime.

It is currently difficult to use these packages within the Briefcase web workflows. For example, if you add numpy to the pyproject.toml requires list, and then briefcase run web, Pyodide gives a runtime error "Wheel platform 'win_amd64' is not compatible with Pyodide's platform".

Describe the solution you'd like

Briefcase static web format should support adding Pyodide packages as requirements to the app.

Describe alternatives you've considered

Manually add the needed package to pyscript.toml. This needs to happen each time the app is rebuilt.

packages = [
    "/static/wheels/helloworld-0.0.1-py3-none-any.whl",
    "/static/wheels/toga_core-0.3.0-py3-none-any.whl",
    "/static/wheels/toga_web-0.3.0-py3-none-any.whl",
    "/static/wheels/travertino-0.1.3-py3-none-any.whl",
    "numpy",
]

See the package loaded in the browser console:

[pyscript/main] Packages to install:  (5) ['/static/wheels/helloworld-0.0.1-py3-none-any.whl', '/static/wheels/toga_core-0.3.0-py3-none-any.whl', '/static/wheels/toga_web-0.3.0-py3-none-any.whl', '/static/wheels/travertino-0.1.3-py3-none-any.whl', 'numpy']0: "/static/wheels/helloworld-0.0.1-py3-none-any.whl"1: "/static/wheels/toga_core-0.3.0-py3-none-any.whl"2: "/static/wheels/toga_web-0.3.0-py3-none-any.whl"3: "/static/wheels/travertino-0.1.3-py3-none-any.whl"4: "numpy"]
pyodide.asm.js:10 Loading numpy
pyodide.asm.js:10 Loaded numpy

It does seem eventually that library maintainers will be able to provide packages on PyPI that will be compatible with browser runtimes.

In theory, it should also probably be possible to download Pyodide's numpy wheel but pip gets upset about the platform for the downloaded wheel.

>>> Running Command:
>>>     /home/russell/github/beeware/briefcase/venv-3.10-briefcase/bin/python -u -m pip wheel --wheel-dir '/home/russell/tmp/beeware/helloworld/web/static/Hello World/www/static/wheels' -r '/home/russell/tmp/beeware/helloworld/web/static/Hello World/requirements.txt'
>>> Working Directory:
>>>     /home/russell/tmp/beeware/helloworld
ERROR: numpy-1.22.4-cp310-cp310-emscripten_3_1_14_wasm32.whl is not a supported wheel on this platform.

[freakboy3742]

Agreed we should definitely support these wheels (and binary wheels in general) on the web backend.

The error about unsupported wheel formats should be relatively straightforward; we either need to pass in a --platform argument when invoking pip (we might need to pass in a couple of other flags as well), or do the trick we do with iOS, where we create a custom site package that does enough fakery to convince pip that the local Python is iOS.

A second approach would be to pre-process the requirements list, looking for the names that are known to be available as binaries, and handle them differently during the creation process. We do a version of this for local source packages on AppImage; we'd need to do different processing for web, but at first glance, it seems like it shouldn't be too complicated (other than the complication of knowing the list of packages pyodide provides).

Of course, the long term approach is for wasm packages to be made available on PyPI. However, as a short term solution,. we could provide those packages on a private PyPI repository (again, mirroring what we do on iOS). Ideally, we wouldn't have to take ownership of that task; I might chase this down with the PyScript team at Anaconda and see what they say.

[rmartin16]

The error about unsupported wheel formats should be relatively straightforward; we either need to pass in a --platform argument when invoking pip

As a note:

I started down this rabbit hole after someone in Discord couldn't use aiohttp because Pyodide was erroring on the wheel because it was for macosx. So, I initially just wanted to update briefcase build web to error if an appropriate wheel for emscription wasn't available instead of letting users send entirely incompatible wheels to the browser just to throw a potentially confusing error there.

However, the CPython team seems to be waiting on a PEP to officially allow wheels for wasm on PyPI....an important part of that seemingly being what even delineates wasm platforms.

[reynoldsnlp]

@hoodmane Thanks for your help yesterday with getting an index of pyodide wheels. My bash history does not have the commands you entered. Can you paste the code we can use to get an index of pyodide wheels?

[reynoldsnlp]

The code to get a local index of pyodide wheels is...

pip install pyodide-build
pyodide venv .pyodide-venv

The index is now located in the directory listed under --extra-index-url in .pyodide-venv/pip.conf

[rmartin16]

A bit more lightweight but potentially ultimately untenable source for available packages:

curl -s https://cdn.jsdelivr.net/pyodide/v0.22.1/full/repodata.json | jq .packages[].name

[reynoldsnlp]

@rmartin16 why is that less tenable than building it from pyodide-build?

[rmartin16]

@rmartin16 why is that less tenable than building it from pyodide-build?

Sure; my qualification in that sentence stems from my ignorance of this JSON resource. I found it by looking at what PyScript does when it loads in the browser. So, I'm not sure if this is intended for random use by the public. Additionally, it's a blob of JSON on a CDN; I'd prefer a more canonical link that redirects to a CDN.

Additionally, though, IIRC, Briefcase is not currently controlling the version of PyScript being used; however, this JSON of the packages is inherently tied to the version. I am not sure if there's any need to keep the two in sync but I, at least, imagine the JSON version would need to be updated to get updates to the package list.

So, I'm just not sure if this is a "good idea" basically. More research is needed.....but I think ideally we'd want be able to dynamically request some sort of payload under PyScript's control for this package list....maybe they even have a pypi server set up...

[mhsmith]

I think we're making things unnecessarily hard on ourselves here by going against the flow of how PyScript is designed. If the only reliable way of installing binary packages on Pydodide is to use their own package installer, then Briefcase should just do that instead of running pip itself.

Specifically, we can scan the requires list and process each item as follows:

• If it's a local file, copy it into the static wheels directory.
• If it's a local directory, build a wheel from it and put it into the static wheels directory.
• Otherwise, copy the requirement into the pyscript.toml file unchanged.

This is similar to what we already do on Android, which depends on a forked copy of pip.

[rmartin16]

I think I'd be fine with this. However, the current design is more resilient to supply-chain attacks. If we don't let the app deployment serve the dependencies then each user will fetch them from some index. So, unlike Android, this would be different in that dependencies are not "vendored" (so to speak).

[edit]
As long as the pyscript pip supports it, an obvious protection is including hashes with the requirements; but then your app is susceptible to breaking in prod if someone does replace the underlying artifacts in the index.

[hoodmane]

Pyodide has its own pip that installs packages for Pyodide. You can make a Pyodide virtual enviroment with:

pip install pyodide-build
pyodide venv .venv-pyodide

and then .venv-pyodide/bin/pip install -r requirements.txt will get your requirements.

[freakboy3742]

Having the app's packages be part of the deployed bundle was a deliberate decision, specifically because of the supply-side attack issue, as well as the "Perth is a long way from everything so resolving and downloading dependencies on first use can be slow" issue :-) It's also a partial solution to the "app works while offline" issue, but obviously we're dependent on pyscript's CDN at present, so it's not a complete solution.

If we can use pyodide's own tooling within a "pyodide venv", that sounds like it might be a viable approach. The other would be a solution to pyodide/pyodide#3049, which might just mean getting PyPI to accept Pyodide wheels (something that should be back on the cards with Tier 3 support being restored).

It's also interesting to reflect how the "using a pyodide venv" approach would fit nicely with the general "enable use of uv and/or conda" discussion that we've had going in the background - and would also fit nicely with the work I've been doing around cibuildwheel regarding creating an "iOS virtual environment" for the purposes of wheel building.

[hoodmane]

Yeah. Some day it would be nice to have an upstream solution for cross-venvs. @FFY00 has been gradually working towards this as I understand it but it's a lot of work.