Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-40280: Add Tools/wasm with helpers for cross building (GH-29984) #29984

Merged
merged 5 commits into from
Dec 18, 2021

Conversation

tiran
Copy link
Member

@tiran tiran commented Dec 8, 2021

@tiran tiran force-pushed the bpo-40280-wasm-tools branch from 3305617 to 877bb04 Compare December 13, 2021 20:03
@tiran tiran marked this pull request as ready for review December 13, 2021 20:10
@tiran tiran requested a review from brettcannon December 13, 2021 20:10
@tiran tiran force-pushed the bpo-40280-wasm-tools branch from 877bb04 to f65a480 Compare December 13, 2021 20:10
@tiran
Copy link
Member Author

tiran commented Dec 13, 2021

$ CONFIG_SITE=../../Tools/wasm/config.site-wasm32-emscripten \
  emconfigure ../../configure -C \
    --host=wasm32-unknown-emscripten \
    --build=$(../../config.guess) \
    --with-build-python=$(pwd)/../build/python
$ emmake make -j$(nproc) python.html
$ ls -s --si python.*
3.0M python.data
107k python.html
361k python.js
6.6M python.wasm

Copy link
Contributor

@emmatyping emmatyping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will definitely make experimenting with Emscripten based CPython a lot easier!

I have a few suggestions/comments but overall this looks good.

Tools/wasm/wasm_assets.py Outdated Show resolved Hide resolved
Tools/wasm/wasm_assets.py Outdated Show resolved Hide resolved
Tools/wasm/README.md Outdated Show resolved Hide resolved
Tools/wasm/wasm_assets.py Show resolved Hide resolved
Tools/wasm/wasm_assets.py Outdated Show resolved Hide resolved
@emmatyping
Copy link
Contributor

The binary size looks really great! Though with the number of things disabled, it will probably grow over time...

@tiran tiran force-pushed the bpo-40280-wasm-tools branch from a3797c0 to 6a103d9 Compare December 14, 2021 08:56
@tiran
Copy link
Member Author

tiran commented Dec 14, 2021

With --disabled-test-modules, without decimal unittest xml and without some C extension

# Modules/Setup.local
*disabled*
_decimal
pyexpat
_elementtree
_lsprof
audioop

I'm able to shrink the core to about 6 MB

# du --si -s -c python.*
2.7M    python.data
107k    python.html
275k    python.js
3.3M    python.wasm
6.4M    total

The python.wasm file can be compressed well. If we assume that all browsers support transparent compression, then we are down to about 4 MB with room for more removals.

# du --si -s -c python.*
2.7M    python.data
107k    python.html
275k    python.js
1.1M    python.wasm.gz
4.2M    total

The largest 25 files are:

# unzip -l usr/local/lib/python311.zip  | sort -n | tail -n 26
    57953  2021-12-13 19:58   threading.pyc
    59930  2021-12-13 19:59   optparse.pyc
    61650  2021-12-13 19:58   subprocess.pyc
    61916  2021-12-13 19:56   collections/__init__.pyc
    63723  2021-12-13 19:56   enum.pyc
    63980  2021-12-13 19:59   pickle.pyc
    66135  2021-12-13 19:59   pdb.pyc
    67936  2021-12-13 19:58   locale.pyc
    68530  2021-12-13 19:58   asyncio/base_events.pyc
    70300  2021-12-14 09:56   _sysconfigdata__emscripten_.pyc
    71134  2021-12-13 19:59   difflib.pyc
    77197  2021-12-13 19:58   datetime.pyc
    77293  2021-12-13 19:59   ipaddress.pyc
    78354  2021-12-13 19:56   ast.pyc
    79675  2021-12-13 19:59   pickletools.pyc
    82993  2021-12-13 19:58   logging/__init__.pyc
    85730  2021-12-13 19:56   zipfile.pyc
    86382  2021-12-13 19:59   mailbox.pyc
    86537  2021-12-13 19:56   argparse.pyc
    88029  2021-12-13 19:59   tarfile.pyc
    92877  2021-12-13 19:59   doctest.pyc
   108300  2021-12-13 19:56   inspect.pyc
   109640  2021-12-13 19:58   typing.pyc
   118759  2021-12-13 19:59   pydoc.pyc
   253750  2021-12-13 19:59   html/entities.pyc
  5711264                     347 files

@brettcannon now is a good time to discuss what modules belong in a minimum stdlib.

@emmatyping
Copy link
Contributor

Wow that's a great reduction in size! I played with thin lto and for some reason it greatly increases the binary size (to 21MB!). I guess inlining makes the build a lot bigger. One other idea I had which gives small gains in binary size is compiling with -Os, which seems to not make much of a difference vs -Oz.

@tiran
Copy link
Member Author

tiran commented Dec 14, 2021

Actually it is less. I think I introduced a regression in GH-30001. makesetup is dropping too many modules from the final binary.

@tiran tiran force-pushed the bpo-40280-wasm-tools branch 2 times, most recently from 59d8c8e to 218aef5 Compare December 14, 2021 14:47
@tiran tiran force-pushed the bpo-40280-wasm-tools branch from 218aef5 to f933b91 Compare December 17, 2021 16:05
Tools/wasm/README.md Outdated Show resolved Hide resolved
Tools/wasm/README.md Outdated Show resolved Hide resolved
## wasm32-emscripten build

Cross compiling to wasm32-emscripten platform needs [Emscripten](https://emscripten.org/)
tool chain and a build Python interpreter.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
tool chain and a build Python interpreter.
tool chain and a built Python interpreter.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build here is I believe a reference to the autotools/configure --build argument. (Same for line 11 below).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. In lack of a better term I introduced the term "build Python"/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My brain just keeps reading "build" as a verb instead of an adjective. 😄

Tools/wasm/README.md Show resolved Hide resolved
Tools/wasm/README.md Show resolved Hide resolved
Tools/wasm/wasm_assets.py Show resolved Hide resolved
Tools/wasm/wasm_assets.py Outdated Show resolved Hide resolved
Tools/wasm/wasm_assets.py Outdated Show resolved Hide resolved
Tools/wasm/wasm_assets.py Outdated Show resolved Hide resolved
Tools/wasm/wasm_assets.py Outdated Show resolved Hide resolved
@tiran tiran changed the title bpo-40280: Add Tools/wasm with helpers for cross building bpo-40280: Add Tools/wasm with helpers for cross building (GH-29984) Dec 18, 2021
@tiran tiran merged commit 0339434 into python:main Dec 18, 2021
@tiran tiran deleted the bpo-40280-wasm-tools branch December 18, 2021 14:54
tiran added a commit to tiran/python-wasm that referenced this pull request Dec 18, 2021
CPython upstream now has a config.site for WASM `and make python.html`.

See: python/cpython#29984
Signed-off-by: Christian Heimes <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants