Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Python version coverage to template CI. #55

Merged
merged 1 commit into from
Nov 1, 2022
Merged

Add Python version coverage to template CI. #55

merged 1 commit into from
Nov 1, 2022

Conversation

freakboy3742
Copy link
Member

Adds python versions to the test matrix for template builds.

PR Checklist:

  • All new features have been tested
  • All new features have been documented
  • I have read the CONTRIBUTING.md file
  • I will abide by the code of conduct

@freakboy3742
Copy link
Member Author

@mhsmith The test failures for 3.11 are to be expected; once you've published 3.11 support to Maven Central, you should be to able to re-start the failed jobs and get a clean board for merge.

@freakboy3742 freakboy3742 requested a review from mhsmith October 31, 2022 05:22
@mhsmith
Copy link
Member

mhsmith commented Oct 31, 2022

I haven't published 3.11.0 to Maven Central yet, because there's something strange about the file sizes which didn't happen with the 3.11 RC:

target-3.11.0-2-arm64-v8a.zip
   264368  2022-10-31 16:48   lib-dynload/arm64-v8a/_codecs_jp.so
   298808  2022-10-31 16:48   lib-dynload/arm64-v8a/_decimal.so
   604912  2022-10-31 16:48   jniLibs/arm64-v8a/libssl_chaquopy.so
   613416  2022-09-17 18:34   include/sqlite3.h
  1011840  2022-10-31 16:48   jniLibs/arm64-v8a/libsqlite3_chaquopy.so
  1129192  2022-10-31 16:48   lib-dynload/arm64-v8a/unicodedata.so
  3334336  2022-10-31 16:48   jniLibs/arm64-v8a/libcrypto_chaquopy.so
  5035872  2022-10-31 16:48   jniLibs/arm64-v8a/libpython3.11.so
  5076824  2022-10-31 16:48   lib-dynload/arm64-v8a/_pickle.so
 22692548                     395 files
target-3.11.0-2-armeabi-v7a.zip
   222916  2022-10-31 16:48   lib-dynload/armeabi-v7a/_codecs_jp.so
   228668  2022-09-17 19:57   include/openssl/obj_mac.h
   237040  2022-10-31 16:48   lib-dynload/armeabi-v7a/_decimal.so
   457136  2022-10-31 16:48   jniLibs/armeabi-v7a/libssl_chaquopy.so
   613416  2022-09-17 18:34   include/sqlite3.h
   733084  2022-10-31 16:48   jniLibs/armeabi-v7a/libsqlite3_chaquopy.so
  1116400  2022-10-31 16:48   lib-dynload/armeabi-v7a/unicodedata.so
  2383776  2022-10-31 16:48   jniLibs/armeabi-v7a/libcrypto_chaquopy.so
  3397180  2022-10-31 16:48   jniLibs/armeabi-v7a/libpython3.11.so
 14086394                     395 files
target-3.11.0-2-x86_64.zip
   249176  2022-10-31 16:48   lib-dynload/x86_64/pyexpat.so
   268464  2022-10-31 16:48   lib-dynload/x86_64/_codecs_jp.so
   335672  2022-10-31 16:48   lib-dynload/x86_64/_decimal.so
   613416  2022-09-17 18:36   include/sqlite3.h
   711408  2022-10-31 16:48   jniLibs/x86_64/libssl_chaquopy.so
  1129192  2022-10-31 16:48   lib-dynload/x86_64/unicodedata.so
  1134720  2022-10-31 16:48   jniLibs/x86_64/libsqlite3_chaquopy.so
  3956928  2022-10-31 16:48   jniLibs/x86_64/libcrypto_chaquopy.so
  5449560  2022-10-31 16:48   jniLibs/x86_64/libpython3.11.so
 19205902                     395 files
target-3.11.0-2-x86.zip
   228668  2022-09-17 19:59   include/openssl/obj_mac.h
   294152  2022-10-31 16:48   lib-dynload/x86/_decimal.so
   600228  2022-10-31 16:48   jniLibs/x86/libssl_chaquopy.so
   613416  2022-09-17 18:35   include/sqlite3.h
  1007196  2022-10-31 16:48   jniLibs/x86/libsqlite3_chaquopy.so
  1120464  2022-10-31 16:48   lib-dynload/x86/unicodedata.so
  3210908  2022-10-31 16:48   jniLibs/x86/libcrypto_chaquopy.so
  4105592  2022-10-31 16:48   lib-dynload/x86/_queue.so
  4146460  2022-10-31 16:48   jniLibs/x86/libpython3.11.so
 20495171                     395 files

arm64-v8a/_pickle.so and x86/_queue.so have both massively increased in size. In fact, they're both about the same size as libpython, which suggests they've somehow ended up statically linked against it.

I'll look into this tomorrow.

@mhsmith
Copy link
Member

mhsmith commented Nov 1, 2022

It looks like python/cpython#93977 introduced a bug that when cross-compiling, the Makefile no longer requires libpython to be built before it calls setup.py to build the stdlib binary modules. But exactly what happens is non-deterministic because of the parallel build. setup.py might fail to build all modules, or only some of them, or it might link some against libpython.a and some against libpython.so.

The reason I didn't notice this before is that after make we call make install, which calls setup.py again and gives it a second chance to build any modules that failed. But of course, if any modules were already linked against libpython.a, it won't build them again.

@freakboy3742: You might want to check whether this affects iOS as well. I worked around it with this patch.

@mhsmith mhsmith merged commit 6f0a5ea into main Nov 1, 2022
@freakboy3742 freakboy3742 deleted the py-versions branch November 1, 2022 22:21
@freakboy3742
Copy link
Member Author

@freakboy3742: You might want to check whether this affects iOS as well. I worked around it with this patch.

@mhsmith That's for the heads up; that's a nasty one. Looks like the Apple builds aren't affected though - the Apple Support package Makefile don't enable the parallel build options in the CPython makefile, so the core library is always built before the stdlib is compiled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants