Update all dependencies to get more that officially support Python 3.11

[dataflake]

No description provided.

[mauritsvanrees]

LGTM.
As you have seen, Python 3.11 on Mac OS fails on getting zope.interface 5.5.1. Locally (macOS Monterey 12.6) this works fine for me.
The zope.interface Mac OS egg for Python 3.10 is built in a specific way. Just a guess: would it help to extend this to Python 3.11 and get a new release out?

[mauritsvanrees]

Okay, you are aware of that, and there is more involved, as seen in the issue you created a week ago: zopefoundation/meta#166

[dataflake]

The build story on macOS is a mess. There are several interlocking issues:

• the GHA macos runners' Python interpreters are set up in a non-standard way, they are not built for the specific architecture of the runner (x86_64) but explicitly as universal2. That means whenever a buildout or setuptools or pip run wants to build software it is built as universal2 wheel. This does not match any local installs I have.
• I am unable to actually set up Python 3.11 for universal2 architecture on my local machine at all so I cannot directly reproduce the issues from GHA.
• Those GHA configurations are not comparable with what cibuildwheel does, that's yet another and totally different way of setting up Python. It actually creates a qemu virtual machine and installs the binary installer .pkg from python.org. So I cannot re-use the flags etc. they use when setting up a build environment.
• The failures themselves happen in subprocesses, I haven't managed to just use pdb to step through the affected code in setuptools. I think setuptools is the main issue here.

What I can say is that affected eggs like BTrees, zope.container or zope.security have some common characteristics:

• they want to build binary extensions
• they rely on header files that are not in the package itself but in other packages that must be downloaded and installed to read the files, this happens in a setup_requires dependency.

It seems like the installation from setup_requires succeeds, but when the setup.py bdist_XXX step is run to build the C extensions that dependency is downloaded and installed again, ignoring the already-installed egg from the setup_requires step.

Binary extension packages such as AccessControl or ExtensionClass maintain a copy of dependency header files in the package itself, they are not affected.

[dataflake]

P.S.: Funny thing, and a sign of how the differences in Python interpreter setup play into it as well:

• on GHA the Python 3.10 tests run successfully. Locally (macOS 13.1 on arm64) Python 3.10 reliably fails with that "directory not empty" issue that points to the erroneous attempt to re-build an egg that already exists.
• On GHA the Python 3.11 tests fail, but there's an underlying error that is not shown, even with debug logging active in GHA. However, locally Python 3.11 works just fine.

I wish I had an Intel Mac around to see if the architecture is a factor. Intel Macs should pick up the precompiled eggs as those are specifically built for x86_64, they should not see these issues.

[mauritsvanrees]

I wish I had an Intel Mac around to see if the architecture is a factor. Intel Macs should pick up the precompiled eggs as those are specifically built for x86_64, they should not see these issues.

I have a MacBook with Intel Core i9 running Monterey. This Zope branch works fine, all tox tests pass. In my eggs directory I now have cp311/zope.interface-5.5.1-py3.11-macosx-12.6-x86_64.egg

[dataflake]

So as I see it we have the following situation:

• except for the macOS/Python 3.11 combination GHA manages to create all eggs
• macOS/x86_64 is not affected at all, the tox tests run for all Python versions
• macOS/arm64 has problems building/testing on Python 3.10

Is that a blocker for a Zope 5 release? I don't know how much effort there is to find and tackle the underlying (setuptools?) issue. @icemac what do you think?

[dataflake]

@mauritsvanrees Just to be on the safe side, can you make sure your sandbox on macOS is fully clean? Like, do a pip cache purge and configure buildout to ignore a shared eggs folder so it really has to fetch/build everything?

[mauritsvanrees]

@mauritsvanrees Just to be on the safe side, can you make sure your sandbox on macOS is fully clean? Like, do a pip cache purge and configure buildout to ignore a shared eggs folder so it really has to fetch/build everything?

It is fully clean now, and it still works. I may simply have enough development tools installed so that source distributions work for me as well.

[dataflake]

I have made some progress. My local issues with Python 3.10 are solved by rebuilding the interpreter while making sure that the MACOSX_DEPLOYMENT_TARGET environment variable remains empty. After finally managing to step through the setuptools code that checks if an egg is already installed it turns out the already-installed egg from the setup_requires step is found, but deemed incompatible due to differences that stem from setting MACOSX_DEPLOYMENT_TARGET.

The GHA runs for macOS remain a problem. Python 3.11 still fails reliably installing some packages with binary extensions. Other Python versions fail spuriously with test failures in ZCatalog and CompositeIndex, I am guessing there are race conditions.

As both Maurits and I can install Zope cleanly now I suggest this PR is ready for review and merge.

[d-maurer]

Jens Vagelpohl wrote at 2022-11-11 02:43 -0800:

... The GHA runs for macOS remain a problem. Python 3.11 still fails reliably installing some packages with binary extensions. Other Python versions fail spuriously with test failures in ZCatalog and CompositeIndex, I am guessing there are race conditions.

I remember a "speed" test for `CompositeIndex` (it tries to check that the use of `CompositeIndex` is more efficient than a direct approach) which occasionally fails.

[dataflake]

The speed test is one of them, correct. There's another test for a query plan where the list of returned indices is not sorted the same way as the expected list. I don't even know if that's relevant at all. See https://github.com/zopefoundation/Zope/actions/runs/3444212276/jobs/5746887732

[dataflake]

Here's the current status:

• I have new GHA configurations in place for all c-code packages that will reliably build and test all wheels on macOS. It will also build arm64 wheels on Python 3.8 and up.
• Running tox on this branch on my local machine succeeds reliably.
• On GHA the macOS/Python 3.11 combo still fails reliably while trying to install zope.container. The error message is not informative, but it probably tries to pull and build the dependencies zope.proxy and persistent and since there is no wheel it will build them and fail. This issue should (will?) go away when there is a full complement of pre-built wheels for those packages.

Next steps:

• @icemac please review Add a flag to explicitly add testing on macOS test runners meta#172 and Enable building x86_64, arm64 and universal2 wheels on supported macOS Python versions meta#173, the current GHA known good configurations are based on those. Then I will merge them.
• I will go through the list of c-code packages from New Zope releases? 4.8.3 and 5.6.1 #1068 (comment) and create new bugfix releases for each. That will provide full sets of wheels.
• I will update the Zope versions files to specify these new releases and then try again.

[dataflake]

The "critical" binary packages and those binary packages that depend on them have been re-released and they all have the full set of macOS x86_64, arm64 and universal2 wheels. The test on Python 3.11 still fails, see https://github.com/zopefoundation/Zope/actions/runs/3479685094/jobs/5818541069.

I have stepped through the code in a local Python 3.10 environment that attempts to match how Python is built on those GHA runners and found one really dumb issue in zc.buildout, but I am not sure how to solve it.

Essentially, zc.buildout assumes that a query to an index for a given combination of (OS, architecture, Python version, package name, package version) will at most return two matches, the source distribution and a matching binary package. But on macOS this falls down. You get the source distribution, a matching binary package PLUS the universal2 format package because both binary packages match the requirement. I have added a comment on an issue that appears to be related, see buildout/buildout#609 (comment).

To see if this is the real issue I will experiment with leaving the universal2 wheel as the only binary wheel on PyPI for the affected packages by removing the arm64 and x86_64 wheels. Dumb workaround, but may work.

[dataflake]

None of my experiments have succeeded. I am caught in a series of issues in zc.buildout and setuptools, see buildout/buildout#609 (comment) and pypa/setuptools#3687. I doubt any of this will be solved in a reasonable time.

Question to @mauritsvanrees, @icemac and @d-maurer: Is it OK to "give up" when it comes to supporting oddly-built Python versions (namely, Python on macOS built with universal2 architecture and not the default from the platform itself, which would be arm64 or x86_64)?

If that's OK I would do the following:

• remove the universal2 eggs published for the five packages I am currently testing with (persistent, BTrees, zope.proxy, zope.container, zope.security)
• re-upload the arm64 and x86_64 wheels for these five packages where I have removed them as part of my tests
• remove the building of universal2 wheels from the GHA configurations
• remove at least the macOS/Python 3.11 build/test configuration from GHA, this will always fail because the GHA macOS runner is built with universal2 architecture
• continue making new releases for the remaining seven or eight binary packages needed by Zope so there are x86_64 and arm64 wheels for all of them

Then I will test on macOS/arm64 and will ask Maurits to test on macOS/x86_64.

[d-maurer]

Jens Vagelpohl wrote at 2022-11-17 04:17 -0800:

... Question to @mauritsvanrees, @icemac and @d-maurer: Is it OK to "give up" when it comes to supporting oddly-built Python versions (namely, Python on macOS built with `universal2` architecture and not the default from the platform itself, which would be `arm64` or `x86_64`)?

Neither me nor my clients have access to MAC systems. Thus, I have no strong feeling about MAC related problems. In all cases relevant to me, build tools are available. Thus, binary wheels are not necessary.

…

-- Dieter

[mauritsvanrees]

A full solution would be nice, but I can see it will not happen soon, despite your best efforts. Thanks a lot for trying!
I must say I had not even heard of universal2 wheels before this issue...

Your plan seems the best option for now.

BTW, we are planning to do the first Plone 6 release candidate today (or tomorrow). Would be cool if all those packages and Zope have releases before that. But don't stretch yourself. You have put in a lot of work already.

[dataflake]

BTW, we are planning to do the first Plone 6 release candidate today (or tomorrow). Would be cool if all those packages and Zope have releases before that. But don't stretch yourself. You have put in a lot of work already.

I could have it all done tomorrow early afternoon. The big time drain is waiting for the GHA test runs that want to build and test Linux on Arm packages.

[mauritsvanrees]

Wow, it is green! Great work.
I would say go for merge.