TST: ppc64le fails with run-on-arch-install.sh error

[pllim]

It is in weekly cron job and it uses Python 3.12 currently.

astropy/.github/workflows/ci_cron_weekly.yml

Line 124 in 1f7a718

- uses: uraimo/run-on-arch-action@5397f9e30a9b62422f302092631c99ae1effcd9e # v2.8.1

Last success 2025-01-20: https://github.com/astropy/astropy/actions/runs/12862054703/job/35856390305

2025-01-20T06:09:12.6748094Z Image: ubuntu-24.04
2025-01-20T06:09:12.6748626Z Version: 20250105.1.0
2025-01-20T06:09:12.6749674Z Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250105.1/images/ubuntu/Ubuntu2404-Readme.md
2025-01-20T06:09:12.6751040Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250105.1

First failure 2025-01-23 (or so): https://github.com/astropy/astropy/actions/runs/12953740177/job/36134095980

2025-01-24T16:39:10.8596203Z Image: ubuntu-24.04
2025-01-24T16:39:10.8596748Z Version: 20250120.5.0
2025-01-24T16:39:10.8598184Z Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250120.5/images/ubuntu/Ubuntu2404-Readme.md
2025-01-24T16:39:10.8599582Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250120.5
...
2025-01-24T16:43:12.0483336Z #8 210.1 Setting up python3-pytest-mock (3.14.0-1) ...
2025-01-24T16:43:13.2961421Z #8 211.3 Traceback (most recent call last):
2025-01-24T16:43:13.5126577Z #8 211.3   File "/usr/bin/py3compile", line 323, in <module>
2025-01-24T16:43:13.5128505Z #8 211.3     main()
2025-01-24T16:43:13.5129158Z #8 211.3   File "/usr/bin/py3compile", line 302, in main
2025-01-24T16:43:13.5129678Z #8 211.3     compile(files, versions,
2025-01-24T16:43:13.5130165Z #8 211.3   File "/usr/bin/py3compile", line 203, in compile
2025-01-24T16:43:13.5130698Z #8 211.3     interpreter.magic_number(version),
2025-01-24T16:43:13.5131124Z #8 211.3     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24T16:43:13.5131794Z #8 211.3   File "/usr/share/python3/debpython/interpreter.py", line 233, in magic_number
2025-01-24T16:43:13.5132839Z #8 211.3     result = self._execute('import importlib.util; print(importlib.util.MAGIC_NUMBER)', version)
2025-01-24T16:43:13.5133623Z #8 211.3              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24T16:43:13.5134319Z #8 211.3   File "/usr/share/python3/debpython/interpreter.py", line 359, in _execute
2025-01-24T16:43:13.5135214Z #8 211.3     raise Exception('{} failed with status code {}'.format(command, output['returncode']))
2025-01-24T16:43:13.5136496Z #8 211.3 Exception: ('python3.12', '-c', 'import importlib.util; print(importlib.util.MAGIC_NUMBER)') failed with status code -11
2025-01-24T16:43:13.5137774Z #8 211.4 dpkg: error processing package python3-pytest-mock (--configure):
2025-01-24T16:43:13.5138782Z #8 211.4  installed python3-pytest-mock package post-installation script subprocess returned error exit status 1
2025-01-24T16:43:13.5139754Z #8 211.4 Setting up python3-pytest-filter-subpackage (0.2.0-1) ...
2025-01-24T16:43:15.1514061Z #8 213.2 Setting up python3-pytest-astropy-header (0.2.2-1) ...
2025-01-24T16:43:16.6670014Z #8 214.7 Setting up python3-pkg-resources (74.1.2-1) ...
2025-01-24T16:43:17.4970780Z #8 215.5 Traceback (most recent call last):
2025-01-24T16:43:17.6699215Z #8 215.5   File "/usr/bin/py3compile", line 323, in <module>
2025-01-24T16:43:17.6701647Z #8 215.5     main()
2025-01-24T16:43:17.6702427Z #8 215.5   File "/usr/bin/py3compile", line 302, in main
2025-01-24T16:43:17.6703121Z #8 215.5     compile(files, versions,
2025-01-24T16:43:17.6703622Z #8 215.5   File "/usr/bin/py3compile", line 187, in compile
2025-01-24T16:43:17.6704155Z #8 215.5     cfn = interpreter.cache_file(fn, version)
2025-01-24T16:43:17.6704622Z #8 215.5           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24T16:43:17.6705256Z #8 215.5   File "/usr/share/python3/debpython/interpreter.py", line 212, in cache_file
2025-01-24T16:43:17.6705970Z #8 215.5     (fname[:-3], self.magic_tag(version), last_char))
2025-01-24T16:43:17.6706452Z #8 215.5                  ^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24T16:43:17.6707467Z #8 215.5   File "/usr/share/python3/debpython/interpreter.py", line 246, in magic_tag
2025-01-24T16:43:17.6708325Z #8 215.5     return self._execute('import sys; print(sys.implementation.cache_tag)', version)
2025-01-24T16:43:17.6708755Z #8 215.5            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24T16:43:17.6709646Z #8 215.5   File "/usr/share/python3/debpython/interpreter.py", line 359, in _execute
2025-01-24T16:43:17.6710399Z #8 215.5     raise Exception('{} failed with status code {}'.format(command, output['returncode']))
2025-01-24T16:43:17.6711414Z #8 215.5 Exception: ('python3.12', '-c', 'import sys; print(sys.implementation.cache_tag)') failed with status code -11
...
2025-01-24T16:43:30.7393756Z #8 228.7 Errors were encountered while processing:
2025-01-24T16:43:30.7394226Z #8 228.7  python3-pytest-mock
2025-01-24T16:43:30.9142336Z #8 228.7  python3-pkg-resources
2025-01-24T16:43:30.9143015Z #8 228.7  python3-setuptools
2025-01-24T16:43:30.9143440Z #8 228.7  python3-extension-helpers
2025-01-24T16:43:30.9143888Z #8 228.7  python3-coverage
2025-01-24T16:43:30.9144375Z #8 228.7  python3-pytest-cov
2025-01-24T16:43:30.9147273Z #8 228.7  python3-setuptools-scm
2025-01-24T16:43:30.9147722Z #8 228.7  python3-numpy
2025-01-24T16:43:30.9148793Z #8 228.7  python3-pytest-doctestplus
2025-01-24T16:43:30.9149222Z #8 228.7  python3-pytest-astropy
2025-01-24T16:43:30.9149766Z #8 228.7  python3-pytest-arraydiff
2025-01-24T16:43:30.9150046Z #8 228.7  python3-erfa
...
2025-01-24T16:43:30.9254437Z 228.8 E: Sub-process /usr/bin/dpkg returned an error code (1)
2025-01-24T16:43:30.9254921Z ------
2025-01-24T16:43:30.9269364Z Dockerfile.ppc64le.ubuntu_rolling:4
2025-01-24T16:43:30.9269837Z --------------------
2025-01-24T16:43:30.9270139Z    2 |     
2025-01-24T16:43:30.9270545Z    3 |     COPY ./run-on-arch-install.sh /root/run-on-arch-install.sh
2025-01-24T16:43:30.9271246Z    4 | >>> RUN chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh
2025-01-24T16:43:30.9271793Z    5 |     
2025-01-24T16:43:30.9272055Z --------------------
2025-01-24T16:43:30.9272964Z ERROR: failed to solve: process "/bin/sh -c chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh" did not complete successfully: exit code: 100
2025-01-24T16:43:30.9366199Z ##[error]The process '/home/runner/work/_actions/uraimo/run-on-arch-action/5397f9e30a9b62422f302092631c99ae1effcd9e/src/run-on-arch.sh' failed with exit code 1

Blocked by (from @neutrinoceros post below):

• qemu-aarch64 nondeterministic crashes docker/setup-qemu-action#188 (comment) (we use setup-qemu-action via OpenAstronomy/github-actions-workflows)
• Intermittent segmentation fault running Node emulated with target arm64, host amd64 tonistiigi/binfmt#215

[neutrinoceros]

This issue appears to affect the ecosystem as a whole and is being discussed upstream.
Let me try to link the breadcrums I found so far:

• qemu-aarch64 nondeterministic crashes docker/setup-qemu-action#188 (comment) (we use setup-qemu-action via OpenAstronomy/github-actions-workflows)
• Intermittent segmentation fault running Node emulated with target arm64, host amd64 tonistiigi/binfmt#215

[geerlingguy]

This is also hitting all of my debian/downstream container builds currently :( (e.g. https://github.com/geerlingguy/docker-debian12-ansible/actions/runs/12970333663/job/36175471137) - thanks for posting this issue as it's the only thing I could find first page of Google.

[neutrinoceros]

It looks like the problem might have been fixed with the release of tonistiigi/binfmt:latest with QEMU 9.2.0

trying it out at https://github.com/astropy/astropy/actions/runs/13280401101

update: no luck yet

[pllim]

I feel like we might need uraimo to catch up or something.