qemu and/or colima install fails (Python 3.12 dependency)

[strickvl]

brew gist-logs <formula> link OR brew config AND brew doctor output

export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1
  brew update
  brew install docker colima
  brew reinstall --force qemu
  
  # We need to mount the /private/tmp/zenml-test/ folder because
  # this folder is also mounted in the Docker containers that are
  # started by local ZenML orchestrators.
  colima start --mount /private/tmp/zenml-test/:w
  
  # This is required for the Docker Python SDK to work
  sudo ln -sf $HOME/.colima/***/docker.sock /var/run/docker.sock
  shell: /bin/bash --noprofile --norc -e -o pipefail ***0***
  env:
    ZENML_DEBUG: 1
    ZENML_ANALYTICS_OPT_IN: false
    PYTHONIOENCODING: utf-8
    UV_HTTP_TIMEOUT: 600
    OBJC_DISABLE_INITIALIZE_FORK_SAFETY: YES
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    AWS_US_EAST_1_SERVER_URL: ***
    AWS_US_EAST_1_SERVER_USERNAME: ***
    AWS_US_EAST_1_SERVER_PASSWORD: ***
    GCP_US_EAST4_SERVER_URL: ***
    GCP_US_EAST4_SERVER_USERNAME: ***
    GCP_US_EAST4_SERVER_PASSWORD: ***
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.11.8/x64
    PKG_CONFIG_PATH: /Users/runner/hostedtoolcache/Python/3.11.8/x64/lib/pkgconfig
    Python_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.8/x64
    Python2_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.8/x64
    Python3_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.8/x64
==> Homebrew's analytics have entirely moved to our InfluxDB instance in the EU.
We gather less data than before and have destroyed all Google Analytics data:
  https://docs.brew.sh/Analytics
Please reconsider re-enabling analytics to help our volunteer maintainers with:
  brew analytics on
==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
  https://github.com/Homebrew/brew#donations

Updated 5 taps (aws/tap, homebrew/bundle, homebrew/services, homebrew/core and homebrew/cask).
==> New Formulae
actions-batch
appwrite
aws/tap/codecatalyst-runner-cli
aws/tap/eksctl
bluez
bpftop
c4core
cargo-fuzz
cmake-language-server
dhall-toml
edbrowse
edgevpn
envio
gimmecert
glasskube
gptscript
kubeshark
liborigin
libsql
lsusb-laniksj
[email protected]
mdsh
mlx
mmdbinspect
mubeng
neosync
nmail
npm-check-updates
pass-import
phodav
pkl
podman-tui
prjtrellis
[email protected]
sigi
sqlboiler
srgn
sshpass
taskopen
typstfmt
vrc-get
vulkan-profiles
wstunnel
==> New Casks
acreom
amneziavpn
apidog-europe
arturia-software-center
deelay
dnsmonitor
dosbox-staging
endless-sky-high-dpi
fixkey
jan
jordanbaird-ice
kit
motion
notesollama
overlayed
segger-ozone
stashpad
wifiman
youlean-loudness-meter
==> Outdated Formulae
aws-sam-cli
azure-cli
c-ares
ca-certificates
composer
gettext
gh
ghostscript
git
git-lfs
glib
gnupg
harfbuzz
highway
icu4c
imagemagick
imath
jasper
jpeg-xl
kotlin
libassuan
libgpg-error
libksba
libnghttp2
libomp
libpng
libpq
librsvg
libunistring
libxcb
libxdmcp
libxext
mongosh
node
node@18
npth
openexr
openjpeg
opus
pango
pcre2
php
postgresql@14
[email protected]
[email protected]
r
readline
sbt
selenium-server
swiftformat
swig
tcl-tk
unbound
wget
xcbeautify
xz
yq
==> Outdated Casks
julia
virtualbox

You have 57 outdated formulae and 2 outdated casks installed.
You can upgrade them with brew upgrade
or list them with brew outdated.
Warning: Treating docker as a formula. For the cask, use homebrew/cask/docker or specify the `--cask` flag.
Warning: colima 0.6.8 is already installed and up-to-date.
To reinstall 0.6.8, run:
  brew reinstall colima
==> Downloading https://ghcr.io/v2/homebrew/core/docker/manifests/25.0.4
==> Fetching dependencies for docker: docker-completion
==> Downloading https://ghcr.io/v2/homebrew/core/docker-completion/manifests/25.0.4
==> Fetching docker-completion
==> Downloading https://ghcr.io/v2/homebrew/core/docker-completion/blobs/sha256:c57dae8c9386a00b6ad63d04cb5a1a7d0a231ec693997bb6ac2d72df401a7945
==> Fetching docker
==> Downloading https://ghcr.io/v2/homebrew/core/docker/blobs/sha256:1d9bb7844614a21943c4f1aef4e42573ae3b4dd3beaf27de8a5d1a7c90f55c40
==> Installing dependencies for docker: docker-completion
==> Installing docker dependency: docker-completion
==> Downloading https://ghcr.io/v2/homebrew/core/docker-completion/manifests/25.0.4
Already downloaded: /Users/runner/Library/Caches/Homebrew/downloads/4076fcf5e615782a62315a677ffc843710ea9bee817fc045450d2e7c042b676f--docker-completion-25.0.4.bottle_manifest.json
==> Pouring docker-completion--25.0.4.all.bottle.tar.gz
🍺  /usr/local/Cellar/docker-completion/25.0.4: 9 files, 338.7KB
==> Installing docker
==> Pouring docker--25.0.4.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/docker/25.0.4: 13 files, 23.6MB
Warning: HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK is set: not checking for outdated
dependents or dependents with broken linkage!
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/manifests/8.2.1
Already downloaded: /Users/runner/Library/Caches/Homebrew/downloads/ae4cef865d333db8a7c7ecb60234da075597b0d8235510d2764d65acaf405309--qemu-8.2.1.bottle_manifest.json
==> Fetching dependencies for qemu: readline and [email protected]
==> Downloading https://ghcr.io/v2/homebrew/core/readline/manifests/8.2.10
==> Fetching readline
==> Downloading https://ghcr.io/v2/homebrew/core/readline/blobs/sha256:3633320dce51662036ea90acfc9adf5bb5e6f1dca7dbdb539839736129c474b0
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.12/manifests/3.12.2_1
==> Fetching [email protected]
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.12/blobs/sha256:20655faca163d3ca0c567a4acb3ccb22388ca68dc3adff598a63525221eadd23
==> Fetching qemu
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:41c77f6bac3e8c1664c665fbe2b19b19ee9da57f8e1ad6697348286710cc3575
Already downloaded: /Users/runner/Library/Caches/Homebrew/downloads/827edd94c716e5674ca81bf012853117713a7bc548c483802b33e36b43132039--qemu--8.2.1.monterey.bottle.tar.gz
==> Reinstalling qemu 
==> Installing dependencies for qemu: readline and [email protected]
==> Installing qemu dependency: readline
==> Downloading https://ghcr.io/v2/homebrew/core/readline/manifests/8.2.10
Already downloaded: /Users/runner/Library/Caches/Homebrew/downloads/4ddd52803319828799f1932d4c7fa8d11c667049b20a56341c0c19246a1be93b--readline-8.2.10.bottle_manifest.json
==> Pouring readline--8.2.10.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/readline/8.2.10: 50 files, 1.7MB
==> Installing qemu dependency: [email protected]
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.12/manifests/3.12.2_1
Already downloaded: /Users/runner/Library/Caches/Homebrew/downloads/db6a7027cff95a641d6d0c07e00f7b8fa26b0733b6d215c3752b745a381ebcb8--python@3.12-3.12.2_1.bottle_manifest.json
==> Pouring [email protected]_1.monterey.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3'

To force the link and overwrite all conflicting files:
  brew link --overwrite [email protected]

To list all files that would be deleted:
  brew link --overwrite [email protected] --dry-run

Possible conflicting files are:
/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/2to3
/usr/local/bin/2to3-3.12 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/2to3-3.12
/usr/local/bin/idle3 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/idle3
/usr/local/bin/idle3.12 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/idle3.12
/usr/local/bin/pydoc3 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/pydoc3
/usr/local/bin/pydoc3.12 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/pydoc3.12
/usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/python3
/usr/local/bin/python3-config -> /Library/Frameworks/Python.framework/Versions/3.12/bin/python3-config
/usr/local/bin/python3.12 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12
/usr/local/bin/python3.12-config -> /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12-config
/usr/local/share/man/man1/python3.1 -> /usr/local/Cellar/[email protected]/3.11.7_1/share/man/man1/python3.1
/usr/local/lib/pkgconfig/python3-embed.pc -> /usr/local/Cellar/[email protected]/3.11.7_1/lib/pkgconfig/python3-embed.pc
/usr/local/lib/pkgconfig/python3.pc -> /usr/local/Cellar/[email protected]/3.11.7_1/lib/pkgconfig/python3.pc
/usr/local/Frameworks/Python.framework/Headers -> /usr/local/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Headers
/usr/local/Frameworks/Python.framework/Python -> /usr/local/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Python
/usr/local/Frameworks/Python.framework/Resources -> /usr/local/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Resources
/usr/local/Frameworks/Python.framework/Versions/Current -> /usr/local/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Versions/Current
==> /usr/local/Cellar/[email protected]/3.12.2_1/bin/python3.12 -Im ensurepip
==> /usr/local/Cellar/[email protected]/3.12.2_1/bin/python3.12 -Im pip install -v --n
==> Summary
🍺  /usr/local/Cellar/[email protected]/3.12.2_1: 3,237 files, 63.8MB
==> Installing qemu
==> Pouring qemu--8.2.1.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/qemu/8.2.1: 162 files, 556.1MB
Warning: HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK is set: not checking for outdated
dependents or dependents with broken linkage!

### Verification

- [X] My "`brew doctor` output" says `Your system is ready to brew.` and am still able to reproduce my issue.
- [X] I ran `brew update` and am still able to reproduce my issue.
- [X] I have resolved all warnings from `brew doctor` and that did not fix my problem.
- [X] I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

### What were you trying to do (and why)?

(See above, but trying to install colima and qemu.)

I'm running this in Github Actions. (https://github.com/zenml-io/zenml/actions/runs/8238362760/job/22529148434#step:13:256 for the direct logs)

### What happened (include all command output)?

See above. The installation fails on account of a Python 3.12 dependency, it seems.

### What did you expect to happen?

I would expect this to work normally. 3 days ago it worked without issue: https://github.com/zenml-io/zenml/actions/runs/8203529720/job/22436459693#step:13:192

### Step-by-step reproduction instructions (by running `brew` commands)

```shell
See above at the top.

[cho-m]

/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.12/bin/2to3

...
I'm running this in Github Actions. (https://github.com/zenml-io/zenml/actions/runs/8238362760/job/22529148434#step:13:256 for the direct logs)

This is more of upstream problem due to GitHub pre-installing a conflicting Python within Homebrew's default directory.

• https://github.com/orgs/Homebrew/discussions/3895
• brew update followed by brew upgrade fails for Python's 2to3 conflict (December 2022) actions/runner-images#11545

---

I would expect this to work normally. 3 days ago it worked without issue: https://github.com/zenml-io/zenml/actions/runs/8203529720/job/22436459693#step:13:192

That's because glib dependency now needs Homebrew Python because glib added a requirement on a non-standard Python package packaging.

---

There are a bunch of CI examples on working around this that you can pick from above links.

Homebrew does https://github.com/Homebrew/brew/blob/master/.github/workflows/tests.yml#L353-L355

        run: |
          # Workaround GitHub Actions Python issues
          brew unlink python && brew link --overwrite python

Meson does https://github.com/mesonbuild/meson/blob/master/.github/workflows/macos.yml#L87-L92

    # github actions overwrites brew's python. Force it to reassert itself, by running in a separate step.
    - name: unbreak python in github actions
      run: |
        find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete
        sudo rm -rf /Library/Frameworks/Python.framework/
        brew install --force python3 && brew unlink python3 && brew unlink python3 && brew link --overwrite python3

And there are probably a bunch of other ways.

[strickvl]

Thank you for this suggestion. I got this:

    # github actions overwrites brew's python. Force it to reassert itself, by running in a separate step.
    - name: unbreak python in github actions
      run: |
        find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete
        sudo rm -rf /Library/Frameworks/Python.framework/
        brew install --force python3 && brew unlink python3 && brew unlink python3 && brew link --overwrite python3

to work for MacOS on Python 3.8, 3.9 and 3.10, but subsequent steps in Python 3.11 complained that python wasn't available. Looking into how to fix this now.

[cho-m]

Python 3.11 complained that python wasn't available

I'm guessing that is due your CI workflow needing GitHub's Python.

Perhaps you can try unlinking Homebrew's Python instead.

[strickvl]

Ok this was the winning ticket for me (where we have Python used in subsequent steps:

run: |
          find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete
          sudo rm -rf /Library/Frameworks/Python.framework/
          brew install --force python3 && brew unlink python3 && brew unlink python3 && brew link --overwrite python3
      - name: Unbreak Python in GHA for 3.11
        if: runner.os=='macOS' && inputs.python-version == '3.11'
        run: |
          # Unlink and re-link to prevent errors when github mac runner images
          # https://github.com/actions/runner-images/issues/11545
          brew list -1 | grep python | while read formula; do brew unlink $formula; brew link --overwrite $formula; done
      - name: Install Docker and Colima on MacOS
        if: runner.os=='macOS'
        run: |
          export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1
          brew update
          brew install docker colima
          brew reinstall --force qemu

Cobbled together from other threads, esp https://github.com/casbin-rs/sqlx-adapter/pull/87/files and actions/runner-images#11545