MacOS Issue with python

[ScreamZ]

Hello !

When running setup for ESP32 i get

✔ Here are the available target devices: · esp32
ℹ Ensuring esp32 install directory
⠸ Installing pyserial through pip3Command failed, Error: Command failed with exit code 1: python3 -m pip install pyserial

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: python3.13 -m pip install --upgrade pip
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:

    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz

    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with

    brew install pipx

    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.

    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.

    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
    at makeError (/Users/screamz/Library/pnpm/global/5/.pnpm/[email protected]/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/Users/screamz/Library/pnpm/global/5/.pnpm/[email protected]/node_modules/execa/index.js:118:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                               1.41s

Could be nice to use pipx instead ?

Also I have try to Create a virtual environment:

python3 -m venv myenv

source myenv/bin/activate

but later it crashes for other reason (related to virtual env ?)

✔ Here are the available target devices: · esp32
ℹ Ensuring esp32 install directory
✔ Installing pyserial through pip3
⠙ Installing esp-idf toolingDetecting the Python interpreter
Checking "python3" ...
⠹ Installing esp-idf toolingPython 3.13.0
"python3" has been detected
Checking Python compatibility
⠸ Installing esp-idf toolingInstalling ESP-IDF tools
Updating /Users/screamz/.espressif/idf-env.json
Selected targets are: esp32p4, esp32c5, esp32c3, esp32c6, esp32c61, esp32h2, esp32s3, esp32s2, esp32c2, esp32
Current system platform: macos-arm64
Installing tools: xtensa-esp-elf-gdb, riscv32-esp-elf-gdb, xtensa-esp-elf, riscv32-esp-elf, esp32ulp-elf, openocd-esp32, esp-rom-elfs
Installing [email protected]_20240403
Downloading https://github.com/espressif/binutils-gdb/releases/download/esp-gdb-v14.2_20240403/xtensa-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz
Destination: /Users/screamz/.espressif/dist/xtensa-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/xtensa-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz to /Users/screamz/.espressif/tools/xtensa-esp-elf-gdb/14.2_20240403
⠼ Installing esp-idf toolingInstalling [email protected]_20240403
Downloading https://github.com/espressif/binutils-gdb/releases/download/esp-gdb-v14.2_20240403/riscv32-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz
Destination: /Users/screamz/.espressif/dist/riscv32-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/riscv32-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz to /Users/screamz/.espressif/tools/riscv32-esp-elf-gdb/14.2_20240403
⠙ Installing esp-idf toolingInstalling [email protected]_20240530
Downloading https://github.com/espressif/crosstool-NG/releases/download/esp-13.2.0_20240530/xtensa-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz
Destination: /Users/screamz/.espressif/dist/xtensa-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/xtensa-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz to /Users/screamz/.espressif/tools/xtensa-esp-elf/esp-13.2.0_20240530
⠏ Installing esp-idf toolingInstalling [email protected]_20240530
Downloading https://github.com/espressif/crosstool-NG/releases/download/esp-13.2.0_20240530/riscv32-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz
Destination: /Users/screamz/.espressif/dist/riscv32-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz.tmp
Donestalling esp-idf tooling
⠴ Installing esp-idf toolingExtracting /Users/screamz/.espressif/dist/riscv32-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz to /Users/screamz/.espressif/tools/riscv32-esp-elf/esp-13.2.0_20240530
⠸ Installing esp-idf toolingInstalling [email protected]_20240113
Downloading https://github.com/espressif/binutils-gdb/releases/download/esp32ulp-elf-2.38_20240113/esp32ulp-elf-2.38_20240113-macos-arm64.tar.gz
Destination: /Users/screamz/.espressif/dist/esp32ulp-elf-2.38_20240113-macos-arm64.tar.gz.tmp
Done Installing esp-idf tooling
Extracting /Users/screamz/.espressif/dist/esp32ulp-elf-2.38_20240113-macos-arm64.tar.gz to /Users/screamz/.espressif/tools/esp32ulp-elf/2.38_20240113
⠙ Installing esp-idf toolingInstalling [email protected]
Downloading https://github.com/espressif/openocd-esp32/releases/download/v0.12.0-esp32-20240318/openocd-esp32-macos-arm64-0.12.0-esp32-20240318.tar.gz
Destination: /Users/screamz/.espressif/dist/openocd-esp32-macos-arm64-0.12.0-esp32-20240318.tar.gz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/openocd-esp32-macos-arm64-0.12.0-esp32-20240318.tar.gz to /Users/screamz/.espressif/tools/openocd-esp32/v0.12.0-esp32-20240318
⠧ Installing esp-idf toolingInstalling esp-rom-elfs@20240305
Downloading https://github.com/espressif/esp-rom-elfs/releases/download/20240305/esp-rom-elfs-20240305.tar.gz
Destination: /Users/screamz/.espressif/dist/esp-rom-elfs-20240305.tar.gz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/esp-rom-elfs-20240305.tar.gz to /Users/screamz/.espressif/tools/esp-rom-elfs/20240305
⠇ Installing esp-idf toolingInstalling Python environment and packages
⠏ Installing esp-idf toolingCommand failed, Error: Command failed with exit code 1: ./install.sh
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
ERROR: This script was called from a virtual environment, can not create a virtual environment again
    at makeError (/Users/screamz/Library/pnpm/global/5/.pnpm/[email protected]/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/Users/screamz/Library/pnpm/global/5/.pnpm/[email protected]/node_modules/execa/index.js:118:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                              36.81s

I would be pleased to help for the macOS setup.

Pip seems managed by homebrew on my system.

xs-dev environment info:
  CLI Version                0.33.0
  OS                         Darwin
  Arch                       arm64
  Shell                      /bin/zsh
  NodeJS Version             v20.11.1 (/Users/screamz/.volta/tools/image/node/20.11.1/bin/node)
  Python Version              (/Library/Frameworks/Python.framework/Versions/2.7/bin/python)
  Moddable SDK Version       5.2.0 (/Users/screamz/dev-workspace/iot-moddable-workspace/moddable)
  Supported target devices   mac

If this is related to an error when using the CLI, please create an issue at "https://github.com/hipsterbrown/xs-dev/issues/new" with the above info.                                                                                918ms

Note

Looks like there is a an issue with the CLI Code

xs-dev/src/toolbox/setup/esp32/mac.ts

Lines 71 to 76 in 61ef071

	// 5. pip install pyserial, if needed
	spinner.start('Installing pyserial through pip3')
	await system.exec('python3 -m pip install pyserial', {
	shell: process.env.SHELL,
	})
	spinner.succeed()

This doesn't check for existence before install

[HipsterBrown]

Thanks for all this information @ScreamZ

It's interesting that you have python 2.7 on your system but not python 3 by default. This is a bad assumption in the CLI.

Could be nice to use pipx instead ?

Since pyserial is called by the underlying ESP-IDF tooling for flashing those devices, xs-dev is not able to swap out the pip install for pipx. But there should be additional checks before attempting that install step.

but later it crashes for other reason (related to virtual env ?)

Yes, the ESP-IDF install script will attempt to create a virtual environment for its dependencies and other scripts that get called when compiling and flashing a program.

[ScreamZ]

I can confirm that switching to mise and installing python this way solved the issue.

[github-actions]

🚀 Issue was released in v0.33.1 🚀