AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

[aorumbayev]

Describe the problem

The following repo was specifically created to outline a reproducible environment with the bug https://github.com/aorumbayev/pipx_bug.

The root cause is incompatibility with python 3.12.

Template

N/A

To Reproduce

Refer to readme on https://github.com/aorumbayev/pipx_bug

Logs

Traceback (most recent call last):
  File "/usr/local/py-utils/bin/pipx-bug-cli", line 5, in <module>
    from pipx_bug.cli import hello
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/pipx_bug/cli.py", line 2, in <module>
    from copier.main import Worker
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/copier/__init__.py", line 6, in <module>
    from .main import *  # noqa: F401,F403
    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/copier/main.py", line 33, in <module>
    from plumbum.cli.terminal import ask
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/plumbum/cli/__init__.py", line 1, in <module>
    from .application import Application
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/plumbum/cli/application.py", line 9, in <module>
    from plumbum.cli.i18n import get_translation_for
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/plumbum/cli/i18n.py", line 28, in <module>
    import pkg_resources
  File "/usr/local/py-utils/shared/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2191, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

Expected behavior

copier works on >=3.12

Screenshots/screencasts/logs

No response

Operating system

Linux

Operating system distribution and version

ubuntu 20

Copier version

9.2.0

Python version

3.12.2

Installation method

pipx+pypi

Additional context

No response

[pawamoy]

I cannot reproduce the issue in a local environment.

While you install for Python 3.12, pkg_resources is picked up in a Python 3.10 environment: /usr/local/py-utils/shared/lib/python3.10/site-packages/pkg_resources. These are the shared libraries of pipx if I understand correctly. Seems like an environment issue to me 😕 Needs further investigation.

[aorumbayev]

@pawamoy exactly, its only happening when you are installing a package that has an executable via pipx where pipx itself was initially installed with anything <3.12. Pipx has this mechanism where they chain virtual envs, there is a shared virtual env and a virtual env per each package you install via pipx - the shared venv is created during initial pipx install.

On all default ubuntu github actions workers as well as github codespaces, pipx is preinstalled via 3.10. Hence when you additionally install 3.12 and try to use pipx to install using 3.12 python - it will fail due to that virtual env chaining where it tries to access the 3.10 venv (containing pkg_resources). Why is it doing so in that manner? I have 0 clues.

I will open this on pipx repo as well just in case but i still think the root cause is something to do with the plumbum package that copier is using

[pawamoy]

Thanks for the explanation @aorumbayev! I agree it's confusing and I have 0 clue either how to fix. Maybe removing the shared venv of pipx before installing your project would fix it? That's a hack though. Maybe don't install with pipx? But if your project is an extension for pipx, you definitely want to install with it to test propertly.

The use of pkg_resources in Plumbum indicates code that could be modernized a bit indeed, as nowadays I think the preferred way is to use importlib.resources. Feel free to open an issue on their tracker, linking back to this one 🙂

[aorumbayev]

@pawamoy thanks, sure will fill the same issue on their repo. Hopefully a quick patch from their side is all it takes

[henryiii]

The root issue is the environment is misconfigured; a Python 3.10 site packages is in the path below the 3.12 one; trying to import setuptools gets the 3.10 one.

However, if this is not running localization, it's supposed to take a fast path and avoid this, but it wasn't happening for the "C" locale. Fixing that (out in 1.8.3) should fix this in most cases. And it's technically not a plumbum bug.

I've also got a importlib.metadata version, but I'm not sure we have great testing for localization, so going to wait on that a bit (and likely go 3.8+ soon).

[pawamoy]

Thanks for your help @henryiii!

I've followed along the other opened issues, and can't understand where the Python 3.10 path is coming from either.

I feel like Copier can't do anything here to fix the issue, so I'd vote to close (@copier-org/maintainers). Happy to try and help in the Plumbum/pipx issues if someone tags me there 🙂

[pawamoy]

Definitely looks like an issue with pipx. Seems the shared env is always there in sys.path. I don't know pipx enough to understand why a shared venv is required, but it also doesn't seem to provide an option to disable it.

[yajo]

Yes, I will close it here because there seems to be nothing Copier can do to fix it 🤷🏼‍♂️