[BUG] python setup.py develop/install installs package in site-packages, but only dist-packages is on path

[shughes-uk]

setuptools version

67.6.1

Python version

Python 3.8

OS

Ubuntu 20:04

Additional environment information

Using python 3.8 installed via apt-get.

Description

Installing a package via python setup.py develop/install on ubuntu:20.04, results in the package being installed to the wrong place, site-packages instead of dist-packages

I've confirmed previous versions of setuptools correctly install things to dist-packages (45.2.0)

Expected behavior

Running python setup.py install makes the package importable

How to Reproduce

python setup.py install with the system python on ubuntu:20.04

Output

+ python3 setup.py develop
running develop
/usr/local/lib/python3.8/dist-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer, pypa/build or
        other standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer, pypa/build or
        other standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running egg_info
writing egg_link_package.egg-info/PKG-INFO
writing dependency_links to egg_link_package.egg-info/dependency_links.txt
writing top-level names to egg_link_package.egg-info/top_level.txt
reading manifest file 'egg_link_package.egg-info/SOURCES.txt'
writing manifest file 'egg_link_package.egg-info/SOURCES.txt'
running build_ext
Creating /usr/lib/python3.8/site-packages/egg-link-package.egg-link (link to .)
Adding egg-link-package 0.1.3 to easy-install.pth file

Installed /workdir/integration-tests/package-sync/dummy-packages/egg_link_package
Processing dependencies for egg-link-package==0.1.3
Finished processing dependencies for egg-link-package==0.1.3

[abravalheri]

Hi @shughes-uk , isn't dist-packages supposed to be used by the OS package manager only (apt)?

What is the output of python3 -m site? Where does pip install places packages?

Please also try to avoid using python setup.py install this is no longer recommended and not fully supported.

[shughes-uk]

Here's the output of things. I too thought dist-packages was only supposed to be used by apt but I guess it makes sense that you'd sometimes need to install a local package.

docker run -it ubuntu:20.04
apt-get update -y && apt-get install -y python3 python3-pip
python3 -m site
--- output ----
          sys.path = [
              '/',
              '/usr/lib/python38.zip',
              '/usr/lib/python3.8',
              '/usr/lib/python3.8/lib-dynload',
              '/usr/local/lib/python3.8/dist-packages',
              '/usr/lib/python3/dist-packages',
          ]
          USER_BASE: '/root/.local' (doesn't exist)
          USER_SITE: '/root/.local/lib/python3.8/site-packages' (doesn't exist)
          ENABLE_USER_SITE: True
pip3 install backoff
python3 -c "import backoff; print(backoff.__file__)"
---- output ----
/usr/local/lib/python3.8/dist-packages/backoff/__init__.py

I understand setup.py is definitely not recommended. We only use it to set up an test environment for part of our product that scans user python environments, specifically we create some old fashioned .egg and egg-link installs!

We can probably work around that by pinning to an older setuptools in our testing. Things are deprecated but nothing so far claims it wont work entirely , especially on something as common at ubuntu 20.04 system python. So this was fairly surprising and would have been quite painful to debug if I wasn't very experienced with the technical details of python environments.

[abravalheri]

Probably what is happening here is that prior to Python 3.10, debian-systems patch the distutils that comes with the standard library to modify the paths.

However setuptools will try to use a more modern version of the distutils with some improvements and bug fixes.

A new mechanism was devised in distutils to allow debian maintainers to patch installation paths. It seems that debian maintainers are aware of this methodology and that it was implemented for Python > 3.10, but probably not ported back to Python 3.8.

So there are a few ways of dealing with this:

1. You can set SETUPTOOLS_USE_DISTUTILS=stdlib to force setuptools use the stdlib version (ref).
2. You can use the setuptools distribution that comes with apt (the general recommendation of distribution maintainers is to stick with the packages they curate and distribute; so everything is tested and integrates well).
3. You can use a virtual environment.

For more information, there are quite a few issues in pypa/setuptools and pypa/distutils discussing the approach, but I think this post summarises the overall reasoning: #2956 (comment).