CI: Builds failing in `pandas.show_versions()` #44980

johnzangwill · 2021-12-20T11:07:10Z

Failing on all platofrms. E.g.
https://github.com/pandas-dev/pandas/runs/4581277621

Run python -c "import pandas; pandas.show_versions();"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/runner/work/pandas/pandas/pandas/util/_print_versions.py", line 109, in show_versions
    deps = _get_dependency_info()
  File "/home/runner/work/pandas/pandas/pandas/util/_print_versions.py", line 88, in _get_dependency_info
    mod = import_optional_dependency(modname, errors="ignore")
  File "/home/runner/work/pandas/pandas/pandas/compat/_optional.py", line 115, in import_optional_dependency
    module = importlib.import_module(name)
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 83, in find_spec
    return method()
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 104, in spec_for_pip
    if self.pip_imported_during_build():
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 115, in pip_imported_during_build
    return any(
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 116, in <genexpr>
    frame.f_globals['__file__'].endswith('setup.py')
KeyError: '__file__'
Error: Process completed with exit code 1.

The text was updated successfully, but these errors were encountered:

jreback · 2021-12-20T18:57:10Z

yeah maybe something on a pip release?

cc @pandas-dev/pandas-core

MarcoGorelli · 2021-12-20T19:00:27Z

I was looking into this earlier and couldn't reproduce it, even in a clean conda environment with the latest setuptools and wheel, couldn't even find other issues about it

phofl · 2021-12-20T20:52:49Z

pypa/setuptools#2941

jreback · 2021-12-21T02:32:37Z

looks like 60.0.3 was released to fix this

phofl · 2021-12-21T12:40:18Z

60.0.3 did not fix all problems for us unfortunately. When running this locally I get

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/testenv2/lib/python3.10/site-packages/pandas/util/_print_versions.py", line 109, in show_versions
    deps = _get_dependency_info()
  File "/tmp/testenv2/lib/python3.10/site-packages/pandas/util/_print_versions.py", line 88, in _get_dependency_info
    mod = import_optional_dependency(modname, errors="ignore")
  File "/tmp/testenv2/lib/python3.10/site-packages/pandas/compat/_optional.py", line 115, in import_optional_dependency
    module = importlib.import_module(name)
  File "/tmp/testenv2/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/tmp/testenv2/lib/python3.10/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/tmp/testenv2/lib/python3.10/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/tmp/testenv2/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 76, in do_override
    ensure_local_distutils()
  File "/tmp/testenv2/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 63, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /tmp/testenv2/lib/python3.10/distutils/core.py

It looks like it tries to distutils from stdlib, but if running

import importlib
importlib.import_module("setuptools")

it is pulling the distutils from

'/tmp/testenv2/lib/python3.10/site-packages/setuptools/_distutils'

Not sure what is different when doing this via pandas.show_versions()

lithomas1 · 2021-12-26T20:15:16Z

This happens because show_version() imports all the dependencies installed including pip and setuptools. Apparently importing pip before setuptools causes this to happen. Swapping these too lines seems to fix it for me.

pandas/pandas/util/_print_versions.py

Lines 63 to 64 in e18921e

    
           "pip", 
        
           "setuptools",

Perhaps its better to wait for an upstream fix, though?

mgorny · 2022-01-22T21:25:34Z

I couldn't find an upstream bug report about this and I've reproduced it with git HEAD, so I've filed pypa/setuptools#3044.

mgorny · 2022-01-29T19:39:23Z

Perhaps its better to wait for an upstream fix, though?

Upstream has closed the bug as wontfix. Basically, pip has weird internal expectations that they can't change right now and importing pip is discouraged anyway.

MarcoGorelli · 2022-01-29T20:27:33Z

Probably fine to just remove pip from the list of packages to print the version of then?

max-uzunov · 2022-05-06T07:22:16Z

To workaround this issue one can set an OS environment variable: SETUPTOOLS_USE_DISTUTILS=stdlib - this should work for setuptools v60+.

pypa/setuptools#2965

simonjayhawkins · 2022-05-23T12:19:19Z

This happens because show_version() imports all the dependencies installed including pip and setuptools. Apparently importing pip before setuptools causes this to happen. Swapping these too lines seems to fix it for me.

pandas/pandas/util/_print_versions.py

Lines 63 to 64 in e18921e

"pip",

"setuptools",

This appears to be correct

import importlib

importlib.import_module("pip")
importlib.import_module("setuptools")

gives

...
File ~/miniconda3/envs/pandas-1.4.2/lib/python3.10/site-packages/_distutils_hack/__init__.py:59, in ensure_local_distutils()
     57 # check that submodules load as expected
     58 core = importlib.import_module('distutils.core')
---> 59 assert '_distutils' in core.__file__, core.__file__
     60 assert 'setuptools._distutils.log' not in sys.modules

AssertionError: /home/simon/miniconda3/envs/pandas-1.4.2/lib/python3.10/distutils/core.py

as now appearing in pre-release checks https://github.com/simonjayhawkins/pandas-release/runs/6552535479?check_suite_focus=true

rerunning the above code sample in a jupyter notebook, gives a warning second time round

/home/simon/miniconda3/envs/pandas-1.4.2/lib/python3.10/site-packages/_distutils_hack/__init__.py:17: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/home/simon/miniconda3/envs/pandas-1.4.2/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

Perhaps its better to wait for an upstream fix, though?

switching order of imports, works fine...

import importlib

importlib.import_module("setuptools")
importlib.import_module("pip")

johnzangwill mentioned this issue Dec 20, 2021

ENH: reset_index on a MultiIndex with duplicate levels raises a ValueError #44755

Merged

MarcoGorelli added the CI Continuous Integration label Dec 20, 2021

phofl mentioned this issue Dec 20, 2021

CI: Pin setuptools for python dev #44989

Merged

1 task

snowman2 mentioned this issue Jan 26, 2022

BUG: Reorder deps in show_versions for setuptools issue pyproj4/pyproj#1019

Merged

2 tasks

santhisenan mentioned this issue Feb 12, 2022

Passing an empty list to read_csv causes segmentation fault #45957

Closed

simonjayhawkins added this to the 1.4.3 milestone May 23, 2022

simonjayhawkins added Dependencies Required and optional dependencies Python 3.10 and removed Python 3.10 labels May 23, 2022

simonjayhawkins mentioned this issue May 23, 2022

fix pandas.show_versions() and remove pin for setuptools #47096

Merged

mroeschke mentioned this issue May 24, 2022

Issue template versions command no longer works #47106

Closed

jreback closed this as completed in #47096 May 25, 2022

hildeweerts mentioned this issue Jul 12, 2023

MNT tests are failing fairlearn/fairlearn#1254

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Builds failing in `pandas.show_versions()` #44980

CI: Builds failing in `pandas.show_versions()` #44980

johnzangwill commented Dec 20, 2021 •

edited

Loading

jreback commented Dec 20, 2021

MarcoGorelli commented Dec 20, 2021

phofl commented Dec 20, 2021

jreback commented Dec 21, 2021

phofl commented Dec 21, 2021

lithomas1 commented Dec 26, 2021

mgorny commented Jan 22, 2022

mgorny commented Jan 29, 2022

MarcoGorelli commented Jan 29, 2022

max-uzunov commented May 6, 2022

simonjayhawkins commented May 23, 2022

CI: Builds failing in pandas.show_versions() #44980

CI: Builds failing in pandas.show_versions() #44980

Comments

johnzangwill commented Dec 20, 2021 • edited Loading

jreback commented Dec 20, 2021

MarcoGorelli commented Dec 20, 2021

phofl commented Dec 20, 2021

jreback commented Dec 21, 2021

phofl commented Dec 21, 2021

lithomas1 commented Dec 26, 2021

mgorny commented Jan 22, 2022

mgorny commented Jan 29, 2022

MarcoGorelli commented Jan 29, 2022

max-uzunov commented May 6, 2022

simonjayhawkins commented May 23, 2022

CI: Builds failing in `pandas.show_versions()` #44980

CI: Builds failing in `pandas.show_versions()` #44980

johnzangwill commented Dec 20, 2021 •

edited

Loading